Toil Reduction

Toil is defined by specific characteristics: it is manual, repetitive, automatable, tactical, lacks enduring value, and grows proportionally with the service. Restarting a failed pipeline, manually provisioning user accounts, or hand-editing configuration files are all examples of toil. SRE teams aim to spend no more than 50% of their time on toil, investing the remainder in automation and improvements that reduce future toil.

For AI product teams, common sources of toil include manually retraining models on schedule, hand-labeling edge cases, manually updating feature store configurations, and responding to model performance alerts by adjusting thresholds. Each of these tasks can be automated with appropriate investment. Growth teams accumulate toil through manual experiment setup, repetitive metric report generation, and hand-managed audience segmentation. Investing in automation tooling, experiment platforms that reduce setup time, automated reporting pipelines, and self-serve audience management frees growth engineers to focus on high-value strategic work rather than operational overhead that scales with the number of active experiments.