Toil

Toil is work that is manual, repetitive, automatable, interrupt-driven, leaves the system no better than it found it, and scales linearly with traffic or users. Each instance is small and urgent, so it always beats project work in the moment. Because it grows with the service and engineering time does not, a team that never pushes back ends up doing nothing else.

Google's datacenter network engineers had to handle each failed line card in its Saturn and Jupiter switch fabrics by hand: check that traffic could move, drain the switch, reboot or replace the card, and undrain it. Jupiter was more than six times larger than any earlier Google fabric, and the volume of these repairs grew with it. The team automated the procedure, first reboot and reinstall, then replacement on a second failure, which freed engineers for other work; it also concluded that spending nearly three years (2012-2015) analyzing memory errors before automating had been unnecessary.

Ironies of Automation: automating a task removes the practice people needed to handle it when the automation fails, and some automation costs more to build and keep alive than the toil it replaces.