Hammurabi
Back to the codex

52

Distributed Systems

The Tail at Scale

Tail latency dominates at fan-out.

The article

If

In a service that fans out to many components, rare per-node slowness (GC pauses, contention, a failing disk) is near-certain to hit some node on every request. So tail latency (p99, p99.9) governs the user experience even when medians look healthy.

Then

“Set SLOs on p99/p99.9, not the mean, and deploy tail-tolerance — hedged requests, micro-partitioning — instead of trying to make every node uniformly fast.”

Unless

Redundancy Has a Price — hedged and tied requests add load; don't chase the tail where the median is what users feel

One request fanning out to 100 leaves; the odds that at least one is slowRequestLeavesslow leafeach leaf independently slow 1% of the timeFan-outP(at least one slow leaf)1 leaf1%10 leaves10%100 leaves63%
A leaf that is slow one request in a hundred is invisible on its own. Fan out to 100 of them and the same leaf-level p99 becomes the median user experience: 63% of requests now wait on a straggler. Tail latency is not a per-node property at scale — it is an arithmetic consequence of the fan-out.

Source

Jeffrey Dean & Luiz André Barroso, The Tail at Scale (Communications of the ACM, 2013).

Go to the source

See also

Improve this article on GitHub