baam's wiki

Tail latency: the number the average hides

Evergreen

Jan 23, 2025 3 min read

An average hides the requests that hurt. A service can post a healthy mean and a fine p95 while its slowest requests - the ones a real user actually notices - are quietly unacceptable. Tail latency is the study of that slow end of the distribution, and in most systems it is where performance work actually pays off.

Percentiles in one paragraph

A percentile reads the latency distribution from the slow end. P99 = 100 ms means 99% of requests finish in under 100 ms and the slowest 1% take longer; P99.9 says the same for the slowest one in a thousand. The average blends all of this into a single number that no individual user ever experiences. People feel the slow requests, not the mean, so the honest way to describe performance is a percentile, not an average.

A latency distribution drawn as a bell curve. Most requests cluster around p50; the region past p95 is shaded as the slow tail, with p99 and p99.9 marked far out on the right.
Most requests sit near p50. The shaded tail past p95 is the slow minority - rare, but the part users actually feel.

Why the tail dominates at fan-out

The tail gets worse, not better, as a system grows. When one user request fans out into many internal service calls and has to wait for all of them, it is exposed to the slowest of the batch. A request that touches 100 services, each with a rock-solid p99, will almost always land in someone’s slow 1% - so the “rare” tail of each service becomes the typical experience of the whole request. This is why tail latency is a systems problem, not a single-service one.

Why it happens

The slow end usually traces back to a handful of recurring causes:

  • Garbage-collection pauses that freeze a process at the wrong moment. See garbage_collection.
  • Queueing and contention - a request waits behind others for a thread, a lock, a connection, or a disk. See Queues Are Everywhere.
  • Cold caches - the unlucky request is the one that misses and pays the full price.
  • Distance - a far-away client or replica pays network round-trips the median local request never sees.

How to defend it

Tail latency is defended in layers, not with a single trick.

At the request level:

  • Timeouts and backpressure cap how long anything is allowed to be slow, and stop a slow consumer from silently building an unbounded backlog.
  • Hedged requests send a second copy of a slow request to another replica and take whichever answers first, trading a little extra load for a much tighter tail.

Through replication:

  • Data availability. Multiple copies across servers or regions mean a slow replica does not block the request; it is served from a healthy one.
  • Load balancing. Spreading work across replicas keeps any single node from becoming the bottleneck that creates the tail.
  • Geographically distributed replication. Serving each user from the nearest region cuts the round-trip distance, and distance is often where the tail comes from.

Closer to the user:

  • Edge computing processes data nearer the client, shortening the distance data has to travel.
  • Caching keeps hot data in fast storage so the common request skips the slow back-end path entirely. See Caching Isn't Cheating.

Measure the tail, not the mean

You cannot defend what you do not watch. Put the SLO on the percentile users actually feel - p99 or p99.9 - and alert on it. An SLA written against an average will look healthy right up until the tail drives your users away.