baam's wiki

Rate limiting as an invariant

Evergreen

Jul 30, 2025 3 min read

A rate limit looks like a restriction from the outside - the 429 Too Many Requests standing between a client and what it wants. From inside the system it is the opposite: a rate limit is an invariant you enforce so the system stays in a regime where it actually works. It is the standing promise “no matter what the outside world does, we will not accept more than X per second,” and holding that line is what keeps a service alive under a load it was never sized for.

Why enforce one

  • Protection. A system that accepts unlimited load will eventually meet a load that topples it. A limit caps the blast radius.
  • Fairness. With no limit, the loudest client starves everyone else. A per-client limit shares a finite resource deliberately instead of by accident.
  • Cost control. Unbounded usage is an unbounded bill, especially when each request fans out to a metered downstream.

The common algorithms

Each answers “have we exceeded the rate?” with a different trade of accuracy against bookkeeping:

  • Token bucket - tokens refill at a steady rate, each request spends one, and bursts are allowed up to the bucket size. The usual default: simple and burst-friendly.
  • Leaky bucket - requests drain at a fixed rate regardless of how they arrive, smoothing bursts into a steady stream.
  • Sliding window - counts requests over a moving window; more accurate at the edges than a fixed window, for a little more state.
Three hand-drawn panels. Token bucket: tokens refill into an open bucket and each request spends one, so short bursts are allowed. Leaky bucket: bursty arrivals pour in the top and drain at a single steady rate out a hole in the bottom. Sliding window: request ticks along a timeline with a rectangular window that slides along it, counting at most N.
The three side by side: the token bucket lets bursts through, the leaky bucket smooths them into a steady drain, and the sliding window counts arrivals inside a frame that moves with time.

Where it lives

A limit can sit at the gateway, before requests reach your code - the cheapest place, and it shields the whole service - or in the application, where it can be finer-grained (per user, per endpoint) but the request has already cost you something to get there. Most real systems use both: a coarse gateway limit as a shield, finer app-level limits for fairness.

The client’s half of the contract

Rate limiting is a contract, and the client has to hold up its end. A well-behaved client treats a 429 not as an error but as an instruction: back off - ideally with exponential backoff and jitter - and retry later. That is the same discipline as Retries and idempotency: a retry is only safe when the operation can absorb being run again.

The bottom line

A rate limit is not a punishment; it is the invariant that lets a system make a promise about its own behavior. Choose the number deliberately, enforce it at the right layer, and expect your clients to respect it.