baam's wiki

Caching Isn't Cheating

Evergreen

Mar 8, 2025 4 min read

Caching gets dismissed as a hack - the thing you bolt on when you couldn’t be bothered to design the system properly. I think that is exactly backwards. Caching is one of the oldest and most principled ideas in computing: you spend a little space, and a little tolerance for being out of date, to buy back time. And it is not something you sprinkle on at the end. The whole memory hierarchy - CPU registers, L1/L2/L3, RAM, disk, the CDN sitting at the edge of the network - is caching, all the way down. You are already doing it everywhere, whether you named it or not.

What you’re actually trading

Every cache is the same bet placed at a different scale: that something you paid for once will be asked for again, soon enough to be worth holding onto. You spend memory, or some faster tier of storage, so you don’t have to recompute or re-fetch. In exchange you accept that the copy you kept might have gone stale - the source moved on and your cache hasn’t noticed yet.

And it only pays off when there is locality. The same items have to get asked for again soon (temporal locality), or things sitting near each other have to get asked for together (spatial locality). Take that away and caching earns you nothing - you’re just keeping a second copy of data nobody wanted twice, and now you have two of them to keep honest.

Seen this way a cache is less a data structure than a wager that the near future will rhyme with the recent past.

It’s a hierarchy, not a bolt-on

The reason “add a cache” is the wrong mental image is that caching was never one box. It’s a chain of layers, and each one quietly catches what the layer below it would have charged full price for. A CPU cache saves a trip to RAM. An in-memory cache - Redis, or just a map in your process - saves a trip to the database. A CDN saves a trip across an ocean.

The numbers are the whole argument. An L1 hit is on the order of a nanosecond; main memory is a hundred times slower than that; a cross-region round trip is a hundred million times slower. Caching is just the discipline of moving a request as far up that ladder as you can get away with.

The hard part is invalidation

Here is where it stops being free. Keeping a copy adds a question that storage alone never had: not where the copy lives, but when it stops matching the source. That gap between the cache and the truth is the real cost, and managing it is the part that earns caching its reputation for being hard.

The common strategies are just different answers to one question: how stale is this data allowed to be?

  • TTL. Hold the copy for a fixed window, then drop it and reload. Predictable and simple, at the price of serving data up to one window out of date.
  • Write-through. Every write updates the cache and the source together, so the copy is never stale. The cost lands on every write, including data that is never read again.
  • Cache-aside. Load on a miss, drop the entry on a write. The usual default: it sits between the other two and tolerates a brief window where cache and source disagree.

Each one buys freshness with either latency or staleness. Picking a strategy is really picking a staleness budget - and the value in doing it deliberately is naming that budget up front, instead of discovering it in production.

When caching hurts

A cache in the wrong place does not just fail to help. It adds a failure mode the system did not have before.

  • Thundering herd. A hot key expires and every request that wanted it stampedes the origin at the same instant. This is how a cache meant to protect your database becomes the thing that knocks it over.
  • Stale reads that matter. A cached balance, a revoked permission, a price that changed a minute ago. Here a slightly slow answer that’s correct beats a fast answer that’s wrong, and caching trades in exactly the wrong direction.
  • No locality. Already said, but it bears repeating: caching data nobody asks for twice buys you overhead and a second source of truth, and nothing else.

The bottom line

So, not cheating. Not a shortcut, and not the thing you do because you were lazy. Caching is a deliberate trade - space and freshness for time - and like every trade in Latency vs Throughput vs IOPS: why your fast API still fails at scale, the skill is knowing when the exchange rate is actually in your favor, and being honest with yourself about the staleness you just agreed to.