Caching 101: Latency, Layers, and Eviction

A cache hit can avoid storage I/O, a network hop, a database query, or repeated computation. Which saving matters depends on the request: browser caching may remove the request entirely, a CDN may protect the origin, and a database buffer pool may make an app-level cache redundant. Measure before and after at the same percentile and workload.

Cache data that is reused and can tolerate a copy. Avoid shared caching for secrets and highly personal data unless keys, authorization, encryption, and expiry are designed together—a shared Redis key is not an access-control boundary.

Speed is a hierarchy, not a universal table. CPU caches are closest to execution, RAM avoids storage access, local storage avoids a remote hop, and a network service adds transport plus its own processing. Hardware, queueing, payload size, connection reuse, and cache warmth can move every observed number.

Use the hierarchy to form a hypothesis, then measure the actual path. If a product endpoint spends most of its time rendering or calling a second service, replacing a database read with Redis may barely change user latency. Pick the layer that removes the measured bottleneck, not the layer with the most impressive benchmark.

Caches stack from the browser inward. Browser holds static assets. CDN / edge (Cloudflare, Akamai) serves near the user. Application caches API or computed results. Database buffer/query caches live inside the engine. In-memory stores like Redis or Memcached sit beside the app. OS page cache speeds disk. CPU L1/L2/L3 is the innermost hardware layer.

Most product teams deliberately own browser headers, CDN rules, and Redis. OS and CPU caches still matter — they explain why warm boxes feel faster — but you configure them less often.

Trace one product request: browser and CDN miss, the application checks product:42, and Redis returns either a value or no value. On a miss, the app reads the database, handles a missing product as 404, attempts to populate Redis, and returns the database result. If Redis is unavailable, the database remains the fallback; if the database fails, the app returns a deliberate service error rather than caching an error page.

Use this path for reused catalog data. Be careful with inventory, money, permissions, and private responses: define freshness and authorization first, then choose a cache. A short TTL reduces the stale window but does not make a wrong cache key safe.

Caches are finite. LRU drops what has not been touched longest. LFU drops lowest access counts. FIFO drops oldest inserts. TTL expires entries by clock regardless of popularity. Random is simple and sometimes good enough under uniform access.

Many systems combine TTL with LRU: expire stale data and, under pressure, evict cold keys. Wrong policy looks like thrashing — hot keys bouncing in and out — or stale pages living forever.

Start from the request path: browser → edge → application → shared cache → database. At each step ask whether the response is reusable, who may see it, how stale it may become, and whether the next layer can absorb misses. Stop adding layers when the measured bottleneck moves elsewhere.

Common candidates include versioned static assets, public API responses, media, search results, catalogs, carefully scoped sessions, inference outputs, and templates. Prices, permissions, and personalized pages need stronger invalidation and key isolation than a public image.