Caching

Where it shows up in real systems

• CDN

  Images, scripts, styles and videos are cached at the edge, so most requests never reach your servers. Longer TTLs raise the hit ratio but make content updates slower to appear; an origin shield collapses cache misses from all edges into one.

  In production: Amazon CloudFront, Cloudflare, Akamai, Fastly, Azure Front Door

• In-memory cache

  The hit ratio depends on how much of the working set fits in memory, the TTL and the eviction policy. If the cache goes down or is flushed, every read falls on the database at once: plan for that stampede. Write-behind is fast but loses writes if a node dies without a replica.

  In production: Redis (Amazon ElastiCache, Azure Cache), Valkey, Memcached, Dragonfly

• In-process cache

  No network hop, so it is the fastest cache possible, but each instance holds its own copy: memory is duplicated and invalidation is hard, so instances can disagree for up to the TTL.

  In production: Caffeine (Java), Guava cache, lru-cache (Node.js), functools.lru_cache

• Model inference service

  Each request takes tens to hundreds of milliseconds of GPU time, so a few hundred requests per second already need several expensive GPUs. Batching requests, caching answers and choosing the right accelerator change the bill by an order of magnitude.

  In production: NVIDIA Triton, vLLM, Amazon SageMaker endpoints, Google Vertex AI, KServe

• Third-party API

  Its latency, rate limits and outages become yours. Cache what you can, set short timeouts, wrap it in a circuit breaker and design a degraded mode for when it is down.

  In production: Google Maps Platform, OpenWeather, Shippo, Anthropic API, OpenAI API