Tail latency

The p99 is the time under which 99 % of requests finish. When one page calls many services, it almost always meets one slow call, so the tail, not the average, decides user experience.

Where it shows up in real systems

  • Distributed tracing

    Shows which hop is slow in a chain of microservices. Sampling keeps the cost down while still catching tail latency.

    In production: OpenTelemetry, Jaeger, Grafana Tempo, AWS X-Ray, Honeycomb

  • Model inference service

    Each request takes tens to hundreds of milliseconds of GPU time, so a few hundred requests per second already need several expensive GPUs. Batching requests, caching answers and choosing the right accelerator change the bill by an order of magnitude.

    In production: NVIDIA Triton, vLLM, Amazon SageMaker endpoints, Google Vertex AI, KServe

See it break, then fix it

Build an architecture, send it a day of traffic and outages, and learn why it holds or fails. Free to start.

Open the learning path Open the sandbox