Author(s): “The AI Engineer”

Originally published on Towards AI.

Your p99 Latency Belongs to Someone Else

One percent. That is how many calls to a dependency I made slow in a simulation, and it was enough to slow 63 out of every 100 requests that called that dependency 100 times.

The article argues that the p99 on your dashboard is effectively measuring a single “hop,” but the user experience is shaped by many hops—where the slowest component (often someone else’s service, like a database, vector store, shared GPU pool, or model provider) dominates end-to-end latency. After walking through how tail latency compounds differently under fan-out (maximum of parallel calls) versus chains (sum across sequential calls), it shows simulation results where fan-out turns a small slow-call probability into a large fraction of slow user requests, and where chains still surface slow calls repeatedly even when ratios can look healthier. It explains why the tail usually comes from rare, shared-resource contention outside your control, then presents hedged requests as a practical fix: send a second replica copy only after a delay near the slow percentile, cancel the loser, and thereby “square” the probability of slow outcomes—dramatically shrinking the tail for a modest extra load, with caveats when slowness is shared across replicas. It also provides guidance on measurement pitfalls (averaging percentiles, coordinated omission, and collapsing LLM timing into one number), and ends with a checklist for computing odds from your fan-out counts and dependency slow rates, validating metrics with histograms, and trying hedges on idempotent reads.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI