Author(s): Veera RS

Originally published on Towards AI.

Tokenization, routing, continuous batching, the KV cache, and why prefill and decode hit completely different hardware limits

Training a frontier model can cost hundreds of millions of dollars and take months. But once that model is produced, it’s just a huge set of numbers sitting idly in memory. A model can’t do anything on its own. So what happens behind the scenes to bring that model to life?

After the lead, the article explains how inference differs from training, why serving (inference) is repeated and often more expensive, and what inference engines do—scheduling requests, managing GPU resources, and deciding how to process work together. It walks through the end-to-end path of a user request: authentication and routing in the data center, prompt assembly (including system instructions, history, retrieved context, and tool definitions), tokenization, and batching strategies like continuous batching to keep GPUs efficient while balancing throughput vs. latency. It then details the two core inference phases—prefill (reading the full prompt and building the KV cache, typically compute-bound) and decode (generating tokens one at a time while repeatedly using the KV cache, typically memory-bandwidth-bound)—and how latency is measured via TTFT and inter-token latency (ITL). The piece concludes by framing the whole system as the machinery that turns static model weights into usable real-time responses.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI