Author(s): Ahmad Bilal

Originally published on Towards AI.

LLM Inference: What Actually Happens Inside an LLM

A beginner-friendly walk through tokens, embeddings, attention, prefill, decode, and the KV cache, and why they decide how fast AI feels.

You type a question into ChatGPT, press Enter, and a moment later words start to appear. It feels as if the model read your question, thought for a second, and began to write.

But what happened during that second? Why does the first word often take longer to show up than the ones after it? Why does a long conversation use more GPU memory? And why can a server answer one person instantly but slow down when hundreds of people arrive at once?

To find out, we’ll follow a single request all the way through a large language model (LLM). We’ll use tiny, made-up numbers so you can check the math in your head, and we’ll introduce each technical term only after we’ve seen the problem it solves. By the end, you’ll know what prefill and decode mean, why the KV cache matters so much, and what inference engineers actually spend their days optimizing.

One note on scope: we’ll focus on the decoder-only transformer, the design behind GPT, Llama, Mistral, and most other text-generating LLMs. “Decoder-only” simply means the model has one job: continue a sequence of text. Real systems differ in the details, but the core flow is the same.

The journey at a glance

Our example prompt is:

What is the capital of France?

And the model’s answer is:

Paris.

Here is the whole trip in one picture:

Two kinds of components appear in this picture. The tokenizer is a separate piece of software that ships alongside the model. Everything inside the “trained neural network” section is made of numbers learned during training. Those learned numbers are called parameters (or weights). When you hear “a 7-billion-parameter model,” that’s how many of these numbers it contains.

Training is the process of adjusting those numbers using huge amounts of text.

Inference is using the finished numbers to respond to a new prompt. This article is entirely about inference.

Let’s open each box in the order our request reaches it.

1. Tokenization: the model sees numbers, not words

A transformer can’t work on raw text. Its calculations need numbers. So the first step is a tokenizer, which splits text into small pieces called tokens and gives each piece an integer ID.

Think of the tokenizer as a rulebook for splitting text plus a dictionary that maps each piece to a number. The full set of pieces it knows is called its vocabulary. A small slice of one might look like this (the IDs are made up):

Piece ID
"What" 42
" is" 815
" the" 279
" capital" 6864
" of" 315
" France" 9822
"?" 30

Notice that several pieces start with a space. Tokenizers often attach a space to the word that follows it, so " is" and "is" can be different tokens.

Our question becomes:

Text: What is the capital of France?
Pieces: [What][ is][ the][ capital][ of][ France][?]
IDs: [42] [815][279] [6864] [315][9822] [30]

A token isn’t always a word. Common words are usually one token, a rare name may be split into several pieces, and punctuation is often a token of its own. The exact split depends on the tokenizer.

Where does the vocabulary come from?

The vocabulary is built before the model is trained, usually with an algorithm called byte-pair encoding (BPE). BPE starts with individual characters (or raw bytes) and scans a large body of text for the pair of neighbors that appears most often, such as t followed by h. It merges that pair into a new token, th, and repeats, merging frequent pairs again and again until the vocabulary reaches a target size. Modern vocabularies typically hold between 100,000 and 250,000 tokens. The result is that common words become single tokens, while rare words can still be spelled out from smaller pieces.

Different LLMs use different tokenizers, and a model is permanently tied to the one it was trained with. You can’t swap in an unrelated tokenizer later: ID 42 might mean "What" in one vocabulary and something completely different in another, and the model would receive nonsense.

2. Embeddings: turning an ID into a vector

The model now has a list of IDs like 42 and 815. But an ID is only a label, like a locker number. ID 815 isn't "bigger" or "more meaningful" than ID 42, and doing arithmetic on IDs tells the model nothing about language.

So the next step is an embedding lookup. The model contains an embedding table: one row of learned numbers for every token in the vocabulary. A tiny version might look like this:

When the tokenizer outputs ID 815, the model simply fetches row 815. That list of numbers is called a vector, and the number of values in it is the vector's dimension.

" is" -> ID 815 -> row 815 -> [-0.10, 0.30, 0.80]

This answers a common question: the embedding layer does not read the word and invent a vector on the spot. It looks up numbers that were learned during training, when they were gradually adjusted to help the model make better predictions.

Real vectors are much longer than three values. Llama 3 8B, for example, uses vectors of 4,096 values and a vocabulary of about 128,000 tokens, so its embedding table is a grid of 128,000 × 4,096 numbers, more than half a billion parameters in this one table. A rectangular grid of numbers like this is called a matrix.

Does the vector hold the word’s meaning?

Partly. It holds features that proved useful during training, but it isn’t a dictionary definition. Take the word bank: it could mean a financial institution or the side of a river, and its lookup vector is the same either way. Working out which meaning fits the current sentence is the job of the transformer layers, which use the surrounding words to build a context-dependent representation.

What about word order?

“Dog bites man” and “Man bites dog” contain the same words but mean very different things, so the model needs to know where each token sits. A popular technique called RoPE (rotary positional embeddings) handles this by rotating certain vectors by an angle that depends on the token’s position. The details aren’t needed here. Just remember that the model tracks where each token appears, not only which tokens appear.

3. How Attention Works: Connecting Tokens Through Q, K, and V

We have converted our prompt into tokens and looked up their embedding vectors. Each token now has a numerical representation.

But there’s a problem.

Consider this sentence:

The customer visited the bank because it was nearby.

When the model encounters the word “it”, how does it know what “it” refers to?

The embedding of “it” alone cannot answer that question. The model needs information from the surrounding context. In a decoder-only language model, that means looking at the tokens that came before it.

This is where self-attention comes in.

3.1 The idea behind attention

Imagine you’re reading a sentence. You don’t give equal importance to every earlier word when trying to understand the current one.

For example, when interpreting “it” in our sentence, “bank” might be more relevant than “customer” or “visited”.

A transformer learns to make similar connections mathematically.

For every token, it calculates how strongly that token should attend to other available tokens and then combines information from them.

But how does it decide which tokens are relevant?

3.2 Meet Query, Key, and Value

Inside every transformer layer, each token’s vector is transformed into three different vectors:

• Query (Q): Represents what the current token is looking for.
• Key (K): Represents the features another token can be matched against.
• Value (V): Represents the information that token can contribute.

Think of it like searching a library.

You enter a search query. The library compares it against the catalog entries of different books. Once it identifies relevant books, it retrieves information from their contents.

The mapping is straightforward:

Library search -> Transformer attention

Your search -> requestQuery (Q)

Book catalog -> entryKey (K)

Book contents -> Value (V)

Relevance of each book -> Attention score

Combining information from relevant books -> Attention output

The important distinction is that Keys determine how strongly tokens match, while Values determine what information gets passed forward.

3.3 Where do Q, K, and V come from?

Here’s something important: Q, K, and V are not stored directly in the model.

What the trained model stores are three sets of learned weights:

• (W_Q) — Query projection weights
• (W_K) — Key projection weights
• (W_V) — Value projection weights

These are numerical matrices learned during training.

When a token enters a transformer layer, its current vector is multiplied by each matrix:

[ Q=XW_Q ]
[ K=XW_K ]
[ V=XW_V ]

The same input goes through three different transformations, producing three different representations.

For example, suppose a token enters the layer with this vector:

X = [0.5, 0.2]

The model applies its learned matrices and might produce:

Query = [1.0, 0.0]
Key = [0.3, 0.7]
Value = [2.0, 4.0]

These are illustrative numbers.

The model doesn’t decide which weight matrix to use for a particular word. Every token goes through the same Q, K, and V projection operations within that attention layer.

What changes is the input vector, so different tokens produce different Queries, Keys, and Values.

And because each transformer layer has its own learned matrices, the representations can change as information moves through the network.

3.4 How does a Query find relevant tokens?

Let’s work through a tiny example.

Suppose the current token has the following Query, and we have two candidate tokens:

Current Query = [1, 0]

Token A:
 Key = [1, 0]
 Value = [10, 0]
Token B:
 Key = [0, 1]
 Value = [0, 20]

Our goal is to determine how much information the current token should take from A and B.

Step 1: Calculate matching scores

The model compares the Query with each Key using a dot product.

A dot product multiplies corresponding numbers and adds the results.

For Token A:

Q · K_A = (1 × 1) + (0 × 0) = 1

For Token B:

Q · K_B = (1 × 0) + (0 × 1) = 0

So our attention scores are:

Token A → 1
Token B → 0

Token A has a stronger match.

But these scores aren’t probabilities yet.

Step 2: Convert scores into attention weights

The model uses a function called softmax to convert scores into positive weights that add up to 1.

For simplicity, let’s first apply softmax directly to our scores, without the scaling factor used in real attention.

Scores: [1, 0]

Softmax: [0.73, 0.27]

This means:

• Token A contributes approximately 73%.
• Token B contributes approximately 27%.

Notice that Token B isn’t completely ignored. Attention usually blends information from multiple tokens rather than selecting just one.

Step 3: Combine the Values

Now comes the interesting part.

We used Keys to calculate relevance, but we use Values to collect information.

The model multiplies each Value by its attention weight:

Token A: 0.73 × [10, 0] = [7.3, 0]
Token B: 0.27 × [0, 20] = [0, 5.4]

Then it adds them together:

Attention Output = [7.3, 5.4]

That’s the result of our simplified attention calculation.

The current token now has an attention output that incorporates information from other tokens, with more influence from the stronger match.

One subtle point: the model hasn’t literally identified a grammatical relationship or stored the meaning of the sentence in these two numbers. It has performed a learned numerical transformation that can help later layers make better predictions.

3.5 How does this work in a real transformer?

Our example intentionally simplified a few things.

In a real decoder-only transformer:

First, the vectors are much larger. Instead of two numbers, Q, K, and V may each contain dozens or hundreds of numbers per attention head.

Second, attention scores are scaled. Before softmax, the model divides the dot-product scores by the square root of the Key dimension:

[ Attention Scores=frac{QK^T}/{sqrt{d_k}} ]

This helps keep the score magnitudes manageable.

Third, future tokens are masked. During causal self-attention, a token can attend to itself and earlier tokens, but not future tokens. This is essential for next-token prediction.

Fourth, the attention output isn’t the end of the transformer layer. It passes through an output projection and residual connection, followed by additional normalization and feed-forward computations. These allow the model to transform and refine the information further.

3.6 Why do transformers have multiple attention heads?

So far, we’ve calculated attention once using one set of Q, K, and V projections.

That’s called single-head attention.

But a sentence can contain many kinds of relationships.

Consider:

The customer visited the bank because it was nearby.

One attention head might learn patterns related to nearby words. Another might learn connections between nouns and pronouns. Another might focus on different contextual features.

These are possible learned behaviors, not roles assigned manually to particular heads.

Instead of relying on a single attention calculation, transformers use multi-head attention.

For example, imagine a model with four heads:

Token representations
 |
 +--------------+--------------+
 | | |
 Head 1 Head 2 Head 3 Head 4
 Q1 K1 V1 Q2 K2 V2 Q3 K3 V3 Q4 K4 V4
 | | | |
 Attention Attention Attention Attention
 | | | |
 +--------------+--------------+------------+
 |
 Concatenate
 |
 Output projection
 |
 Combined result

Each head has its own learned Q, K, and V projections, so it can calculate a different attention pattern over the same input tokens.

The outputs are concatenated and passed through another learned weight matrix to combine them.

In a typical implementation, these per-head weights are stored efficiently as slices of larger matrices rather than as entirely separate objects.

3.7 The connection to KV caching

Now we’ve reached an important observation.

To calculate attention, the current token needs its Query and the Keys and Values of the tokens it is allowed to attend to.

But during text generation, the model processes an ever-growing sequence.

Suppose our prompt is:

The customer visited the bank

After processing this prompt, the model has already calculated Keys and Values for those tokens in every transformer layer.

When it generates another token, should it recalculate all those earlier Keys and Values?

No. Those calculations can be reused.

The learned model weights haven’t changed, and the previously processed tokens haven’t changed. In a causal transformer, their cached representations remain valid as new tokens are appended.

So the model stores the calculated Keys and Values in memory.

This is called the KV cache.

During subsequent decoding steps, the model calculates Q, K, and V for the newly processed token, appends its K and V to the cache, and uses its Query to attend over the available cached Keys and Values.

That’s the foundation of efficient autoregressive inference.

In the next section, we’ll see how this works during prefill and decode, and why KV caching dramatically reduces repeated computation during token generation.

Why can’t a token look ahead?

A text-generating model is trained to predict the next token, so it must never see the answer in advance. Each position may attend only to itself and earlier positions, never to later ones. This rule is called causal masking. Keep it in mind: it will turn out to be the reason the KV cache works.

The rest of a transformer layer

Attention is only half of each layer. The other half is a feed-forward network, often called an MLP (multi-layer perceptron). It applies further learned calculations to each token’s vector on its own, refining what attention gathered. A full model stacks dozens of these attention-plus-MLP layers; Llama 3 8B has 32.

We’re now ready for the two phases of inference.

4. Prefill: reading the whole prompt at once

Our prompt has seven tokens. Prefill is the phase in which the model processes all of them and produces the first token of the answer.

Here’s what happens, step by step:

1. Look up vectors. The embedding table returns one vector for each of the seven token IDs.
2. Run the first layer. The layer computes Q, K, and V for all seven positions, and each position attends to itself and the positions before it.
3. Run every remaining layer. Each layer takes the previous layer’s output and refines it. The layers run one after another, but within a layer the GPU processes all seven positions at the same time.
4. Save the Keys and Values. In every layer, the model stores the K and V vectors it computed for the prompt. Together, these stored vectors are called the KV cache.
5. Score the next token. The final vector at the last position goes through the output layer, which produces one raw score, called a logit, for every token in the vocabulary. Softmax turns those scores into probabilities, and the model picks a token: Paris.

[What][ is][ the][ capital][ of][ France][?]
 |
 All transformer layers, 7 positions
 processed in parallel
 |
 K and V saved for all 7 positions
 |
 First output token: [Paris]

What exactly is in the KV cache? Not the original words and not a summary of the prompt. It holds the Key and Value vectors for every earlier position, in every layer. That’s why it grows with the length of the conversation and takes up GPU memory.

Why prefill is usually compute-bound

A GPU (graphics processing unit) is a processor with thousands of small cores built to perform enormous numbers of multiplications in parallel. Prefill hands it large, regular chunks of work: many prompt positions multiplied by the same weight matrices at once. That keeps the GPU’s cores busy, so the limiting factor is usually how fast it can do math. This is called being compute-bound. It’s a common pattern rather than a strict rule, but it holds for most workloads.

How the token is actually chosen

The model doesn’t output a single answer. It outputs a probability for every token in its vocabulary. Picking one from that list is called sampling, and there are a few common strategies:

1. Greedy Decoding

• Always selects the token with the highest probability.
• Produces predictable, deterministic output.
• Example: Paris (80%), Lyon (15%), Rome (5%) → Always selects Paris.

Limitation: Can produce repetitive text and doesn’t always lead to the best overall response.

2. Temperature

• Controls how random or predictable token selection is by adjusting the probability distribution before sampling.
• Low temperature (0.1–0.3): Favors high-probability tokens, producing more predictable output.
• Temperature = 1: Keeps the original probability distribution unchanged.
• High temperature (1.5–2.0): Makes probabilities more evenly distributed, increasing randomness and diversity.
• Note: Temperature doesn’t change the model’s knowledge or weights; it only affects token selection.

3. Top-p Sampling (Nucleus Sampling)

• Selects the smallest group of highest-probability tokens whose combined probability reaches or exceeds the specified threshold p.
• Example: With top_p = 0.9:
• Paris: 80%
• Lyon: 10%
• Rome: 5%
• Berlin: 5%
• Only Paris and Lyon are considered because their combined probability reaches 90%.
• The remaining probabilities are normalized, and the next token is randomly sampled from the selected group.
• Benefit: Dynamically limits unlikely tokens while maintaining diversity.

Key Difference:

• Greedy: Always picks the most likely token.
• Temperature: Adjusts how strongly the model favors likely tokens.
• Top-p: Controls which tokens are eligible for random selection.

In practice: Temperature and top-p are commonly combined. For example, temperature = 0.7 and top_p = 0.9 provide a reasonable starting point for balanced text generation.

5. Decode: writing the answer one token at a time

Prefill has picked Paris. Now the model needs the next token, which will probably be ..

The naive approach would be to feed the entire prompt plus Paris back through the model from scratch. That would repeat almost all of the work prefill just did. The KV cache avoids this.

Here is one decode step:

1. Take the newest token. That’s Paris, which is looked up in the embedding table.
2. Process only that position. In each layer, the model computes Q, K, and V for Paris alone.
3. Attend using the cache. The new Query is compared with all the cached Keys plus the new token’s own Key, and the matching Values are blended together.
4. Extend the cache. The new K and V are appended to the cache in every layer. The old entries are reused, not recomputed.
5. Pick the next token. The output layer scores the vocabulary, and sampling picks ..

Then the cycle repeats with . as the newest token. Because each new token depends on the tokens generated before it, this style of generation is called autoregressive.

After prefill:
 Cache: [What][ is][ the][ capital][ of][ France][?]
 Picked: [Paris]
Decode step 1:
 Process [Paris], attend over the cache
 Append [Paris] to cache, pick [.]
Decode step 2:
 Process [.], attend over the larger cache
 Append [.] to cache, pick [end-of-sequence]

Why reusing old Keys and Values is safe

Here’s where causal masking pays off. Because no token is allowed to look ahead, each position’s vectors, in every layer, depend only on itself and the tokens before it. When Paris arrives, nothing about the earlier tokens' Keys and Values changes, so they can be kept as they are. If tokens could look forward, every new token would change every earlier vector and the whole cache would have to be rebuilt at each step. Causal masking is what makes the KV cache possible.

Two details that often cause confusion

Attention happens in both phases. Prefill computes attention for all prompt positions; decode computes it for one new position at a time. The difference is that decode reuses the earlier Keys and Values instead of recomputing them.

Choosing a token and processing it are separate moments. At the end of prefill, Paris has been chosen, but its own K and V aren't in the cache yet. They're added during the next decode step, when Paris is processed as input.

When does it stop?

Generation ends when the model picks a special end-of-sequence token, when it hits a maximum length set by the application, or when it produces a configured stop phrase. Because each token can be turned back into text as soon as it’s chosen, the interface can display the answer as it’s being written. That’s why responses appear to stream in word by word (strictly, token by token).

6. A small experiment you can run

The code below is a toy, not a real transformer, but it uses real arithmetic. It shows two things: an embedding is just a row lookup, and attention computed with a KV cache gives exactly the same answer as recomputing everything from scratch. It needs only Python and NumPy.

import numpy as np

# 1. Embedding lookup: an ID simply selects a row.
vocab = {"What": 0, " is": 1, " AI": 2}
rng = np.random.default_rng(0)
d = 4 # vector length
embeddings = rng.normal(size=(len(vocab), d)) # pretend these were learned
ids = [vocab[t] for t in ["What", " is", " AI"]]
x = embeddings[ids] # row lookup, nothing more
print("IDs:", ids)

# 2. Learned weight matrices that turn a vector into Q, K, and V.
Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))
def attend(q, K, V):
 scores = K @ q / np.sqrt(d) # dot products, scaled
 w = np.exp(scores - scores.max())
 w /= w.sum() # softmax: weights sum to 1
 return w @ V # weighted mix of Values

# 3. No cache: recompute K and V for every token from scratch.
no_cache = attend(x[2] @ Wq, x @ Wk, x @ Wv)

# 4. With cache: prefill stored K/V for the first two tokens.
K_cache, V_cache = x[:2] @ Wk, x[:2] @ Wv
# Decode computes Q/K/V for the new token only, then appends.
K_cache = np.vstack([K_cache, x[2] @ Wk])
V_cache = np.vstack([V_cache, x[2] @ Wv])
with_cache = attend(x[2] @ Wq, K_cache, V_cache)
print("Same result:", np.allclose(no_cache, with_cache))
print("Cached K rows:", K_cache.shape[0])

Output:

IDs: [0, 1, 2]
Same result: True
Cached K rows: 3

This toy has a single layer, where the match is easy to see. In a real model with dozens of layers, the results still match exactly, and the reason is causal masking: earlier tokens’ vectors never depend on later ones.

7. Why prefill and decode stress the GPU differently

To understand inference speed, you need two hardware ideas:

• Compute: how many calculations the GPU can perform per second.
• Memory bandwidth: how fast the GPU can move data, such as model weights and the KV cache, from its memory to its cores. A GPU’s main memory is called HBM (high-bandwidth memory), and its speed is measured in terabytes per second (TB/s).

During prefill, many tokens share each pass through the weights, so the GPU’s math units stay busy. During decode, a single request produces just one token per step, yet the GPU still has to read every weight in the model to produce it, plus the growing KV cache. The cores spend much of their time waiting for data to arrive. This is called being memory-bandwidth-bound.

A quick calculation

A 7-billion-parameter model stored in FP16 (16-bit numbers, 2 bytes each) has about 14 GB of weights. A high-end GPU like the NVIDIA H100 can read memory at roughly 3.35 TB/s. Reading all 14 GB takes at least:

14 GB / 3,350 GB per second ≈ 4.2 ms per decode step
 ≈ 240 tokens per second, at most

That’s an upper limit for a single user before any other overhead, and real systems land below it. It also shows why batching, processing many users’ requests together, helps so much during decode: one read of the weights can produce a token for every request in the batch. The number of requests processed together is called the batch size.

How big is the KV cache?

The cache size per token follows a simple formula:

KV cache per token =
 2 (one K, one V) × layers × KV heads × head size × bytes per number

For a Llama-2–7B-style model in FP16, with 32 layers, 32 heads, and 128 values per head:

2 × 32 × 32 × 128 × 2 bytes ≈ 0.5 MB per token

So a 4,000-token conversation needs about 2 GB of cache, for a single user. Serve dozens of long conversations and the cache can rival the size of the model itself. This is why long chats use more GPU memory, and why memory, not compute, often limits how many users a server can handle at once.

Newer models shrink this with grouped-query attention (GQA), where several Query heads share one set of Keys and Values. Llama 3 8B has 32 Query heads but only 8 KV heads, which cuts the cache to about 128 KB per token, a quarter of the size.

The metrics engineers actually watch

Imagine a request with 1,000 input tokens and a 200-token answer. Suppose prefill takes 0.4 seconds and the remaining tokens arrive at 50 per second (invented numbers, not a benchmark):

Time to first token (TTFT) ≈ 0.4 s
Time per output token (TPOT) = 1 / 50 = 20 ms
Remaining tokens = 199
Time for remaining tokens ≈ 199 × 0.02 = 3.98 s
Total time ≈ 4.38 s

• TTFT (time to first token) is how long the user waits before anything appears. In practice it includes not just prefill but also time spent waiting in the server’s queue, tokenization, and the network.
• TPOT (time per output token) is the average gap between tokens once the answer starts streaming. Some tools call this ITL (inter-token latency).
• Latency is the total time a single request takes.
• Throughput is how much work the whole server completes per second, often measured as output tokens per second across all users.

These two goals pull against each other. Bigger batches raise throughput but can make each user wait longer. The real goal in production is usually the highest throughput that still keeps latency acceptable.

One more practical point: in a chat, each new message usually sends the entire conversation history back to the model, which then has to prefill all of it again. This is why TTFT tends to creep up as a conversation gets longer, unless the server uses prefix caching (covered next).

8. The optimization toolbox

With the workload in view, the common optimizations stop sounding like buzzwords. Each one targets a specific bottleneck.

Continuous batching. Older servers grouped requests into a fixed batch and waited for the slowest one to finish before starting the next group. Continuous batching lets new requests join and finished ones leave at every decode step, so the GPU never sits idle waiting on a straggler.

PagedAttention. Because nobody knows in advance how long an answer will be, reserving one big block of memory per request wastes a lot of space. PagedAttention, introduced by the vLLM project, stores the KV cache in small fixed-size blocks that can sit anywhere in memory, much like how an operating system manages memory in pages. Less waste means more users fit on one GPU.

Prefix caching. If many requests start with the same text, such as a long system prompt (the hidden instructions an application sends before the user’s message), the server can compute that prefix’s KV cache once and reuse it, skipping most of the prefill work.

Chunked prefill. A very long prompt can hog the GPU and stall everyone else’s decoding. Chunked prefill splits the prompt into pieces and interleaves them with other users’ decode steps, keeping streaming smooth for everyone.

FlashAttention. Standard attention writes a large table of scores to the GPU’s main memory and reads it back. FlashAttention computes attention in small tiles that fit in the GPU’s much faster on-chip memory, cutting memory traffic and speeding things up without changing the result.

Quantization. This stores weights (and sometimes the KV cache) with fewer bits per number. A 7-billion-parameter model needs about 14 GB in FP16, about 7 GB in INT8 (8-bit integers), and about 3.5 GB in INT4. Smaller weights mean less data to read on every decode step, which helps directly with the bandwidth bottleneck. The trade-off is a possible small drop in quality, and actual memory use is always higher than the weights alone because of the KV cache and other overhead.

Speculative decoding. A small, fast draft model guesses the next several tokens, and the large model checks all of the guesses in a single pass, which costs about the same as generating one token. Every guess the large model agrees with is a token gained for free. With the standard acceptance method, the final output follows exactly the same probabilities as the large model running alone, so speed improves without changing quality. Note that the large model isn’t fact-checking the draft; it’s checking whether each guessed token is one it would have produced itself.

Tensor parallelism and pipeline parallelism. When a model is too big for one GPU, it has to be split. Tensor parallelism divides the work within each layer across several GPUs, which cooperate on every step. Pipeline parallelism gives each GPU a different group of whole layers, so data flows from one GPU to the next like an assembly line.

Disaggregated inference. Since prefill is compute-bound and decode is bandwidth-bound, some systems run them on separate pools of GPUs, each tuned for its phase. The cost is shipping the KV cache from the prefill machines to the decode machines.

No single technique wins everywhere. The right mix depends on model size, prompt and answer lengths, hardware, number of users, and how much latency is acceptable.

9. Sidebar: these aren’t the embeddings your RAG app uses

If you’ve built a RAG (retrieval-augmented generation) app, one that searches your own documents and passes the relevant ones to an LLM, you’ve probably used an “embedding model” to store documents in a vector database. That’s a different thing from the embedding table we discussed.

The LLM’s internal embedding table maps individual token IDs to vectors used inside the transformer. A RAG embedding model is a separate model that turns a whole chunk of text into one vector so that similar passages can be found by comparing vectors. The retrieved documents are then handed to the LLM as plain text, which the LLM tokenizes and embeds in its own way.

This explains two practical rules. You can switch to a different LLM without rebuilding your document index, because the LLM only ever receives text. But if you switch to a different RAG embedding model, you do have to rebuild the index, because vectors from two different embedding models live in different embedding spaces (each model’s own coordinate system) and can’t be meaningfully compared.

10. Thinking like an inference engineer

Knowing the flow is the foundation. The engineering challenge is making it fast, affordable, and reliable for real users. The two phases give you a map for diagnosing problems.

Say you run a customer-support assistant and users complain that responses take too long to start. That’s a TTFT problem, so before adding GPUs, ask: Are prompts unnecessarily long? Are requests waiting in a queue? Is prefill itself slow? Could a shared system prompt benefit from prefix caching?

Now say the first token arrives quickly but the rest of the answer crawls. That points to decode: model size, memory bandwidth, KV cache pressure, and batching.

Finally, say each request is fast on its own but performance collapses at 100 simultaneous users. Now you care about scheduling, GPU utilization, total throughput, and latency under load, not single-request benchmarks.

A good first hands-on experiment is to run a small open-weight model (one whose parameters are publicly downloadable, such as Llama or Mistral) with vLLM, a popular open-source inference server. Compare short prompts with long ones, short answers with long ones, and one user with many. Record TTFT, output tokens per second, and GPU memory use, and try to predict which phase each change will affect before you measure. That’s how these concepts become engineering intuition.

11. The whole story, in two paragraphs

When you send a prompt, the tokenizer turns your text into token IDs, and the model looks up an embedding vector for each one. During prefill, the transformer layers process all prompt positions in parallel, compute attention, and save each position’s Keys and Values in the KV cache. The output layer scores every possible next token, and sampling picks the first word of the answer.

During decode, the model processes that newly chosen token, computes its Query, Key, and Value, attends over the cached Keys and Values, adds its own to the cache, and picks the next token. Causal masking guarantees the old cache entries never need updating. This repeats until the model emits an end-of-sequence token or hits a limit, and the token IDs are turned back into text as they arrive.

If you can explain why prefill handles many tokens at once, why decode moves one token at a time, and why the KV cache lets decode skip repeated work, you have the mental model behind nearly every inference optimization out there.

Glossary

Autoregressive: generating one token at a time, with each new token depending on the ones before it.

Batching: processing several users’ requests together so one pass over the model’s weights serves all of them.

Causal masking: the rule that a token can attend only to itself and earlier tokens, never later ones.

Compute-bound: limited by how fast the GPU can do math.

Decode: the phase that processes each newly generated token to predict the next one.

Embedding table: a matrix of learned numbers with one row (vector) per token ID.

KV cache: stored Key and Value vectors for every earlier token, in every layer, reused during decode.

Logit: the raw score the model assigns to a possible next token before softmax.

Memory-bandwidth-bound: limited by how fast data can be moved from GPU memory to its cores.

Prefill: the phase that processes the whole prompt and picks the first output token.

Query / Key / Value: three vectors computed for each token. Queries are matched against Keys to decide how much of each Value to mix in.

Softmax: a function that turns a list of scores into positive weights that add up to 1.

Throughput: total work a server completes per second, usually measured in tokens.

Tokenizer: software that splits text into tokens and maps each one to an integer ID.

TTFT / TPOT: time to first token, and average time per output token after that.

Further reading

• Vaswani et al., “Attention Is All You Need” (2017), the paper that introduced the transformer.
• Jay Alammar, “The Illustrated Transformer”, a visual walkthrough of attention.
• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (2023), the vLLM paper.
• Dao et al., “FlashAttention” (2022), on faster, memory-efficient attention.
• Leviathan et al., “Fast Inference from Transformers via Speculative Decoding” (2023).

All token IDs, vectors, scores, and timings in the examples are illustrative. Real values depend on the model, tokenizer, hardware, and inference engine.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI