Author(s): Chew Loong Nian – AI ENGINEER

Originally published on Towards AI.

Why Self-Hosted GLM-5.3 Needs 8 GPUs, Not 4, for Long Coding Sessions

​

Revolut’s David Abitbol says GLM-5.3 fits on 4 GPUs but runs short of memory. The reason is in vLLM’s source: every GPU keeps a full copy of each chat’s memory, the KV cache.

If you self-host GLM-5.3 with vLLM for long coding sessions, give each copy of the model 8 B200 GPUs, not 4, even though it fits on 4. On paper, 4 GPUs leave room for 3.0 full 256K-token sessions per copy; 8 GPUs leave room for 7.6. This article shows why and gives you a 69-line script to check your own setup before you rent anything.

01 — Where this comes from

On October 29 the AI Engineer YouTube channel posted “How Revolut Cut AI Coding Costs 90% With Self-Hosted Open Models”, a 12-minute talk by David Abitbol of Revolut.

Paraphrasing his points from the auto-generated captions: serving about 2,000 engineers takes roughly 56 B200s for GLM-5.3. Keep time to first token under 3 seconds. Long context is a scam; 256K is what you need. Use NVFP4 on Blackwell GPUs, about 40% smaller than FP8. And although GLM fits on 4 GPUs (TP4), you don’t get enough KV cache, so 8 (TP8) may work better.

He gave the rules but not the math. Here it is.

02 — The short answer: the memory is copied

The KV cache is the model’s memory of each conversation. GLM-5.3 uses DeepSeek-style latent attention, which stores a small cache per token.

When you split a model across 4 GPUs (tensor parallelism), the weights get split. In vLLM 0.32.0 this cache does not. The attention layer declares its cache with max_tp_shards=1, and the field's docstring in kv_cache_interface.py says GPUs beyond that "hold replicas". So each GPU keeps a full copy of every conversation's cache.

Going from 4 GPUs to 8 halves the weights on each GPU. All the memory that frees up becomes cache.

03 — How big is the cache per token?

Two numbers come from the vLLM 0.32.0 source:

• 576 bytes per token per layer for the main cache. The NVFP4 checkpoint declares an 8-bit float KV cache (and vLLM also defaults GLM-5.3 to fp8 on B200s). For an fp8 cache, vLLM’s backend list puts FlashInfer’s sparse kernel first, and it stores 576 one-byte values.
• 132 bytes per token for the sparse-attention “indexer” cache, which only some layers have.

vLLM builds an indexer on a layer when max(layer - offset + 1, 0) % freq == 0. With GLM-5.3's freq of 4 and offset of 3, that's 21 of 78 layers. The config's own indexer_types list also marks 21 layers as full.

04 — The script​

glmfit.py reads the model's config.json and model.safetensors.index.json. It uses only Python's standard library and needs no GPU.

#!/usr/bin/env python3
"""How much KV cache does GLM-5.3 get on B200s at TP4 vs TP8?
usage: glmfit.py config.json model.safetensors.index.json [other.index.json]
Byte constants are read from vLLM 0.32.0 source (see comments)."""
import json, os, sys
# MLA row: 576 = fp8_e4m3, FlashInfer sparse (B200 default for an fp8 cache)
# 656 = fp8_ds_mla, --attention-backend FLASHMLA_SPARSE
IDX = 128 + 128 // 128 * 4 # Indexer k_cache: head_dim + scales
env = os.environ.get
MLA = int(env("MLA", 576))
HBM, UTIL = float(env("HBM_GB", 180)), float(env("UTIL", 0.92))
OVH, GPUS = float(env("OVERHEAD_GB", 12)), int(env("GPUS", 56))
CTX, MTP = int(env("CTX", 262144)), int(env("MTP", 0))
DCP = int(env("DCP", 1)) # vLLM -dcp; 1 = off
def load(p):
 try:
 return json.load(open(p))
 except (OSError, ValueError) as e:
 sys.exit(f"usage: glmfit.py config.json index.json [index2]\n{e}")
def indexer_layers(c):
 # vLLM models/deepseek_v32/attention.py: Indexer unless skip_topk
 f, off = c.get("index_topk_freq", 1), c.get("index_skip_topk_offset", 2)
 pat = c.get("index_topk_pattern")
 n = c["num_hidden_layers"]
 if pat:
 return sum(pat[i] != "S" for i in range(n))
 return sum(max(i - off + 1, 0) % f == 0 for i in range(n))
if len(sys.argv) < 3:
 sys.exit(__doc__.splitlines()[1])
cfg, idx = load(sys.argv[1]), load(sys.argv[2])
wt = idx["metadata"]["total_size"] / 1e9
n = cfg["num_hidden_layers"] + MTP
ni = indexer_layers(cfg) + MTP
per_tok = MLA * n + IDX * ni
sess = CTX * per_tok / 1e9
listed = sum(t == "full" for t in cfg.get("indexer_types", []))
print(f"weights {wt:7.2f} GB (index.json total_size)")
print(f"indexer {ni} of {n} layers (vLLM rule; config lists {listed})")
print(f"KV per token {per_tok:,} B = {MLA} x {n} MLA"
 f" + {IDX} x {ni} indexer")
print(f"one {CTX:,}-token session = {sess:.2f} GB;"
 " at dcp=1 every TP rank holds all of it")
print(f"budget {HBM:.0f} GB x {UTIL:.2f} - {OVH:.0f} GB overhead"
 " (assumed)\n")
full = cfg["max_position_embeddings"] * per_tok / 1e9
print("TP dcp wt/GPU KV/GPU tok/replica sess reps fleet tok 1M?")
for tp in (4, 8):
 w = wt / tp # even split: approximation
 kv = HBM * UTIL - OVH - w
 reps = GPUS // tp
 if kv <= 0:
 print(f"{tp:<2} {'-':>3} {w:6.2f} weights alone exceed the budget")
 continue
 d = DCP if tp % DCP == 0 else 1 # vLLM refuses tp % dcp != 0
 tok = kv * 1e9 * d / per_tok # dcp=1: every rank holds a full copy
 boot = "boots" if kv * d >= full else "no"
 print(f"{tp:<2} {d:3} {w:6.2f} {kv:6.2f} {tok:11,.0f} {tok / CTX:5.2f}"
 f" {reps:4} {tok * reps / 1e6:8.2f}M {boot}")
print(f"\n1M? = can vLLM start with max_model_len"
 f" {cfg['max_position_embeddings']:,} ({full:.2f} GB)")
if len(sys.argv) > 3:
 o = load(sys.argv[3])["metadata"]["total_size"] / 1e9
 print(f"other checkpoint {o:.2f} GB; this one is"
 f" {100 * (1 - wt / o):.1f}% smaller")

​

05 — Run it

From nvidia/GLM-5.3-NVFP4, download config.json and model.safetensors.index.json (save it as nvfp4.index.json). From zai-org/GLM-5.3, the FP8 release, download the index too (save it as fp8.index.json). Then:

python3 glmfit.py config.json nvfp4.index.json fp8.index.json

weights 464.12 GB (index.json total_size)
indexer 21 of 78 layers (vLLM rule; config lists 21)
KV per token 47,700 B = 576 x 78 MLA + 132 x 21 indexer
one 262,144-token session = 12.50 GB; at dcp=1 every TP rank holds all of it
budget 180 GB x 0.92 - 12 GB overhead (assumed)
TP dcp wt/GPU KV/GPU tok/replica sess reps fleet tok 1M?
4 1 116.03 37.57 787,653 3.00 14 11.03M no
8 1 58.01 95.59 2,003,889 7.64 7 14.03M boots
1M? = can vLLM start with max_model_len 1,048,576 (50.02 GB)
other checkpoint 755.62 GB; this one is 38.6% smaller

Read the sess column: full 256K sessions per copy of the model. TP4 gets 3.00 and TP8 gets 7.64. Across 56 GPUs, TP4's 14 copies hold 11.03M cached tokens and TP8's 7 copies hold 14.03M. That's 27% more cache from half as many copies.

06 — Is NVFP4 really 40% smaller?

The last line compares the two checkpoints’ total_size: 464.12 GB for NVIDIA's NVFP4 against 755.62 GB for Z.ai's FP8. That's 38.6% smaller, close to the talk's "about 40%".

Why not 50%, if 4 bits is half of 8? The Hub lists 27.9 billion BF16 parameters in the NVFP4 repo. Its config puts only the mlp.experts modules in NVFP4 and the first three dense MLPs in FP8. The multi-token-prediction layer, model.layers.78*, is excluded from quantization. The rest stays BF16.

07 — Why 1M context won’t start on 4 GPUs by default

GLM-5.3’s config advertises max_position_embeddings: 1048576. vLLM's check_enough_kv_cache_memory refuses to start if a single request of max_model_len can't fit. At 47,700 bytes a token, one 1M-token request needs 50.02 GB.

On TP4 each GPU has 37.57 GB for cache, and each GPU must hold the whole request. So the 1M? column says no. Set --max-model-len to 256K (anything under about 787K tokens fits) and, by the script's numbers, the startup check passes. "Long context is a scam" is literally true here.

08 — The FP8 version doesn’t fit on 4 GPUs at all

python3 glmfit.py config.json fp8.index.json

​

TP dcp wt/GPU KV/GPU tok/replica sess reps fleet tok 1M?
4 - 188.90 weights alone exceed the budget
8 1 94.45 59.15 1,239,997 4.73 7 8.68M boots

Same config.json; only the weight size changes. At 188.90 GB per GPU, FP8 weights alone are bigger than the 153.6 GB the script leaves for weights and cache. On 8 GPUs it fits but holds 4.73 sessions against NVFP4's 7.64. NVFP4 is what makes 4 GPUs possible at all.

09 — Two settings worth trying

First, vLLM can split the cache across GPUs instead of copying it: --decode-context-parallel-size (-dcp). When vLLM checks whether a max-length request fits, it divides max_model_len by the DCP size. The script assumes the cache splits evenly. With DCP=8:

TP dcp wt/GPU KV/GPU tok/replica sess reps fleet tok 1M?
4 1 116.03 37.57 787,653 3.00 14 11.03M no
8 8 58.01 95.59 16,031,116 61.15 7 112.22M boots

TP8 goes from 7.64 to 61.15 sessions per copy. vLLM refuses -dcp 8 on 4 GPUs (the GPU count must be divisible by the DCP size), so the script shows that row with DCP off. With DCP=4, TP4 reaches 12.02 sessions and TP8 30.58, so 8 GPUs still hold more.

I read this in source and did not run it on B200s. DCP adds GPU-to-GPU traffic on every decode step, so measure speed before you trust it. The default FlashInfer backend supports DCP with GLM’s a2a default. If you switch to the FlashMLA backend, it only accepts --dcp-comm-backend ag_rs.

Second, the FlashMLA backend (--attention-backend FLASHMLA_SPARSE) stores 656 bytes per layer instead of 576. Run MLA=656 and the numbers drop to 2.66 and 6.76 sessions. 8 GPUs still hold over 2.5 times as many.​

10 — What the script assumes

• 180 GB per B200 and vLLM’s default gpu_memory_utilization of 0.92.
• 12 GB per GPU for activations, CUDA graphs and communication. That’s a guess. With OVERHEAD_GB=24, TP4 drops to 2.04 sessions and TP8 to 6.68.
• Weights split evenly across GPUs. Real splits copy a few small tensors.
• No speculative-decoding layer. MTP=1 adds one cache layer and one indexer layer.
• vLLM’s data-parallel attention mode, which also avoids the copy, isn’t modelled.

Every session, token, GB and percentage in sections 05–10 comes from glmfit.py, either in the output blocks or from the env runs named beside them (DCP=8, DCP=4, MLA=656, OVERHEAD_GB=24, MTP=1). The 27% and 153.6 GB are arithmetic on its output, and the lede rounds 3.00 and 7.64. The talk's figures (56 B200s, 2,000 engineers, 3 seconds, about 40%) come from the captions. The Hub's dtype counts and quantization layout in section 06, and the vLLM file references, come from the Hub pages and the vLLM 0.32.0 wheel. None of this was run on a GPU.

11 — Verdict

vLLM keeps a full copy of GLM-5.3’s attention cache on every GPU in a tensor-parallel group. On paper that gives 8 GPUs 7.64 full 256K sessions per copy against 3.00 for 4 GPUs, and without DCP, 4 GPUs can’t even start at the advertised 1M context. Revolut’s advice, to cap context at 256K and try 8 GPUs, matches what vLLM’s code implies.

When you size a self-hosted coding model, do you count the weights, or the sessions the cache can hold?

If this saved you a GPU order, a clap helps others find it.

Follow me for daily LLM deep-dives.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI