Author(s): Jun Nishimura

Originally published on Towards AI.

How a hierarchical architecture shrinks the growing KV cache — explained mostly in pictures.

This is Part 1 of 3. Here we explain the PHOTON architecture (Ichikawa et al., 2025). Part 2 retrofits it onto a pretrained Qwen3 by distillation; Part 3 is the technical deep-dive, a live demo, and an open-source release.

The bottleneck nobody optimizes away

When a language model generates text, it produces one token at a time — and for every new token, it re-reads its entire memory of the conversation so far. That memory is the KV cache, and it has two nasty properties:

1. It grows by one entry every single token, across every layer. 2. The work of reading it grows too — step 1 reads 1 entry, step 100 reads 100.

Add those reads up and you get ~t²/2 total memory traffic to generate t tokens. On a modern GPU the arithmetic isn't the problem — fetching that cache from memory is. This is what people mean when they say "decoding is memory-bound," and for long contexts or on-device models it's the wall that matters most.

Plenty of tricks chip at this — grouped-query attention, KV quantization, paged caches. They help, but they all keep the same basic shape: one flat sequence, re-read in full, forever. PHOTON asks a different question — what if the sequence you re-read barely grew at all?

The idea: stop reading horizontally

A Transformer reads horizontally — left to right along one flat row of tokens. PHOTON reads vertically: it compresses tokens into a handful of coarse summaries and generates from those, keeping the full-resolution detail only in tiny local windows.

The intuition: most of what you need from far-back context is a summary, not every word verbatim. So keep a running summary at the top of a pyramid, and only that top level has to persist as you generate.

Three simple pieces

The whole architecture is built from three small parts:

• Chunker — merges C neighboring tokens into a single coarse latent (compression).
• Converter — unpacks one parent latent into R short "briefing" vectors for the level below.
• Local decoder — reconstructs tokens inside a fixed window of R + C positions. Crucially, this window never grows with context length — it's always the same tiny size.

That fixed window is the key. A normal decoder attends over everything it has ever seen; PHOTON’s local decoders attend over ~4 positions, no matter how long the document is.

The shape: compress up, reconstruct down

Stack those pieces and you get an hourglass. A bottom-up encoder compresses tokens into ever-coarser latents; a top-down decoder takes the coarsest summary and reconstructs the tokens back out, one local window at a time.

Here’s the payoff hiding in that diagram: only the coarse stream at the top keeps a growing cache. The encoder’s fine levels are summarized upward and then dropped; the two decoder stacks never look back more than a handful of positions, so their caches have nothing to remember between chunks.

One quick label: (C₁, C₂)

You’ll see PHOTON described by two numbers — its compression schedule — and it’s worth pinning down, because everything else scales with it. The two numbers are simply the merge factors at each level: C₁ is how many tokens the first chunker merges into one latent; C₂ is how many of those it merges again into one summary.

So (2,2) — the gentle setting we use to convert Qwen3 — merges 2, then 2 again: 4 tokens under each summary. The paper’s headline (4,4) merges 4, then 4: 16 tokens per summary, with correspondingly larger savings. Bigger numbers → more compression → more speedup → but a harder model to train. Keep the little (2,2) picture in mind; the next section is just it, in motion.

Generating: 4 tokens, 1 cache entry

Now watch what happens during generation. In a vanilla model, every token adds one entry to the growing cache. In PHOTON, one pass down the hierarchy produces a whole chunk of tokens but adds only one entry to the single growing stream.

Two things compound here. PHOTON caches fewer moments (one entry per C₁×C₂ tokens — 4, at the (2,2) setting shown) and on fewer layers (only the top encoder stack). At the specific setup this series builds toward — Qwen3-0.6B split into 4 stages of 7 layers — that's 4× fewer moments × 4× fewer layers ≈ 16× less growing cache. (That 16× is our conversion's arithmetic, not the paper's headline — the real paper numbers are below.)

RecGen: refreshing the summary without re-reading

There’s one subtlety left — and it’s the clever part. After you generate a chunk of tokens, the summary at the top is stale: it doesn’t yet know about the words you just wrote. Before you can generate the next chunk, you have to update it.

The obvious way is to take the new tokens and push them back up through the encoder. But that means re-reading them — exactly the cost we set out to avoid. PHOTON’s trick, called RecGen (recursive generation), sidesteps it in four small moves. Let’s take them one at a time.

Step 1 — Decode down.

The current summary flows down through the decoders, which write the next few tokens. Nothing surprising yet — this is just normal generation.

Step 2 — A free byproduct.

Here’s the key observation. To produce those tokens, the top-down decoder had to compute a reconstruction of them along the way. That reconstruction is just sitting there — a free byproduct of decoding, no extra work.

Step 3 — Refresh the summary (the trick).

Instead of sending the new tokens all the way back up through the encoder (the dashed, crossed-out path), PHOTON applies just the cheap chunker to that reconstruction and uses the result as the updated summary. The encoder never runs on the new tokens at all — that’s the whole savings.

Step 4 — Cache +1, then repeat.

The summary is current again, the growing cache has gained exactly one entry, and we’re ready to generate the next chunk. Loop back to Step 1.

So when is this exact? RecGen gives the same answer as actually re-encoding whenever the decoder’s reconstruction equals what the encoder would have produced. PHOTON trains a dedicated recursive loss to make that true. When it’s only approximately true, RecGen becomes an approximation — and measuring exactly how good that approximation is turns out to be a fun rabbit hole we’ll fall into in Part 3.

The payoff: a much smaller growing cache

Put it all together and the growing cache — the thing that dominates decode memory traffic — shrinks dramatically. Here’s the arithmetic for the specific setup this series builds toward: converting Qwen3–0.6B at the gentle (2,2) compression (that’s Part 2).

That ~16× is our conversion’s analytic number — not the paper’s. PHOTON’s own results come from models trained from scratch at a more aggressive (4,4) compression (600M / 900M / 1.2B parameters, on the Pile). What the paper actually measures is bigger and real, not idealized arithmetic: on a 1.2B model, KV memory 0.390 → 0.036 GiB (10.8×), decode throughput 43.8×, and up to ~1⁰³× higher throughput per unit memory — their headline “tokens-per-memory” metric.

Two honest caveats:

• The step-count speedup is config-dependent. At the gentle (2,2) setting we use, there’s no sequential-step speedup at all — every gain is memory traffic (still the dominant lever, since decode is memory-bound). At the paper’s (4,4), PHOTON does post real wall-clock throughput gains (e.g. 2.3× for a from-scratch 600M model under RecGen). More compression, more speedup.
• Analytic ≠ realized. Our 16× is idealized arithmetic for one (2,2) conversion; the paper’s numbers are measured on different, from-scratch (4,4) models. Different systems, different settings — but the honest pattern is the same in both: real-world reductions land somewhat below the clean analytic figure once local caches, activations, and kernel overhead are counted. The reduction is real; it just isn’t free.

The catch — and what’s next

PHOTON is a genuinely different way to generate text: read vertically, cache only the summary, reconstruct locally. But there’s a practical catch — it’s pretrained from scratch. Every result in the paper comes from training a PHOTON model from zero, which is exactly what most of us can’t afford.

So the question that kicked off this series: could you take an existing pretrained model — say Qwen3–0.6B — and convert it into the PHOTON architecture, cheaply, without pretraining?

That’s Part 2.

PHOTON is the work of Ichikawa et al. — “PHOTON: Parallel Hierarchical Operation for TOp-down Networks” (arXiv:2512.20687). This series is an independent, distillation-based reproduction and study; all diagrams here are my own illustrations of their architecture.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI