Author(s): Enzo Lombardi

Originally published on Towards AI.

Reading answers out of a hot KV cache instead of generating them

Most of what a coding agent asks its model is not a question that needs an essay. Is this conversation worth remembering. Does this command touch the network. Should I compact now. These are single bits, and the standard way to get one is to ask for text and then parse whatever comes back. You pay for a full decode loop, you pay again for the retry when the model wraps its JSON in a markdown fence, and the answer you finally get carries no honest measure of how sure the model was.

After explaining why single-bit “answers” shouldn’t require full text generation, the article describes “TypeSafe’s” Jev-style decision making (System 1 vs System 2) and the key cost model: a decoder’s probability for the answer is already present after the first forward pass, so generation is mostly ceremony. It then contrasts Jev with an alternative decision model (Laya), arguing that while such models can be fast and calibrated, practical constraints like context handling and state “already paid for” make them less suitable for an agent. The core win comes from keeping the model’s KV cache hot across turns in a persistent Rust agent (plank), allowing decision queries to run as cheap forward passes over cached state rather than full decode loops—while also noting a crucial limitation where naive rewinds fail on DeepSeek and require careful cache handling. The author then shares a major “perfect no-op” bug: tests looked fine but letter probabilities were near zero, causing the feature to abstain silently, and the fix involved scoring the correct tokenization (e.g., removing a trailing space). Finally, the piece explains how plank applies these mechanisms to decide whether to write memory entries (and when), introduces gatekeeping to avoid wasting expensive passes, discusses edge cases around interrupts and idle-time costs, and distills the broader lesson: features that fail silently must expose the relevant probability mass (not just a supported/unsupported boolean) before being trusted.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI