Author(s): Shrashti Singhal

Originally published on Towards AI.

Propagating a latency budget into the reasoning, so the agent trades depth for time on purpose instead of timing out mid-thought.

Here is a race condition currently running in production at more companies than would enjoy hearing it, described in a post-mortem pattern that’s been written up publicly more than once. An agent calls a tool with an eight-second budget. The tool, independently, has an eight-second timeout of its own. Sounds compatible. It isn’t — because the two clocks start at different moments. The agent’s budget starts ticking when the model begins emitting the tool call; the tool’s timer starts when the request actually reaches it. Between those two instants sit a second of prefill, a few hundred milliseconds of routing, some queueing. So the agent has quietly spent 1.4 seconds of its eight before the tool believes the race has begun. At 7.9 seconds on the agent’s clock the agent declares the call dead and replans; at 8.0 on the tool’s clock, the tool finishes successfully and delivers a correct answer to a corpse. The agent, meanwhile, has retried — so the work runs twice, the trace shows a tool_use with no matching tool_result, and somewhere a duplicate side effect is being reconciled by hand.

The rest of the article argues that deadline handling must be treated as a first-class, shared “wall-clock” constraint rather than separate relative timeouts for each component. It explains where time actually goes in an agent turn (queueing, prefill, thinking, output streaming, tool time, and retries), then lays out engineering fixes: express deadlines as absolute timestamps propagated through the entire subtree, compute per-attempt time from a single overall ledger, and implement deadline checks as coordinated cancellation points. It emphasizes the key missing piece—telling the model about remaining time via a “fuel gauge” in the turn context—so the model can actively trade depth for speed. The article introduces an “anytime” agent pattern (draft early, refine in passes, deliver when on the gauge) plus strategies like output reserves, degradation ladders chosen early, deadline-aware routing, hedged parallel attempts, and queue-aware thinking allocation. It concludes with practical guidance (install the gauge, mint and propagate an absolute deadline, reserve output/reaction time, run evals across budget settings) and frames the core insight: budgets are something you spend deliberately, while unshared deadlines become ambushes that waste work.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI