Author(s): Zenefa Rahaman, PhD

Originally published on Towards AI.

Your Agent Didn’t Hallucinate. Your Architecture Laundered a 0.51.

Probabilistic systems are useful precisely because they are not fully deterministic. A language model can interpret an underspecified request, generate several plausible plans, and adapt to context its developers never anticipated — because it is not executing a fixed program, but selecting among continuations based on a learned distribution.

That flexibility is not a defect. It is the source of the abstraction. Yet when these systems behave unreliably in production, our first response is to reduce uncertainty at the source. We lower the temperature, constrain the schema, add examples, swap in a larger model, and hope a more accurate first decision will stabilize everything that follows.

These interventions can improve individual outputs. They do not make the surrounding system reliable.

Temperature zero does not help either. It may make token selection repeatable, but it does not make the interpretation correct — the system can choose the wrong intent with perfect consistency. A repeatable mistake is not a reliable decision.

The deeper problem is not that uncertainty exists. It is that uncertainty propagates. A slightly ambiguous classification changes what is retrieved. The retrieved material changes the prompt. The prompt changes which tool is selected. The tool result changes what is stored in memory. That memory changes the next execution.

By the time the system produces a visibly wrong outcome, the original uncertainty may be several steps behind it. Nothing downstream appears probabilistic anymore; each component simply processes the input it received. The uncertainty has been converted into documents, parameters, and durable records. It has been laundered into apparent fact.

This changes the reliability question. We should not ask only, “How confident is the model?” We should ask, “How far is this uncertainty allowed to travel before something checks it?”

Reliability is not the absence of non-determinism. It is the containment of non-determinism.

The design goal cannot be universal suppression — that would remove the capability we are paying for. It must be selective containment. Allow uncertainty where it creates value: interpreting ambiguous instructions, generating candidate plans, ranking alternatives. Prevent it from silently entering areas where it causes disproportionate damage:

• committing transactions;
• modifying durable state;
• selecting irreversible actions;
• updating long-term memory;
• authorizing external side effects;
• establishing facts on which later decisions will depend.

This is not primarily a statistical problem. It is a topological one. A statistical framing asks how we can make the dice more predictable. A topological framing asks which parts of the system the dice are allowed to reach.

That distinction matters because even an excellent model will remain uncertain at the margins. If our architecture assumes better modeling will eventually eliminate ambiguous requests, incomplete context, and conflicting evidence, reliability will remain perpetually one model release away.

The Propagation Cascade

In a multi-step system, one model decision becomes the next component’s input. That component transforms it, acts on it, or stores it — and each transformation creates another dependency on the original decision.

The compounding is not merely additive. Each early variation changes the entire set of choices available downstream. Once the execution path branches, later components are no longer operating over the same problem.

Consider a support agent responsible for investigating a customer’s account issue. The user writes: “I was charged again after I changed my plan.”

The request could refer to a duplicate charge, a delayed charge from the previous plan, a prorated adjustment, an unsuccessful cancellation, or a renewal that occurred before the plan change took effect.

Step 1: Intent classification

The system first classifies the request. Suppose the model assigns 0.51 probability to duplicate_charge, 0.46 to plan_change_billing, and 0.03 to other categories.

This is not a contrived case. A well-calibrated classifier produces narrow margins precisely when the request is genuinely underspecified — which, in customer support, is most of the time. The 0.51/0.46 split is not the model performing badly; it is the model correctly reporting that the message supports two readings. The system nevertheless selects duplicate_charge.

At this point the uncertainty is small and local. No record has changed; no refund has been issued. The architecture could still contain the ambiguity here. Instead, the label is passed downstream as if it were a fact.

Step 2: Retrieval

The retrieval layer uses duplicate_charge to select policies, prior cases, troubleshooting instructions, and tool documentation. It retrieves duplicate-transaction detection procedures, refund eligibility rules, payment-processor reconciliation guidance, and examples involving two identical charges. It does not retrieve plan-transition billing rules, effective-date calculations, prorated invoice explanations, or renewal cut-off policies.

Retrieval has now amplified the initial decision. The system has assembled an evidence environment in which one interpretation is strongly represented, and the other is nearly absent. The retrieved documents appear authoritative; the system does not see them as consequences of a weak classification, but as the relevant context. A 0.05 difference in confidence has changed the factual universe available to the next step.

Step 3: Prompt construction

The system then assembles a prompt containing the selected intent, retrieved policies, account metadata, and task instructions. The prompt may now state: “The customer is reporting a duplicate charge. Determine whether the two transactions qualify for a refund.”

That sentence has removed the ambiguity entirely. The classifier produced a narrow preference between two interpretations. The prompt builder converted that preference into a declarative premise. The distinction between “the model slightly preferred this interpretation” and “this is what happened” has disappeared.

That is the laundering step. Probabilistic evidence has become deterministic-looking context. The next model is not being asked to reconsider the original request. It is being asked to reason correctly from an incorrect premise — and it may perform that task extremely well.

Step 4: Tool selection

Given the prompt, the system selects a transaction-comparison tool rather than a plan-history tool. It compares two charges and sees that they have similar amounts but different billing-period identifiers. Because the duplicate-charge workflow expects payment anomalies, it interprets the difference as a processor or settlement issue.

It never inspects the plan-change effective date, because that lies outside the selected workflow. The tool layer has not malfunctioned — it received a duplicate-charge investigation and executed the appropriate operation. The uncertainty now governs not only the system’s interpretation, but which external data it examined.

Step 5: Memory and state

The system writes the following summary into durable case memory: “Customer experienced a probable duplicate charge after changing plans. Transaction comparison indicates a billing duplication requiring refund review.”

A human reviewer now sees the case pre-labeled as a probable duplicate. The next model treats the summary as established history. Analytics count it as a duplicate-billing incident. Future messages are interpreted through the stored diagnosis. At step one, the system faced two interpretations separated by five percentage points; at step five it holds a durable record stating that one of them probably occurred.

Nothing in the middle had to fail. The retriever retrieved the correct documents for the chosen label. The prompt builder accurately represented the retrieved workflow. The model reasoned coherently over the prompt. The tool performed the requested comparison. The memory service stored the supplied summary. Each component did what it was told; the failure exists in the chain of dependency connecting them.

This is unbounded probabilistic propagation: uncertainty enters at one point, crosses multiple boundaries, and is repeatedly transformed without being resolved. It also obscures causality. When the customer disputes the refund explanation, the visible failure appears to be in the final response, so engineers inspect the generation prompt or blame the model for hallucinating. But the decisive branch occurred four steps earlier, when a marginal classification was allowed to determine retrieval.

The final response is where the error became visible, not where the system became unreliable.

The same shape appears in a coding agent, where an uncertain interpretation changes the files retrieved and the patch proposed, and in a research agent, where an uncertain decomposition changes the search terms and the citations. In each case the system does not merely carry uncertainty forward; it converts uncertainty into conditions that make the chosen path self-confirming. Most apparent agent failures are propagation failures: one early uncertainty that no layer was positioned to contain.

Bounded Non-Determinism as a Design Principle

If propagation is the problem, reliability requires bounding the blast radius. Bounded non-determinism does not require every model output to be certain. It requires every uncertain output to have a limited sphere of influence. The system should know where uncertainty entered, which downstream decisions depend on it, and where it must be resolved before execution continues.

1. Resolve uncertainty at the layer where it enters

A low-confidence classification should be handled as a classification problem. It should not be passed downstream as an ordinary label and left for retrieval, planning, or tool execution to discover indirectly.

The layer might request clarification, preserve multiple candidate intents, retrieve evidence for both readings, route to a safer generic workflow, or abstain from consequential action. The mechanism matters less than the timing: uncertainty is cheapest to resolve before other components depend on it. Once a tentative classification has shaped retrieval, prompts, tools, and memory, correcting it means unwinding a chain of derived decisions — invalidating results, rewriting state, retracting claims. Containment gets more expensive with every boundary crossed.

2. Collapse uncertainty at boundary crossings

Between stages, probabilistic output should become either a committed decision or an explicit unknown state. What should not cross the boundary is an ambiguous middle presented as certainty.

A classifier could emit duplicate_charge committed after validation, plan_change_billing committed after validation, ambiguous_billing_intent unresolved, or a structured set of candidates with confidence and provenance. It should not select duplicate_charge by a narrow margin and send only that label downstream, stripped of the uncertainty that produced it.

Boundary crossings should preserve epistemic status. A downstream component must be able to distinguish an observed fact, a verified decision, a model inference, and an unresolved ambiguity. Without that distinction, each component reads the previous one’s output as more certain than it was — which is how uncertainty disappears from representation while continuing to drive behavior. Note that collapsing uncertainty does not mean forcing certainty: unknown and requiring confirmation are valid committed states, often more reliable than choosing an answer because the interface demands one.

3. Bound and observe propagation depth

We should be able to answer a basic architectural question: how many downstream decisions depend on this uncertain input? For most systems the answer is unknown. An intent label may influence retrieval, ranking, planning, tool use, memory, and future personalization with no explicit dependency graph connecting those effects. That makes blast radius invisible.

A bounded design assigns a maximum propagation depth or consequence level to uncertain decisions. A low-confidence inference might affect candidate generation but not external actions. A speculative plan might influence a preview but not durable state. An unverified summary might remain session-local rather than entering long-term memory.

Propagation should also be observable after the fact: engineers should be able to trace where the originating uncertainty entered, which boundaries it crossed, and which decisions inherited it. This is not ordinary model logging. Token probabilities do not reveal architectural blast radius. We need lineage across decisions.

Why More Capable Models Can Make the Problem Worse

The obvious counterargument is that propagation will matter less as models improve — that a sufficiently capable model will classify correctly, retrieve correctly, and avoid the cascade. Better models do improve per-step accuracy. But per-step accuracy and propagation are different properties: a more capable model reduces the probability of an error entering the system, not how far it travels once it does.

Capability may actually deepen the problem, because it makes longer autonomous executions possible. As models improve, we ask them to plan over longer horizons, invoke more tools, and act with less review — and each added step is another boundary crossing where an uncertain inference can become a downstream premise. Suppose per-decision reliability rises from 95 to 98 percent. If the more capable system now executes twenty dependent decisions instead of five, it can be less reliable end to end while being more accurate at every step. And once it retrieves the wrong evidence, improved reasoning only helps it pursue that path more coherently.

This is why capable systems produce failures that are more polished, more internally consistent, and harder to detect. The system is not reasoning randomly. It is reasoning well from a state that was corrupted earlier.

A more capable model in an unbounded architecture is a faster car with no brakes. The speed is real. So is the increased consequence of missing a turn.

The Distributed-Systems Parallel: Fault Isolation

Distributed systems learned long ago that failures cannot always be prevented. Machines crash, networks partition, operators deploy bad configurations. The architectural response is not to assume every component can be made infallible, but to prevent one failure from becoming a system-wide event.

The vocabulary here is well established. Michael Nygard’s Release It! popularized bulkheads and circuit breakers as structural defenses against cascading failure; Netflix’s Hystrix turned them into standard practice; AWS’s cell-based architecture guidance treats blast radius as a first-class design property rather than an operational afterthought. In each case the goal is the same: bound the set of components that a single local failure is permitted to reach.

An uncertain inference is not a crashed server, but it plays the same architectural role: a local condition that may be unavoidable, whose system-wide consequences depend entirely on whether isolation boundaries exist. Without isolation, one mistaken decision reaches retrieval, planning, tools, memory, and future behavior. With it, the same decision affects only a provisional candidate list or a reversible draft.

So treat each uncertain decision as belonging to a probabilistic failure domain, which defines what may consume it, what actions may be taken from it, whether it can enter durable state, and what evidence is required before it crosses into a higher-consequence domain. A planning model may generate ten speculative approaches, but they stay in the planning domain. A support model may infer several likely causes, but no refund is initiated until account state confirms one. The uncertainty is not removed. Its operating area is restricted.

I have argued elsewhere that agents need a controller — a deterministic boundary for commitment. [Link: “LLMs Should Reason. Infrastructure Should Enforce.”] Fault isolation is the complementary idea: even with that boundary, we need to bound how far uncertainty travels before it reaches one. A controller decides where commitment happens; bounded non-determinism decides how much has already been contaminated by the time it gets there.

A boundary placed too late still permits a large blast radius: by the time commitment is reviewed, the evidence, plan, and state presented for review may all descend from the same unresolved assumption. Containment must operate throughout the execution topology, not only at the final action.

Reliability Is the Containment of Surprise

Reliable systems bound the blast radius. They resolve uncertainty where it enters rather than forcing downstream components to inherit it. They preserve the difference between verified facts, model inferences, and unresolved states. They collapse ambiguity at architectural boundaries instead of laundering it into certainty. They limit how many consequential decisions can depend on one probabilistic output. And they maintain enough lineage to trace a failure back to the uncertainty that started the cascade.

These properties matter more, not less, as models improve: better models reduce the frequency of some mistakes, but they also enable larger execution graphs in which the remaining mistakes travel farther. Bounded non-determinism is not an optimization, a prompting technique, or a temporary safeguard until the models get good. It is the precondition for operating probabilistic systems in production.

Reliability does not come from a model that never surprises us. It comes from a system where surprise has somewhere to stop.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI