Author(s): Pratik K Rupareliya

Originally published on Towards AI.

Observability for Production AI Agent Systems: The 4-Layer Instrumentation Stack

Why LLM logging is not observability, and the three layers most teams skip until production breaks.

The healthcare workflow agent had been running for eight weeks. The clinical operations dashboard showed everything green. Latency: within target. Token costs: on budget. Faithfulness scores from the offline eval suite: above 0.95. The team had instrumented the LLM call layer carefully and built a clean monitoring view on top of it.

Then, a nurse flagged that the agent had been pulling the wrong medication interaction data for at least three weeks.

We pulled the logs. The model had not changed. The retrieval scores had not changed. The LLM was returning responses that were faithful to whatever the tool calls handed it. The problem was that the tool calls had been quietly returning structurally valid garbage. An upstream catalog service had silently migrated a field. The tool wrapper passed back HTTP 200, the payload was the right shape, and the agent treated the data as truth. Two weeks of debugging just to find the layer in which the failure lived.

This is the most consistent pattern I see across the AI agent deployments I have shipped over the last two years. Teams instrument the LLM call layer carefully, treat that as “AI monitoring,” and then watch production break in three other layers they were not watching.

LLM logging is not agent observability. It is one of four layers, and it is not even the most expensive one when it fails.

The framework below is the 4-layer instrumentation stack we run on every production AI agent deployment. It covers what each layer measures, which tools fit at each layer, the specific failure modes each one catches, and how to phase the layers in under launch pressure. The healthcare case above lived in Layer 3.

Why most teams stop at Layer 1

Layer 1 is where the obvious tooling lives. Every Frontier Lab provides native logging. LangSmith and Helicone install in an afternoon. Datadog has an LLM Observability product. The metrics are concrete: tokens in, tokens out, latency, and cost. The dashboards look professional.

The trap is that Layer 1 metrics correlate poorly with the failure modes that actually kill production agents. The healthcare deployment I opened with had perfect Layer 1 metrics through the entire eight weeks the failure was active. Cost per query: stable. p95 latency: stable. Token usage: stable. The Layer 1 view said everything was fine.

Production AI agent failures live in the layers that Layer 1 cannot see. The agent’s reasoning trajectory, the tool calls it makes, and whether the user actually achieved what they came for are all invisible to the LLM call layer.

For teams building AI agents for business automation at any meaningful scale, the move from Layer 1 alone to the full 4-layer stack is the single most important infrastructure decision after the model choice itself.

Layer 1: the LLM call layer

This is the foundation. Get it right early, then move on.

What instrument: Every LLM API call. The prompt that went in, the response that came out, tokens consumed, end-to-end latency, cost per call, model version used, and request/response metadata. If you are using OpenAI’s Assistants API or Anthropic’s tool-use API, you also instrument the tool selection step inside the call.

Tools that fit: LangSmith (best ecosystem coverage if you are already in the LangChain world), Helicone (lightweight proxy approach, easy to install), Datadog LLM Observability (best fit for enterprises already in the Datadog stack), Phoenix (Arize) if you want open-source and rich span analysis. For teams standardizing on OpenTelemetry, the GenAI semantic conventions (github.com/open-telemetry/semantic-conventions) are now the lingua franca.

What it catches: Token cost spikes, latency regressions, prompt template breakage, model-version drift, and rate limit incidents. These are failures that appear to be operational problems and have clear remediation paths.

What it misses: Almost every multi-step failure mode, every tool execution failure, every drift in upstream data dependencies, and every business outcome gap.

Minimum viable instrumentation:

from opentelemetry import trace
from opentelemetry.semconv.ai import SpanAttributes

tracer = trace.get_tracer(__name__)
def call_llm(prompt: str, model: str = "gpt-4o") -> str:
 with tracer.start_as_current_span("llm.call") as span:
 span.set_attribute(SpanAttributes.LLM_REQUEST_MODEL, model)
 span.set_attribute(SpanAttributes.LLM_PROMPTS, prompt)
 response = client.chat.completions.create(
 model=model,
 messages=[{"role": "user", "content": prompt}],
 )
 span.set_attribute(SpanAttributes.LLM_RESPONSE_MODEL, response.model)
 span.set_attribute(
 SpanAttributes.LLM_USAGE_TOTAL_TOKENS, response.usage.total_tokens
 )
 span.set_attribute(SpanAttributes.LLM_COMPLETIONS, response.choices[0].message.content)
 return response.choices[0].message.content

Fifteen lines, OpenTelemetry-native, vendor-neutral. Every span this emits is visible in any backend that speaks OpenTelemetry (Datadog, Honeycomb, Jaeger, Grafana Tempo, the long tail).

Get Layer 1 in production on day one. Move on.

Layer 2: the agent step layer

This is where you instrument the agent’s reasoning trajectory. Most production agents are not single LLM calls. They are plan-execute-verify cycles, multi-step task decompositions, retry loops, and state transitions between steps.

What instrument: Every step the agent takes inside a single user request. The plan it produced, the step it executed, the result of that step, the next step it chose, any retries it attempted, and the final state it returned. The span hierarchy is critical here: parent span = user request, child spans = each agent step, grandchild spans = tool calls and LLM calls inside each step.

Tools that fit: LangSmith traces (excellent if you are on LangChain or LangGraph), Phoenix from Arize (open-source, strong trace UI, fits non-LangChain stacks well), OpenTelemetry GenAI semantic conventions with a trace backend of your choice. The convergence here is real: OpenTelemetry is becoming the substrate that LangSmith and Phoenix both speak of.

What it catches: Agent loops (the agent gets stuck retrying the same step), plan drift (the agent’s plan diverges from the user’s intent after step 3), premature termination (the agent declares done before actually finishing), and step latency spikes inside long workflows. These are the failures that look like “the agent is being weird” from outside and need trace-level analysis to diagnose.

Production thresholds I monitor:

• Average cycle length per request (alerts on >2x baseline)
• Retry rate per agent step (alerts on >0.05)
• Plan revisions per request (alerts on >3, indicates instability)
• p95 step duration per agent type

A useful debugging pattern: when something goes wrong, the trace tree shows you the entire trajectory of the agent’s decision-making, not just the final output. You see why the agent did what it did, not just what it returned. This is the diagnostic capability that lets you debug agent behavior in hours, not weeks.

The pattern Nik Kale highlighted in his recent InfoQ piece on autonomous agent security (May 2026) maps directly to this layer: when agents run on Kubernetes with broad access to tools, the trace layer is where you watch for runaway behavior. Layer 2 instrumentation is the prerequisite for the security boundaries that piece argues for.

Layer 3: the tool execution layer

This is where the healthcare deployment broke. It is the most consistent failure mode I see in production agents, and it is the layer most teams underinstrument.

What instrument: Every tool the agent calls. The tool name selected, the arguments constructed, the API endpoint hit, the response status, the response payload schema, the response payload semantics, where you can validate them, the latency, and the retry attempts. The non-negotiable: validate the schema of every response before it goes back to the agent.

Tools that fit: OpenTelemetry distributed tracing for cross-system tracing (your agent runtime, downstream APIs, and databases). Datadog APM for the deeper integration into existing service maps. Custom tool wrappers in your agent runtime that enforce schema validation at the boundary.

The failure mode that hides at this layer is: a tool returns HTTP 200 with a structurally valid but semantically incorrect payload. The fields are present. The types match the schema your wrapper validates. The values are wrong because an upstream service silently changed their meaning. The agent treats the data as truth. Failure surfaces weeks later when a downstream consequence becomes visible to users.

Minimum viable Layer 3 wrapper:

from typing import Any
from pydantic import BaseModel, ValidationError
from opentelemetry import trace

tracer = trace.get_tracer(__name__)
def tool_call_with_validation(
 tool_name: str,
 args: dict,
 expected_schema: type[BaseModel],
 fetch_fn,
) -> Any:
 with tracer.start_as_current_span(f"tool.{tool_name}") as span:
 span.set_attribute("tool.name", tool_name)
 span.set_attribute("tool.args", str(args))
 try:
 raw_response = fetch_fn(**args)
 span.set_attribute("tool.http_status", raw_response.status_code)
 validated = expected_schema.model_validate(raw_response.json())
 span.set_attribute("tool.schema_valid", True)
 return validated
 except ValidationError as e:
 span.set_attribute("tool.schema_valid", False)
 span.set_attribute("tool.validation_error", str(e))
 span.record_exception(e)
 raise

The schema validation is the alert mechanism. When the upstream catalog service in the healthcare case silently changed a field, the validation would have failed loudly within minutes, rather than allowing two weeks of silent corruption.

For teams approaching custom AI development projects in regulated industries, Layer 3 instrumentation is not optional. The compliance audit will eventually ask how you would detect this failure mode. The right answer is the schema validation at the tool boundary.

Production thresholds I monitor:

• Tool execution success rate per tool (alerts on >0.98 target failure)
• Schema validation failure rate (alerts on any non-zero rate)
• Tool call latency per tool (alerts on p95 >2x baseline)
• Argument-construction error rate (alerts on >0.01)

Layer 4: the business outcome layer

The first three layers measure what the agent does. Layer 4 measures whether what it did actually produced the user’s intended result. This is the layer almost every team skips, and it is the one that catches the most expensive failure mode.

What you instrument: The user’s actual outcome after the agent’s response. Did they accept the agent’s recommendation, modify it, or reject it? Did they complete the workflow the agent was supposed to enable, abandon it, or escalate to a human? What did they do five minutes after the agent’s last response? Two days after?

Tools that fit: PostHog for product analytics tied to agent interactions. Internal eval pipelines for sampled production traffic. User feedback loops (thumbs up/down with reason taxonomies). Business-metric dashboards that tie agent outputs to revenue, cost, or compliance signals.

The failure mode that hides at this layer: A SaaS customer support deployment I worked on achieved 91 percent offline deflection accuracy. Technical metrics were strong. Layers 1, 2, and 3 were all green. Within three weeks of launch, the support agents had quietly disabled the assistant. The deflection rate from production traffic was 12 percent. The model was fine. The system worked. Users did not use it because the workflow integration created friction; the agent could not see from inside its own metrics.

Layer 4 catches this gap. Nothing else does.

Minimum viable Layer 4 event capture:

import posthog
from opentelemetry import trace

tracer = trace.get_tracer(__name__)
def emit_outcome_event(
 user_id: str,
 agent_response_id: str,
 outcome: str,
 workflow_step: str,
 metadata: dict | None = None,
):
 """Capture downstream user outcome after an agent response."""
 with tracer.start_as_current_span("outcome.capture") as span:
 span.set_attribute("outcome.user_id", user_id)
 span.set_attribute("outcome.agent_response_id", agent_response_id)
 span.set_attribute("outcome.value", outcome)
 posthog.capture(
 distinct_id=user_id,
 event="agent_outcome",
 properties={
 "agent_response_id": agent_response_id,
 "outcome": outcome,
 "workflow_step": workflow_step,
 **(metadata or {}),
 },
 )

The wiring matters more than the tool. PostHog is one option among several. The non-negotiable is that the outcome event is causally tied back to the specific agent response via the response ID, so you can join Layer 4 outcomes to Layer 1–2–3 traces and ask: which agent behaviors correlate with which user outcomes?

Production thresholds I monitor:

• Adoption rate per agent feature (alerts on < 30 percent of intended users active)
• Completion rate per workflow (alerts on < 50 percent finishing the agent-assisted path)
• Escalation-to-human rate (alerts on > 0.15 indicating systematic agent inadequacy)
• Outcome variance across user segments (alerts on > 2x variance, indicates the agent serves some users much worse than others)

The architecture stack we run

The 4 layers compose into a single observability stack that runs in production at scale.

Backbone: OpenTelemetry as the trace and metric transport layer. Every span from every layer flows through the same OTel pipeline. This decouples instrumentation from backend choice.

Layer-specific tooling on top: LangSmith or Phoenix for the LLM call and agent step layers when we want rich agent-specific UI. Datadog APM for the tool execution layer when we want integration with existing service maps. PostHog for the business outcome layer.

Storage: PostgreSQL for evaluation results and structured outcomes. ClickHouse or a similar columnar store for high-volume trace data. S3 for raw payload archives that need retention.

Alerting: Threshold-based alerts at each layer route to PagerDuty for incident-grade signals. Slack for trend signals that need investigation but not immediate response.

Sample rate strategy at scale: 100 percent of Layer 1 spans for the first 90 days, then sampled to 10 percent with 100 percent sampling on error spans. 100 percent of Layer 2 traces for any request that hits a defined risk threshold (medical, financial, legal context). 10 percent baseline sampling otherwise. Layer 3 schema validation always runs at 100 percent. Layer 4 outcome events are always at 100 percent.

Cost economics

Full 4-layer observability accounts for roughly 30-50% of the inference cost in our deployments. That number surprises teams the first time they hear it.

The breakdown: Layer 1 instrumentation is essentially free at the API cost level (you are already paying for the LLM calls). The cost lives in storage and querying for trace data, plus the eval pipeline costs for online evaluation. Layer 2 trace storage scales with agent complexity (multi-step agents generate more spans per request). Layer 3 is cheap unless you are running schema validation against very high-volume tool calls. Layer 4 PostHog or an equivalent has its own subscription cost but is decoupled from inference costs.

Where to economize without blinding production: drop Layer 1 sampling rates first (you need failure spans, not every successful one). Compress trace payloads before storage. Use shorter retention for raw payloads (30–90 days) and longer retention for aggregated metrics (1+ years).

Where not to economize: never drop schema validation at Layer 3, never sample 100 percent below on Layer 4 outcomes (you cannot reconstruct business signal from a sample), never skip alerting on the production thresholds above.

Which layer to instrument first under launch pressure

You cannot instrument all four layers on day one of production without delaying launch. Here is the phasing I default to.

Phase 1 (Day 1 of production): Layer 1 plus Layer 4. Get LLM call instrumentation via OpenTelemetry GenAI conventions, and have business outcome events emitted via PostHog or your existing product analytics. Layer 1 catches operational failures, Layer 4 catches the workflow-fit failures. This pair covers the widest failure surface at the lowest setup cost.

Phase 2 (Weeks 2–4): Add Layer 2. By this point, you have enough production traffic to start seeing anomalies in agent reasoning. The trace layer becomes diagnostically valuable once you have real user behavior to investigate.

Phase 3 (Week 4 onward, or after the first Layer 3 incident): Add Layer 3 schema validation and tool-execution tracing. The reason this is Phase 3 rather than Phase 1 is that most teams have not yet seen the silent-corruption failure mode, and the build cost feels disproportionate. Once you have lived through one Layer 3 incident, you build it without question.

The strict version of this advice is: instrument all four layers from day one if you are in healthcare, fintech, or any regulated industry. The phased approach is for consumer products and internal tools where the cost of a silent failure is recoverable.

Frequently asked questions

How does this 4-layer stack relate to the eval harness?

The eval harness I described in a recent Towards Data Science piece outlines what to measure: 12 metrics spanning retrieval, generation, agent behavior, and production health. The 4-layer instrumentation stack covers how to wire those measurements into the running system. Eval defines the thresholds, and instrumentation enforces them.

Do I need all four layers on day one?

In regulated industries, yes. In consumer products and internal tools, Phase 1 (Layer 1 plus Layer 4) is the minimum viable bar. Phases 2 and 3 follow as production traffic teaches you what to investigate.

LangSmith vs Phoenix vs custom?

LangSmith, if your agent runtime is LangChain or LangGraph, and you want vendor-supported tooling. Phoenix, if you want an open-source and richer trace analysis UI. Customize if your agent runtime is bespoke and the existing tools do not fit. All three work with OpenTelemetry as the underlying substrate.

How do regulated industries handle observability with HIPAA or GDPR?

Sample-rate strategy changes. You cannot log raw prompts and responses containing PHI to a third-party trace backend. The pattern we use: redact PHI from spans before they leave the application boundary, keep the redacted spans in your trace backend, and store unredacted payloads in your own HIPAA-compliant data store with shorter retention. The audit trail still works because the span IDs link the redacted trace to the unredacted source.

What is the sample rate strategy at high volume?

100 percent sampling on Layer 3 schema validation and Layer 4 outcome events, regardless of volume. Layer 1 and Layer 2 sampling drops with volume, but always 100 percent on error spans. The principle: sample successful behavior, never sample failures or business outcomes.

When should I use LLM-as-judge inside the observability stack?

For online faithfulness scoring at Layer 1 and online plan-coherence scoring at Layer 2. The cost discipline is to run LLM-as-judge against a sampled subset of production traffic (5–10 percent) rather than every request, and to use a cheaper model for the judge than for the generator.

How do I avoid Layer 4 false positives?

Layer 4 alerts are noisy by default because user behavior is noisy. The fix is to define outcome events specifically enough that the signal is interpretable. “User abandoned workflow” is too broad. “User abandoned workflow within 30 seconds of agent response without clicking any suggested action” is specific enough to alert on.

Five takeaways to close

1. LLM logging is necessary but not sufficient for agent observability. Production failures live in three layers that most teams do not instrument: the agent step trace, the tool execution boundary, and the business outcome layer.
2. The most expensive failure mode in production agents is a tool that returns HTTP 200 with a structurally valid but semantically wrong payload. Schema validation at the tool boundary is the alert mechanism that catches issues within minutes, not weeks.
3. Layer 4 (business outcome) is the most skipped and most needed layer. A model can hit 91 percent offline accuracy and still fail if users do not actually use the agent or achieve the result they came for. Nothing else catches that gap.
4. Full 4-layer observability typically runs 30 to 50 percent of inference cost. Economization decisions need to happen layer by layer, not as an overall budget cut. Never economize on Layer 3 schema validation or Layer 4 outcome events.
5. Do not instrument all four layers from day one in non-regulated contexts. Phase 1 (Layers 1 and 4) on day one, Layer 2 by week 2–4, Layer 3 after the first incident or before any high-stakes scaling. The phased rollout gets you to production faster and adds observability where you actually need it, rather than where you might.

The teams shipping AI agents that do not break are not running better models. They are running better instrumentation across all four layers.

For teams building production agent systems and looking to map their current instrumentation against the full stack, the Intuz team is happy to help.

Pratik K Rupareliya is the Co-Founder of Intuz, where he leads enterprise AI strategy across 100+ production deployments spanning healthcare, fintech, manufacturing, and retail. He recently published “Building an Evaluation Harness for Production AI Agents” on Towards Data Science (May 2026) and writes regularly on production AI architecture, agent systems, and AI economics. Connect with him on LinkedIn.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI