Author(s): Chhoeun K.Keaphirunphirakbot

Originally published on Towards AI.

Why Your Existing Observability Stack Will Lie to You

Imagine this: your AI service is having a bad day. Users are complaining. Outputs are wrong, inconsistent, off-topic. Your support queue is filling up.

The article explains why traditional observability (metrics for latency/errors) misses LLM “semantic failures” and then provides a practical DevOps playbook to close the gap without replacing existing tools. It starts with a structured logging schema for every inference request (including trace_id/app_id, model/prompt versions, token counts, queue/wait and generation timings, finish_reason, and an asynchronously populated quality_score). It then covers the dashboards and metrics you should build: reuse Prometheus/Grafana for infrastructure but add GPU and inference-specific signals (vLLM queue depth, KV cache utilization, and storage I/O during cold starts), and derive LLM-specific time-series from logs such as TTFT, token throughput, token distributions, refusal rate, and retry/regeneration rate. The dashboard is organized for quick health checks and “what changed in the last 24 hours,” with deployment markers overlaid so regressions are immediately attributable. Next, it outlines semantic monitoring: score real production outputs via a lightweight rules-based scorer first, then optionally a smaller “judge model” on a sampled, near-real-time basis, and feed the scores back into the same logs and dashboards. Finally, it provides a three-tier alerting strategy (page now/ticket later/monitor only) emphasizing paging on symptoms rather than causes, and concludes that most work is plumbing and incremental extensions that can drastically reduce time-to-diagnosis during incidents.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI