Author(s): Moiz Ezzy

Originally published on Towards AI.

Why Your LLM App Will Fail at 3AM (And How to Build One That Won’t)

We shipped our LLM feature on a Tuesday. By Thursday 3AM, it was returning hallucinated answers with 100% confidence. No errors in the logs. No alerts fired. Latency was perfect. The model was working exactly as designed and completely wrong.

That’s the failure mode nobody warns you about before you put an LLM in production.

Traditional APM tools Datadog, New Relic, Prometheus tell you whether your system worked. LLM observability tells you whether it worked well. Those are not the same thing. A language model can serve a response in 200ms with a 200 status code and still answer the wrong question, cite a deleted document, or quietly ignore your system prompt after a code change touched it.

If you’ve already built a RAG pipeline and want the deep-dive on retrieval failures specifically, I covered the 5 ways RAG pipelines break in production here. This article goes wider from model selection through to the 3AM runbook.

The 2026 Model Landscape: Pick the Right Tool First

Before any code, the most consequential decision you’ll make is which model to call. In 2026 the choice has stopped being “which is best” and started being “which fits this workload.”

Here’s where the frontier models actually specialise:

* Gemini 2.5 Pro input jumps to ~$2.50/1M once your prompt crosses 200k tokens relevant if you’re stuffing large retrieved context. Every price here is input-only, per 1M tokens; output runs 4–6x higher. Prices move fast check each provider’s pricing page before you commit a budget to a spreadsheet.

For most RAG applications: Gemini 2.5 Pro or GPT-5.5. The 1M context window on Gemini means you can stuff significantly more retrieved chunks before hitting limits.

For coding assistants or agentic systems: Claude Opus 4.8. It’s the current front-runner on SWE-bench Verified for code reasoning check the live leaderboard before you quote a number, because the top spot changes every few weeks.

For high-volume pipelines where cost matters: Gemini Flash 2.5 at $0.30/1M tokens. Two teams building similar applications can end up with 10x different AI costs based purely on which model tier they chose Opus at $5/1M vs Flash at $0.30/1M is nearly a 17x spread on input. At 100M tokens/day, that’s the difference between roughly $15,000/month and $900/month. Run your own numbers before you commit; the formula below does it in three lines.

The budget decision formula:

def monthly_ai_cost(daily_requests: int, avg_tokens_per_request: int, price_per_million: float) -> float:
 """Estimate monthly LLM API cost."""
 monthly_tokens = daily_requests * avg_tokens_per_request * 30
 return (monthly_tokens / 1_000_000) * price_per_million
# Example: 50,000 requests/day, 2,000 tokens each
print(monthly_ai_cost(50_000, 2_000, 5.00)) # Claude Opus: $15,000/mo
print(monthly_ai_cost(50_000, 2_000, 1.25)) # Gemini 2.5 Pro: $3,750/mo
print(monthly_ai_cost(50_000, 2_000, 0.30)) # Gemini Flash: $900/mo

Run this before you pick a model. The cost difference is real.

Building the Application: The Production Stack

In 2026, most production LLM teams use LangChain for orchestration and LlamaIndex for the retrieval layer underneath.

Pure RAG over documents → LlamaIndex. RAG + agents + tools → LangChain on top.

Here’s a full working application stack. Not a toy snippet — this is close to what we run.

Step 1: The document ingestion pipeline

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
from llama_index.vector_stores.postgres import PGVectorStore
from llama_index.embeddings.openai import OpenAIEmbedding
import psycopg2
# Connect to pgvector (Postgres with vector extension)
# pgvector is the right call for most teams - no separate infra,
# ACID guarantees, and it handles 50+ concurrent requests on a single node
connection = psycopg2.connect(
 host="localhost",
 database="llm_app",
 user="postgres",
 password="your_password"
)
vector_store = PGVectorStore.from_params(
 host="localhost",
 port=5432,
 database="llm_app",
 user="postgres",
 password="your_password",
 table_name="document_embeddings",
 embed_dim=1536, # text-embedding-3-small dimensions
)
# Load and chunk documents
documents = SimpleDirectoryReader("./docs").load_data()
# Chunk size matters. 512 tokens is the sweet spot for most RAG workloads:
# too small = retrieval misses context, too large = noise drowns signal
node_parser = SentenceSplitter(
 chunk_size=512,
 chunk_overlap=50, # overlap prevents cutting sentences at boundaries
)
# Build the index
embed_model = OpenAIEmbedding(model="text-embedding-3-small")
index = VectorStoreIndex.from_documents(
 documents,
 vector_store=vector_store,
 embed_model=embed_model,
 transformations=[node_parser],
 show_progress=True,
)
print(f"Indexed {len(documents)} documents into pgvector")

Step 2: The query pipeline with LangChain

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from llama_index.core import QueryBundle

# The system prompt is production code, not a demo prompt.
# Version it. Track changes. Treat it like a config file.

SYSTEM_PROMPT = """You are a technical support assistant for our platform.
Answer questions using ONLY the context provided below.
If the context does not contain the answer, say "I don't have information on that."
Do not speculate. Do not use knowledge outside the provided context.
Context: {context}"""
def get_retriever(index, similarity_top_k: int = 4):
 """Return a retriever that fetches top-k similar chunks."""
 return index.as_retriever(similarity_top_k=similarity_top_k)
def format_docs(nodes) -> str:
 """Format retrieved nodes into a single context string."""
 return "\n\n---\n\n".join([node.text for node in nodes])
# Build the chain
llm = ChatOpenAI(
 model="gpt-5", # or "gemini-2.5-pro" via langchain_google_genai
 temperature=0, # 0 = deterministic, critical for factual Q&A
 max_tokens=1024,
 request_timeout=30, # explicit timeout - never trust the default
)
prompt = ChatPromptTemplate.from_messages([
 ("system", SYSTEM_PROMPT),
 ("human", "{question}"),
])
retriever = get_retriever(index)
# LCEL chain: question → retrieve → format → prompt → LLM → parse
chain = (
 {
 "context": lambda x: format_docs(retriever.retrieve(x["question"])),
 "question": RunnablePassthrough(),
 }
 | prompt
 | llm
 | StrOutputParser()
)
# Usage
response = chain.invoke({"question": "How do I reset my API key?"})
print(response)

The Failure Modes Nobody Tells You About

The code above works in a demo. Here’s what breaks it in production.

Failure 1: Hallucination with no signal

The model returns a confident, well-formatted answer sourced from a document you deleted 3 days ago. The embedding is still in your vector store. Your deletion pipeline didn’t clean it up.

No error. No alert. Latency: 180ms. Status: 200.

The fix: hard delete from your vector store on document deletion.

from llama_index.vector_stores.postgres import PGVectorStore
def delete_document_embeddings(doc_id: str, vector_store: PGVectorStore):
 """
 Remove all embeddings for a document when it's deleted or updated.
 Call this BEFORE re-ingesting updated documents.
 Without this, you get stale embeddings serving outdated answers.
 """
 vector_store.delete(ref_doc_id=doc_id)
 print(f"Deleted embeddings for document: {doc_id}")
# Hook this into your document management system
# Webhook on document delete → delete_document_embeddings()

Failure 2: System prompt drift

Your code was refactored. Someone moved the system prompt definition. The new version no longer includes the “only answer from context” instruction. The model starts answering from its training data. Nobody notices for 4 days because the answers sound correct.

This happened to us. The answers were plausible but wrong the model was substituting its general knowledge for outdated product-specific context.

The fix: version your prompts and test them in CI.

# prompts.py — treat this like application config, not inline strings
from dataclasses import dataclass
from datetime import datetime
@dataclass
class PromptVersion:
 version: str
 created_at: str
 template: str
 notes: str
PROMPTS = {
 "v1.0": PromptVersion(
 version="v1.0",
 created_at="2026-06-01",
 template="You are a helpful assistant. Context: {context}",
 notes="Initial version - too permissive, allows hallucination"
 ),
 "v1.1": PromptVersion(
 version="v1.1",
 created_at="2026-07-01",
 template="""You are a technical support assistant.
Answer ONLY from the provided context.
If unsure, say 'I don't have information on that.'
Context: {context}""",
 notes="Added hard constraint - significantly reduced hallucination rate"
 ),
}
ACTIVE_PROMPT = PROMPTS["v1.1"]

Failure 3: Token window overflow silent truncation

Your retriever fetches 10 chunks, each 512 tokens. Plus system prompt (300 tokens), user question (50 tokens), conversation history (800 tokens).
Total: ~6,250 tokens. But you capped max_tokens at 1,024 for the response and you're on a model tier with an 8k context window and nobody did the arithmetic on input + output together.

Best case, the API rejects the request with a context-length error and your endpoint 500s. Worse case an older self-hosted model or a middleware layer that truncates for you the last chunks silently fall off the end. Either way the most relevant retrieved context never reaches the model, and the failure shows up as “answers got worse,” not as an error in your logs.

import tiktoken
def check_context_window(
 system_prompt: str,
 context: str,
 question: str,
 model: str = "gpt-5",
 max_tokens: int = 8192,
) -> dict:
 """
 Validate total token count before sending to the model.
 Raise early instead of silently truncating.
 """
 enc = tiktoken.encoding_for_model("gpt-4") # approximation for GPT-5
 
 total = (
 len(enc.encode(system_prompt)) +
 len(enc.encode(context)) +
 len(enc.encode(question))
 )
 
 buffer = 1024 # reserve tokens for the response
 
 return {
 "total_tokens": total,
 "max_allowed": max_tokens - buffer,
 "within_limit": total <= (max_tokens - buffer),
 "overflow_by": max(0, total - (max_tokens - buffer)),
 }
# Usage - call this before every LLM invocation
result = check_context_window(
 system_prompt=ACTIVE_PROMPT.template,
 context=format_docs(retrieved_nodes),
 question=user_question,
)
if not result["within_limit"]:
 # Reduce top_k or truncate context, don't silently overflow
 raise ValueError(
 f"Context window overflow: {result['overflow_by']} tokens over limit. "
 f"Reduce similarity_top_k or truncate documents."
 )

Failure 4: Thundering herd on LLM API rate limits

Your service gets a traffic spike. 500 concurrent requests hit your LLM API simultaneously. OpenAI returns 429 (rate limit exceeded). Your retry logic has no jitter all 500 requests retry at exactly the same interval. You hit the rate limit again. Repeat until the queue is exhausted or you’ve burned your error budget.

import asyncio
import random
from openai import AsyncOpenAI, RateLimitError
client = AsyncOpenAI()
async def call_llm_with_backoff(
 messages: list,
 model: str = "gpt-5",
 max_retries: int = 4,
) -> str:
 """
 LLM call with exponential backoff + full jitter.
 Prevents synchronized retry storms under rate limiting.
 """
 for attempt in range(max_retries):
 try:
 response = await client.chat.completions.create(
 model=model,
 messages=messages,
 temperature=0,
 max_tokens=1024,
 )
 return response.choices[0].message.content
 except RateLimitError:
 if attempt == max_retries - 1:
 raise
 # Full jitter: random sleep up to cap
 # Prevents all retries from firing at the same time
 cap = 60 # max 60 second wait
 sleep_time = random.uniform(0, min(cap, 1 * (2 ** attempt)))
 
 print(f"Rate limited. Attempt {attempt + 1}/{max_retries}. "
 f"Waiting {sleep_time:.1f}s before retry.")
 await asyncio.sleep(sleep_time)
 
 raise RuntimeError("Max retries exceeded on LLM API call")

The Observability Stack: What to Monitor

This is where most teams get it completely wrong. They track HTTP status codes and latency, declare it “monitored,” and go home.

That’s not observability. That’s infrastructure monitoring.

LLM observability requires an entirely different layer. The failure modes above hallucination, prompt drift, context truncation, degraded retrieval quality all look fine on your Datadog latency dashboard.

The four signals for LLM applications

from prometheus_client import Counter, Histogram, Gauge
import time
# 1. Latency - time-to-first-token and total generation time
LLM_LATENCY = Histogram(
 "llm_request_duration_seconds",
 "LLM API call duration",
 ["model", "endpoint"],
 buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0],
)
# 2. Token consumption - drives cost and context window awareness
TOKEN_USAGE = Counter(
 "llm_tokens_total",
 "Total tokens consumed",
 ["model", "token_type"], # token_type: input / output
)
# 3. Retrieval quality - are we fetching relevant context?
RETRIEVAL_SCORE = Histogram(
 "retrieval_similarity_score",
 "Cosine similarity of retrieved chunks to query",
 ["index_name"],
 buckets=[0.5, 0.6, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0],
)
# 4. Answer quality - the metric most teams skip
ANSWER_CONFIDENCE = Gauge(
 "llm_answer_confidence",
 "LLM judge score for answer quality (0-1)",
 ["model", "query_type"],
)
class InstrumentedLLMClient:
 """Wraps LLM calls with full observability instrumentation."""
 def __init__(self, model: str):
 self.model = model
 self.client = AsyncOpenAI()
 async def chat(self, messages: list, query_type: str = "general") -> dict:
 start = time.time()
 response = await call_llm_with_backoff(messages, model=self.model)
 duration = time.time() - start
 # Record every call
 LLM_LATENCY.labels(
 model=self.model,
 endpoint="chat"
 ).observe(duration)
 return {"response": response, "latency": duration}The LLM judge: catching hallucinations automatically

Standard monitoring tells you nothing about answer quality. The only way to catch hallucinations at scale is to use a second LLM call as a judge, a model that evaluates whether the answer is actually grounded in the retrieved context.

async def evaluate_answer_grounding(
 question: str,
 context: str,
 answer: str,
 judge_model: str = "gpt-5", # use a capable model for judging
) -> dict:
 """
 LLM-as-judge: evaluates whether the answer is grounded in the context.
 Returns a score 0-1 and a reason.
 Score < 0.7 = likely hallucination. Alert on this.
 """
 judge_prompt = f"""You are evaluating whether an AI answer is properly grounded in the provided context.

Question: {question}

Context provided to the AI:
{context}

AI Answer:
{answer}

Evaluate strictly:
1. Does the answer use ONLY information from the context? (0.0 = uses outside knowledge, 1.0 = fully grounded)
2. Is the answer factually consistent with the context? (0.0 = contradicts context, 1.0 = fully consistent)

Respond with JSON only:
{{"grounding_score": 0.0-1.0, "consistency_score": 0.0-1.0, "reason": "brief explanation"}}"""
 response = await client.chat.completions.create(
 model=judge_model,
 messages=[{"role": "user", "content": judge_prompt}],
 temperature=0,
 response_format={"type": "json_object"},
 )
 
 import json
 result = json.loads(response.choices[0].message.content)
 
 # Record to Prometheus
 overall_score = (result["grounding_score"] + result["consistency_score"]) / 2
 ANSWER_CONFIDENCE.labels(
 model=judge_model,
 query_type="rag_grounding"
 ).set(overall_score)
 
 # Alert threshold: below 0.7 = probable hallucination
 if overall_score < 0.7:
 print(f"⚠️ Low grounding score: {overall_score:.2f}. "
 f"Reason: {result['reason']}")
 
 return result

The cost of LLM judges: Yes, you’re calling the LLM twice per request. At Gemini Flash prices ($0.30/1M), judging 10,000 answers/day with a 500-token judge prompt costs about $1.50/day. That’s cheaper than one customer support ticket for a hallucinated answer.

Don’t judge every request in production. Sample 5–10%. That’s enough to catch systematic failures before they become incidents.

Putting It Together: The Production Architecture

Here’s what the full system looks like:

User Request
 │
 ▼
┌─────────────────┐
│ FastAPI App │ ← Rate limiting, auth, request validation
└────────┬────────┘
 │
 ▼
┌─────────────────┐
│ Token Budget │ ← Check context window before calling LLM
│ Check │
└────────┬────────┘
 │
 ▼
┌─────────────────┐ ┌──────────────────┐
│ LlamaIndex │────▶│ pgvector │
│ Retriever │ │ (embeddings) │
└────────┬────────┘ └──────────────────┘
 │
 ▼
┌─────────────────┐
│ LangChain │ ← Versioned prompt + model call with backoff
│ Chain (LLM) │
└────────┬────────┘
 │
 ▼
┌─────────────────┐
│ LLM Judge │ ← Sample 5-10% of responses for grounding check
│ (async) │
└────────┬────────┘
 │
 ▼
┌─────────────────┐
│ Prometheus │ ← Latency, tokens, retrieval score, answer quality
│ + Grafana │
└─────────────────┘

The 3AM Runbook for LLM Applications

When your LLM application pages you at 3AM, your first 10 minutes look different from any other service.

Standard service failure: check error rates → check recent deploys → rollback.

LLM application failure: the alert might not be an error rate. It’s a quality degradation that your users noticed before your monitoring did.

# Step 1: Is this a model provider outage?
# Check status pages before doing anything else
# https://status.openai.com
# https://status.anthropic.com
# https://status.cloud.google.com

# Step 2: Check your LLM-specific metrics
# If you wired up the Prometheus metrics above, you can query:
# - llm_request_duration_seconds P99 (is latency spiking?)
# - llm_tokens_total rate (are requests going through?)
# - llm_answer_confidence (has quality dropped suddenly?)
# - retrieval_similarity_score (is retrieval degraded?)

# Step 3: Sample recent prompt/response pairs
# Every LLM call should be logged with: prompt, context, response, latency, tokens
# Pull the last 20 from the past 30 minutes and read them manually
# This takes 5 minutes and catches what no metric can

# Step 4: Check for prompt drift
git log --since="48 hours ago" -- prompts.py # did the prompt change?
git diff HEAD~1 prompts.py # what changed?

# Step 5: Check vector store for stale embeddings
# If a major document update shipped recently, stale embeddings may be serving bad context
SELECT COUNT(*), MIN(created_at), MAX(created_at)
FROM document_embeddings
WHERE created_at < NOW() - INTERVAL '7 days';

What Changes When You Use Open-Source Models

Everything above works with OpenAI, Anthropic, or Google APIs.
If you’re self-hosting Llama 3.3, Mistral Large 2, or DeepSeek V4, the architecture is the same the operational surface expands significantly.

Self-hosting only makes financial sense above roughly $50,000/month in API costs. Below that, the GPU infrastructure, the model serving maintenance (vLLM or TGI), the reliability engineering, and the upgrade cycle cost more than the API savings.

If you do go self-hosted:

# Replace the OpenAI client with an Ollama or vLLM endpoint
from openai import AsyncOpenAI
# vLLM serves an OpenAI-compatible API
# Just change the base_url - everything else stays the same
client = AsyncOpenAI(
 base_url="http://your-vllm-server:8000/v1",
 api_key="not-required-for-local",
)
# Same call, different endpoint
response = await client.chat.completions.create(
 model="meta-llama/Llama-3.3-70B-Instruct", # your deployed model
 messages=[{"role": "user", "content": "Hello"}],
)

The observability layer stays identical. The failure modes change slightly — you now own model loading failures, GPU OOM errors, and model version management on top of everything above.

Key Takeaways :-

• Pick your model based on workload, not hype. At 50k requests/day of ~2k tokens each, Claude Opus runs ~$15,000/month and Gemini Flash ~$900 — a $14,000+ gap on the same traffic. Run the numbers before you commit.
• pgvector over Pinecone for most teams. No extra infra, ACID guarantees, handles 50+ concurrent requests on a single node.
• LangChain for orchestration, LlamaIndex for retrieval. Don’t pick one most production systems use both.
• Hard-delete embeddings when documents are deleted. Stale embeddings serving deleted content is the most common silent failure in RAG applications.
• Version your prompts. They’re production code. Treat them like it.
• Check the token budget before every LLM call. Silent truncation degrades quality with no error signal.
• Exponential backoff with full jitter. Rate limit errors with synchronized retries will take your service down faster than the rate limit itself.
• LLM judge on 5–10% of responses. It’s the only way to catch hallucinations at scale. At Gemini Flash prices it costs less than $2/day.

If you found this article helpful, a clap would help Towards AI share it with more engineers building in this space.

The next article covers LLM observability in depth: exactly what to put in Grafana dashboards so you can tell the difference between a model provider hiccup and actual quality degradation.

If you’re earlier in the journey and building your first RAG pipeline, start with Building a RAG Pipeline That Doesn’t Fall Apart first.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI