Author(s): Vasu Agrawal

Originally published on Towards AI.

RAG Didn’t Solve Hallucinations. It Just Moved the Problem Somewhere Harder to See.

After building three RAG systems from scratch, I stopped blaming the model. The real failure was always upstream — in data I trusted too much and retrieval I tested too little.

When I first learned about Retrieval-Augmented Generation, the pitch was clean: ground the model in facts and it stops making things up. Plug in your documents, build a vector index, and the hallucination problem mostly disappears.

I believed it. I built systems around it. Then I watched those systems confidently return wrong answers to users — answers that weren’t hallucinated in the traditional sense, because the model was technically pulling from the retrieved context. The context was just bad.

That’s when I realized: RAG doesn’t eliminate hallucination. It trades one failure mode for three quieter ones.

The Myth I Was Sold

The myth: Feed the LLM relevant documents at inference time, and the model stays grounded. It’ll cite sources, stay factual, and stop confabulating.

What actually happens: The model does stay grounded — to whatever it retrieved. If what it retrieved is stale, semantically mismatched, or poorly chunked, it generates a fluent, confident answer grounded in garbage.

This isn’t a model failure. The LLM is doing exactly what it’s supposed to. The failure is in the retrieval layer, which most early-stage teams (including me) spend a fraction of the time on compared to prompt engineering.

The Three Failure Modes Nobody Warned Me About

1. Semantic retrieval is not semantic understanding

Embedding similarity is a proxy for relevance, not a guarantee of it. When a user asks a specific operational question — say, “what’s the refund policy for enterprise contracts signed before 2023?” — the top-k retrieved chunks are often thematically related but factually insufficient.

The embedding model sees “refund” and “enterprise” and “contracts” and retrieves three chunks about general refund flows, a sales FAQ, and a blog post. The cosine scores look fine. The content is useless for the actual query.

Retrieval precision is not measured by cosine similarity. It’s measured by whether the retrieved chunks actually contain the answer. These are different things. Evaluate them separately.

My first real debugging session on this looked like this:

query = "enterprise refund policy pre-2023"
retrieved_chunks = retriever.get_relevant_documents(query)
for i, chunk in enumerate(retrieved_chunks):
 print(f"--- Chunk {i} (score: {chunk.metadata['score']:.3f}) ---")
 print(chunk.page_content[:200])
 print()

Scores: 0.87, 0.84, 0.82. All above threshold. All useless for the specific question. The model synthesized them into a confident, wrong answer. I had been logging scores, not content quality. That was the bug.

2. Chunk boundaries are an engineering decision, not a default setting

Most RAG tutorials tell you to chunk at 512 or 1024 tokens with some overlap and move on. For toy demos, this works. For documents with structure — contracts, technical specs, medical records — naive chunking quietly destroys context.

Consider a legal clause that spans a page break. Split at 512 tokens, it becomes two semantically incomplete fragments. The clause’s meaning — the part your users are actually asking about — is split across two chunks that may never co-retrieve. Neither chunk contains the full condition. Your model answers with half the truth.

What helped: moving from fixed-size chunking to structure-aware chunking.

from langchain.text_splitter import RecursiveCharacterTextSplittersplitter = RecursiveCharacterTextSplitter(
 chunk_size=800,
 chunk_overlap=150,
 separators=["\n\n", "\n", ". ", " "]
)# For structured docs: split at logical boundaries first
sections = extract_sections_by_heading(document)
chunks = []
for section in sections:
 section_chunks = splitter.split_text(section.content)
 for c in section_chunks:
 chunks.append({
 "text": c,
 "section": section.title,
 "doc_id": document.id
 })

This isn’t a silver bullet either — it required document-type-specific logic. But retrieval precision on structured queries improved measurably once chunks respected semantic boundaries instead of token counts.

3. Stale context is invisible at query time

Vector databases don’t know when something is outdated. A product specification indexed six months ago sits in your store with the same embedding as the current one. If both are in the index, whichever scores higher wins. If only the old one is indexed because your ingestion pipeline missed a re-upload, the model answers from it confidently.

I hit this during a demo. The model quoted pricing that had been deprecated three months prior. The chunk looked relevant. The embedding distance was low. The information was simply wrong — but nothing in the pipeline flagged it.

Stale retrieval is worse than no retrieval. A model that says “I don’t know” is recoverable. A model that says the wrong thing with citations is a support ticket — or worse, a trust problem.

Minimum viable fix: add ingested_at and doc_version metadata to every chunk, and build a decay-aware retrieval filter for time-sensitive domains.

results = vectorstore.similarity_search_with_score(
 query,
 filter={
 "ingested_at": {"$gte": cutoff_timestamp},
 "doc_type": {"$in": allowed_doc_types}
 },
 k=5
)

What I Actually Changed After Accepting This

Once I stopped treating RAG as “retrieval solves hallucinations,” three things shifted in how I build:

• Evaluate retrieval independently from generation. Build a test set of (query, expected_chunk_ids) pairs. Measure retrieval recall before you ever look at LLM output quality. A pipeline with 60% retrieval recall cannot produce reliable answers — no amount of prompt engineering fixes that.
• Treat your document pipeline like a data pipeline. Ingestion quality, deduplication, version tracking, and staleness handling aren’t nice-to-haves. They’re the product. More of my debugging time is now spent here than in the model layer.
• Add a retrieval confidence gate. If max chunk similarity is below a threshold, don’t synthesize — surface the uncertainty explicitly. “I couldn’t find a confident answer in the knowledge base” is more trustworthy than a fluent hallucination dressed in citations.

The Uncomfortable Conclusion

RAG is genuinely useful. I’m not arguing against it. I’m arguing that the community narrative around it — that it “solves” hallucination — led early engineers like me to under-invest in the retrieval and data pipeline layers for too long.

The model is often the least buggy part of a RAG system. The document store, the chunking logic, the embedding freshness, the retrieval evaluation framework — those are where the real quality lives, and where most teams are flying blind.

If you’re building RAG and you’ve never measured retrieval precision independently of generation quality, that’s the next thing to fix. Not the prompt. Not the model. The retrieval.

Key takeaways:

• RAG shifts the hallucination problem from the model to the retrieval layer — it doesn’t remove it.
• Cosine similarity scores measure embedding distance, not answer quality. Evaluate them separately.
• Naive chunking silently destroys context in structured documents. Match chunk strategy to document type.
• Stale vector index entries are invisible and dangerous. Track ingestion timestamps and filter by them.
• Build a retrieval eval set before you tune prompts. Retrieval quality is the ceiling on generation quality.

Written from 6–18 months of building RAG systems across NLP and document-heavy domains. Part of a series on what actually breaks in applied AI — not what the tutorials warn you about.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI