Author(s): Hubert Nakielski

Originally published on Towards AI.

Your AI Agent Has No Idea What It Knows

Imagine a filing cabinet with half a million documents about home renovation. Somewhere inside sits one file about a football club.

Now someone asks your AI assistant: “Do we have anything about the club’s sponsorship?”

The assistant has a search tool pointed at that cabinet. Should it use it? It knows the tool exists. It has no idea what’s behind it. So it guesses. Sometimes it searches a renovation archive for football news; sometimes it answers confidently from memory while the one document that matters sits unread in the cabinet.

Remember that football file. It comes back near the end, where it almost breaks one of the techniques.

First: what is a “data store”?

The term sounds like database furniture, so let’s pin it down.

A data store — OpenAI says vector store, Google says data store, others say knowledge base — is a box of files an AI can search. You upload PDFs, wikis, contracts, chat logs. Each file is split into chunks, every chunk gets an embedding (a numerical fingerprint of its meaning), and a tool like file_search retrieves the chunks whose fingerprints resemble the question.

Two properties of the box matter here. It’s opaque: the agent sees a name like customer-docs and maybe a one-line description. And it's alive: users add and delete files whenever they want, so last month's truth about the contents probably isn't this month's.

The decision nobody engineers

Every message your agent receives triggers a quiet decision: call the search tool, or answer from what the model already knows. Teams pour energy into what happens after that decision — chunking, rerankers, hybrid search. The decision itself usually rests on one hand-written sentence, drafted months ago, describing content that has changed daily ever since.

Guessing wrong costs both ways. Needless retrieval actively hurts: irrelevant retrieved text can flip otherwise-correct answers, and the worst passages are exactly what a pointless search returns — topically related text with no answer in it. Skipping fails too, because model memory is weakest on the long-tail private facts data stores exist to hold.

The research on this decision — Self-RAG, Adaptive-RAG, FLARE — almost all works the query side: is the question hard, is the model uncertain. Almost nothing asks the data side: what is in the store right now?

That signal is worth real accuracy. Elastic measured it: an agent picking which index to search scored 77% from index names alone, and over 90% once each index carried a stored description. Same model, same data. The only change was telling the agent what’s inside.

Everyone’s answer today is a hand-written sentence

Survey what platforms actually give you for this and a pattern appears:

• OpenAI’s vector stores have a name and developer-facing metadata, and no description field at all. Nothing about content is surfaced to the model.
• Anthropic’s guidance says tool descriptions are “by far the most important factor in tool performance” and that small refinements yield dramatic improvements. You still author them by hand.
• MCP resources carry a description and even change notifications, but nothing regenerates the description when the content behind it drifts.
• LlamaIndex routers select engines from hand-written “Useful for…” strings.
• Google’s Dialogflow CX routes to data stores using, again, hand-written descriptions.

Meanwhile the same design keeps getting reinvented wherever routing works well. llms.txt is a precomputed site summary for LLMs. Aider’s repo map is a token-budgeted summary of a codebase. Anthropic’s Agent Skills preload a 30-to-100-token description and pull the payload on demand. It’s the same pattern five times: a precomputed, bounded summary used as a routing signal.

Every version shares the same missing half: who writes the summary, and who keeps it true when the data changes? For a store fed by user uploads, “a developer writes it” is not an answer. The summary has to be derived from the data, and re-derived when the data moves.

The constraints, with real numbers

Everything below was priced against a real production deployment: 80 non-empty data stores holding 512,287 files, 1.4 million chunks and 722 million tokens. One store dominates, with 511,962 of those files and 94% of the tokens. The largest single file is a 6.7-million-token book.

The naive fix — dump the store’s file list into the router’s prompt — dies on those numbers: half a million slow rows per decision, file names like scan_0042_final_v3.pdf carrying zero signal, and bloated context degrading the model's own reasoning.

So the summary gets hard constraints. Under ~200 tokens, because a router LLM reads it on every turn. Free to read — a plain column read, zero LLM calls. Updated on file events from the ingestion pipeline, never on user traffic (a stable summary also keeps the router’s prompt cacheable, and Manus reports a 10x cost gap between cached and uncached input). And nothing whose LLM cost grows with file count, because a store this size would bankrupt it.

Five ways to summarize a corpus, priced honestly

Five strategies, priced with a Haiku-class model ($0.80/M input tokens). Backfill means summarizing all 80 stores once.

Two die immediately. Strategy A — abstract every file with an LLM — costs about $1,000 in backfill plus an LLM call for every future upload, forever. Strategy C — extract topics from every chunk — lands near $864. Both scale with corpus size; Microsoft’s LazyGraphRAG post prices this style at roughly 1000x vector-RAG indexing cost. The lesson from both corpses: never let the LLM read the corpus. Something cheaper has to read it first and hand the LLM a compressed view.

Strategy B: cluster the embeddings you already paid for — $2.96

The ingestion pipeline already embedded every chunk into a vector database. Those 1.4 million vectors are a sunk cost, and they already encode topical structure. So: pull the store’s vectors (sample down to 50k), run KMeans into ~30 clusters, take the 3 chunks nearest each centroid, and make one small LLM call per cluster: “label this in 2–5 words.” Write the labels to a topics column. Done.

Backfilling all 80 stores: $2.96, or $0.037 per store. LLM cost grows with topics, not files — a 512k-file store costs the same to summarize as a 12-file one, because both reduce to ~30 clusters before any LLM sees a word.

If this looks familiar, it’s BERTopic’s embed-cluster-represent recipe, and a single level of RAPTOR, which clusters chunk embeddings and LLM-summarizes each cluster recursively into a tree. RAPTOR builds that hierarchy to retrieve from; here, one level of it becomes a routing signal.

Strategy D: count words like it’s 1972 — $0.10

The contrarian option: no embeddings, no semantics at all. Tokenize chunks in the ingestion pipeline (lowercase, stopwords, stemming), keep per-file term counts plus a store-wide document-frequency table, and let one LLM call turn the top terms into labels. Backfill for all 722 million tokens: $0.10. If RAPTOR is the maximalist pole of corpus summarization — a whole tree of LLM calls — this is the opposite one: pure counting, a single call at the very end.

Now the football file. Raw counts bury it:

Top terms by count: renovation (4,012,118), wall (1,802,394), permit, contractor, drywall… football is nowhere. The most distinctive file in the store is invisible.

The fix is ranking twice. One list by raw count, for the dominant themes. One by count × IDF, where IDF asks: how rare is this term across the store’s own files? Football’s 200 mentions of a term confined to a single file among half a million score near the top of the novelty list. The sponsorship question from the opening is now answerable.

The honest downsides: bag-of-words can’t see synonyms (renovation and remodeling are strangers to it), and the novelty list promotes noise too — employee first names float up, dense in one file and absent across the rest. Cheap’s price is paid in label quality.

Strategy B+D: the hybrid — $3.45

The two signals fail differently, which is why they combine. Embeddings answer “what is this text about”; keyword statistics answer “what is dense and distinctive here.” The hybrid labels each cluster from its representative chunks plus its top count×IDF keywords — same number of LLM calls as B, richer prompt, $3.45 total.

The gain shows on fuzzy clusters. A real one, full of Java/Spring/Hibernate/Maven content:

• B alone: “Java ecosystem.” True but generic.
• D alone: “Programming languages.” Too broad.
• B+D: “JVM backend development.” Precise.

The hard part: keeping it true

Generating a summary once is a weekend project. The moving data is the real problem — and where published techniques quietly give up. GraphRAG re-summarizes changed communities, but document removal is explicitly out of scope. RAPTOR’s summary tree requires full recomputation after changes; follow-up work patches additions, not deletions. BERTopic’s online mode doesn’t support deletion either.

So freshness has to be designed. The skeleton: a file event marks the store dirty, a ~60-second debounce absorbs bursts (50 uploads trigger one regeneration, not fifty), a cheap incremental update runs, and a drift gate decides whether an LLM gets involved at all.

• B: assign new chunks to the nearest existing centroid — absorbed silently, $0, no LLM. An outlier re-labels just its own cluster: one small LLM call, about a tenth of a cent. Re-labeling all ~30 clusters of a store costs ~$0.04. A full re-cluster fires only when outliers pass 5%.
• D: add the file’s term counts on upload; subtract the stored per-file map on delete. Counting is the rare technique where deletion is exact — the very operation RAPTOR and GraphRAG can’t do incrementally is a subtraction here. Labels refresh when top terms shift by more than 5%.

If a labeling call fails, the old topics stay in place and a retry fires with backoff. Stale-but-present beats fresh-but-flaky for a signal read on every turn.

Steady state: single-digit dollars per week for the store, scaling with file churn — not query traffic, not corpus size.

Did the clusters actually work?

Two findings from running it for real. HDBSCAN behaves well on small, clean, mixed content — clear clusters, an honest refusal to classify the ambiguous quarter of chunks. But on a 20k-vector sample of the store it collapsed 70.5% of everything into one giant blob: two usable topics for half a million files. KMeans with k=10 forced out ten usable topics (~120 tokens) by assigning every point, noise included. Decisive beat honest; KMeans shipped, with silhouette-picked K.

That giant blob is also where the hybrid earns its keep. An embedding summary of a homogeneous store says “enterprise support docs” — but users ask about clients by name, and names are exactly the dense-and-distinctive terms the IDF list promotes. Embeddings see themes; keywords see names.

Verdict

Ship Strategy B first: $2.96 to backfill all 80 stores, at most ~$0.04 per store refresh, no new dependency beyond a clustering library. B+D is the pre-planned upgrade the moment label-quality evals complain — 16% more cost, an addition rather than a rewrite. D is the fallback where embeddings don’t exist.

Three things worth keeping:

1. The retrieve-or-not decision is starving for data-side signal. A stored description was worth 77% → 90% routing accuracy in the one public benchmark that measured it.
2. A data-store summary is derived data, not documentation. Treat it like a materialized view — event-driven, debounced, drift-gated, with deletion designed in, because RAPTOR, GraphRAG, and BERTopic won’t handle it for you.
3. Never let the LLM read the corpus. Anything O(files) or O(chunks) in LLM calls is a cost bomb waiting for its first big store.

This pattern — a precomputed, token-budgeted, machine-maintained summary as a routing signal — keeps getting reinvented as llms.txt, repo maps, skill descriptions, index descriptions. It will keep being reinvented until platforms treat it as infrastructure. Until then, you can build it yourself for the price of a coffee.

And your agent will finally know about the football file.

Numbers come from pricing five prototype strategies against a live production deployment of 80 data stores, using a Haiku-class labeling model at late-2025 rates. Your data will differ; the scaling arguments won’t.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI