Author(s): EMMANUEL NWANGUMA

Originally published on Towards AI.

The answer is: often not. And catching it turned out to cost nothing.

You have probably seen this pattern. You ask an AI something, it gives you an answer, and underneath there’s a little citation — a document name, a page number, maybe a link. It looks rigorous. It looks like the kind of thing you could check.

The author explains that AI citations are often “citation-shaped” but not actually supported, and describes building a verification system to check whether cited claims truly match the exact quoted text in the underlying documents. Using real regulatory filings, they show that retrieval-based approaches are cheaper but don’t reliably pick the current policy when documents contradict each other over time, while long-context approaches can be worse at identifying what’s current. They report a “free” verification ladder—string comparisons to catch literal quote issues and key-word checks, with only a small additional AI step to confirm passage support—then validate the judge and measure how confidence scores behave (including poor calibration at low confidence). The piece also documents UI and measurement bugs found via screenshots and “suspicious cleanliness” checks, discusses what can transfer beyond the specific benchmark, and concludes with a clear takeaway: require verbatim quoted evidence, allow declining when sources don’t support answers, validate any AI used for grading, and treat caveats as claims worth verifying; the underlying system and data are open source.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI