Why Won’t AI Just Say “I Don’t Know”?
Last Updated on August 24, 2026 by Editorial Team
Author(s): Delini
Originally published on Towards AI.
The answer isn’t that it can’t tell. It’s that we spent three years training it not to.
Ask a chatbot something it has no way of knowing: the birthday of a stranger, the contents of a document it has never seen, the ruling in a case that was never decided, and watch what happens. It answers. Fluently. With a date, a summary, a citation. It does not hesitate, hedge, or stop to tell you that it is out of its depth.
After the introduction, the article argues that AI’s reluctance to say “I don’t know” is not a lack of capability but a training-and-evaluation incentive: models are shaped to track confidence and uncertainty early on, then are deliberately trained to suppress honesty because benchmarks and leaderboards reward answers (even bluffing) more than abstaining. It explains why fabricated facts arise—grading rules treat “validity” as something the model must decide for every candidate statement, guaranteeing errors under certain data-frequency conditions—and backs the claim with a formal paper (published in Nature) showing that untrained/calibrated models were more honest, while reinforcement learning shifted them toward overconfident responses. The author also runs a controlled simulation where two candidates share identical knowledge states and differ only in whether they abstain, finding that the “honest” agent loses on accuracy metrics yet is far more reliable when it speaks; changing the scoring rubric flips the ranking, reinforcing that the problem is the evaluation design. The piece further examines real-world consequences (including hallucinated legal citations) and discusses cases where confident outputs are treated as authoritative, then closes by reframing the “fluency as confidence” problem: language models sound like they know because they generate stylistically identical, declarative responses, and human readers import their own uncertainty-reading instincts into a system engineered to defeat that instinct.
Read the full blog for free on Medium.
Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.
Published via Towards AI
Towards AI Academy
We Build Enterprise-Grade AI. We'll Teach You to Master It Too.
15 engineers. 100,000+ students. Towards AI Academy teaches what actually survives production.
Start free — no commitment:
→ 6-Day Agentic AI Engineering Email Guide — one practical lesson per day
→ Agents Architecture Cheatsheet — 3 years of architecture decisions in 6 pages
Our courses:
→ AI Engineering Certification — 90+ lessons from project selection to deployed product. The most comprehensive practical LLM course out there.
→ Agent Engineering Course — Hands on with production agent architectures, memory, routing, and eval frameworks — built from real enterprise engagements.
→ AI for Work — Understand, evaluate, and apply AI for complex work tasks.
Note: Article content contains the views of the contributing authors and not Towards AI.
Related posts
Recent Posts
