The Best AI Agent Scores 34.8% on COBOL. It Cost IBM $30 Billion Anyway.
Last Updated on August 24, 2026 by Editorial Team
Author(s): Andrus
Originally published on Towards AI.
The Best AI Agent Scores 34.8% on COBOL. It Cost IBM $30 Billion Anyway.
Six months after a blog post with no customers in it erased a quarter of IBM’s market value, the only hard numbers anyone has produced come from a benchmark almost nobody quoted.
After Anthropic claimed that AI would let teams modernize COBOL “in quarters instead of years,” the article argues the market reacted because the claim lacked verifiable customer results. IBM’s rebuttal named several banks and cited its own tooling, but the author suggests the dispute was really about narrative velocity rather than actual capability—especially given that COBOL may be structurally harder for current AI agents due to low training data and idiosyncratic semantics like fixed-point decimals and the PICTURE clause. The turning point is a later, more rigorous evaluation: Legacy-Bench, which measures AI agents performing maintenance tasks in real legacy code, finds overall pass rates in the high teens to low 40s, with COBOL specifically around 34.8% and many of the hardest failures involving subtle “silent” correctness issues that can compile and run while still producing wrong outputs. Additional work and analysis reinforce that better results come less from smarter prompts/models and more from wrapping models with formal verification and repair loops. Six months after the hype, the author reports no completed, publicly documented AI-led COBOL migrations, contrasting two benchmarks and a scarcity of finished deployments, while noting that IBM’s later earnings effects appear attributable to broader investment timing rather than direct “COBOL modernization” impact.
Read the full blog for free on Medium.
Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.
Published via Towards AI
Towards AI Academy
We Build Enterprise-Grade AI. We'll Teach You to Master It Too.
15 engineers. 100,000+ students. Towards AI Academy teaches what actually survives production.
Start free — no commitment:
→ 6-Day Agentic AI Engineering Email Guide — one practical lesson per day
→ Agents Architecture Cheatsheet — 3 years of architecture decisions in 6 pages
Our courses:
→ AI Engineering Certification — 90+ lessons from project selection to deployed product. The most comprehensive practical LLM course out there.
→ Agent Engineering Course — Hands on with production agent architectures, memory, routing, and eval frameworks — built from real enterprise engagements.
→ AI for Work — Understand, evaluate, and apply AI for complex work tasks.
Note: Article content contains the views of the contributing authors and not Towards AI.
Related posts
Recent Posts
