Author(s): Chew Loong Nian – AI ENGINEER

Originally published on Towards AI.

Claude Code’s Rust Rival Calls Its Extension Gate 123/123. Its Own Evidence File Reads ‘fail’.

There is a particular kind of document that shows up in ambitious open-source repositories, and if you have been around long enough you can smell it before you open it. It is called something like PROOF, or EVIDENCE, or CERTIFICATION. It opens by telling you that everything inside is reproducible. It has a table near the top where every claim sits in the left column and a filename sits in the right column, and the filename is the point: the author is saying do not take my word for it, open the file.

After introducing the idea of falsifiable “proof” documents, the article walks through an audit of a Rust project’s evidence report for its Pi extension runtime. The author writes a script to extract claimed JSON paths, resolve them against the repository (including how gitignore affects what can be checked), and then compare the document’s asserted numbers to the actual contents of the cited artifacts. The core “123/123” claim fails because the referenced evidence JSON on today’s checkout reports a top-level status of “fail,” with mismatched pass/fail counts and regenerated_at timestamps that prove the report is stale. Some other claims are also uncheckable because they rely on files ignored by git, and others are contradicted by different committed baselines or missing data. Importantly, the README ultimately includes updated caveats and labels older results as historical snapshots, showing the project’s security argument (QuickJS with capability gating and audit logging) is stronger than the proof document itself—but the evidence-chain breaks due to documentation drift and manual maintenance. The author closes by recommending that evidence audits be automated in CI and not trusted to survive time in a markdown table, contrasting this with how other agents like Claude Code handle extension boundaries.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI