Author(s): Chew Loong Nian – AI ENGINEER

Originally published on Towards AI.

DeepSeek V4-Flash vs GLM-5.2: The 1.7-Point Win Collapses When You Swap the Harness

DeepSeek’s own chart says 82.7 on Terminal-Bench 2.1. Artificial Analysis measured 79. That 3.7-point gap is 2.2x the margin the 82.7 was defending.

The article argues that DeepSeek’s headline claim—beating GLM-5.2 across multiple agent benchmarks by a 1.7-point margin on Terminal-Bench 2.1—depends heavily on a specific (and then-unreleased) “Harness” used to test agent performance. After reading the table footnote, the author highlights that DeepSeek’s harness hadn’t been available publicly, so independent reruns were impossible at the time; a later independent run by Artificial Analysis using a different scaffold dropped DeepSeek’s score by 3.7 points and flipped the overall ordering. The author then quantifies how much of the nine-row sweep likely reflects the model versus the harness, using a script to sample documented harness-only effects from prior literature and bootstrap survival probabilities; the result suggests only about ~2% of the full sweep would remain intact after swapping harnesses, making the “across-the-board win” framing fragile. The piece connects this to two 2026 research papers showing harness variance can dwarf model variance, and it closes with practical guidance on when to trust a benchmark score (e.g., cost-bound decisions vs cross-scaffold robustness) and the broader lesson to disclose and publish the harness card rather than rely on vendor charts.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI