Author(s): Shoaib Ahmed Quraishi

Originally published on Towards AI.

The QA model-routing problem: why the best AI model isn’t always the right model — and how to route testing work by reasoning complexity, cost, and failure risk.

Imagine a regression pipeline finishes with 800 failures.

The article argues that treating every QA failure the same way—sending all of them to the most capable, most expensive “frontier” model—is economically lazy and often unnecessary. It introduces a “Model Tax” idea (overpaying for reasoning capacity a task doesn’t need) and proposes routing based on three variables: reasoning required, how often the task runs, and the cost if the model is wrong. It explains how routing works in practice: start with the cheapest tier, use confidence and uncertainty to cluster and triage failures, and escalate only the ambiguous remainder to higher tiers. It discusses where each tier fits (pattern-like log classification for small models, correlation-heavy flaky analysis for mid models, and multi-system debugging for frontier models), emphasizes that “cheap” should be evaluated by cost per successful outcome rather than per-inference cost, and notes that frontier models should not be the default but the escalation path. The piece closes with what the author would build (a classifier plus upstream complexity routing, confidence checks, escalation, and full logging) and reframes the key question as “which model should solve this?” rather than “which model is best available?”

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI