Author(s): Mengliu Zhao

Originally published on Towards AI.

Paper Walkthrough — MACT: A Multi-Agent Collaboration Framework for Visual Document Understanding

From one model doing everything to four specialists doing one thing well

A financial report is not a photograph.

It is a stack of dense tables, cropped charts, multi-column text, and footnotes — all demanding a different kind of attention at every step. Reading it isn’t a single glance; it’s a procedure: figure out what’s being asked, decide where to look, extract the right numbers, check your own work, and only then answer.

Most large Vision-Language Models (VLMs) don’t work that way. They take a document and a question, run one forward pass, and hope the answer falls out. MACT, a CVPR 2026 paper from NUS, Tencent Youtu Lab, Tsinghua, and collaborators, argues that this “monolithic scaling” — just making the model bigger — is the wrong lever to pull for documents. Instead, MACT splits the job into four cooperating agents, each specialized for one cognitive step, and lets a smaller model beat much larger ones.

In this walkthrough, we’ll trace the full story: a) why bigger VLMs are hitting diminishing returns on documents, b) how MACT’s four agents divide the labor, c) how test-time compute is allocated differently to each of them, and d) how the reward signal is designed so the agents don’t just game their own local objective.

Why Documents Break Monolithic Scaling

Scaling up parameter count has been the default strategy for VLMs, and it works well for general vision-language tasks. But for document-based visual question answering (VQA), the paper shows that increasing parameter count yields only marginal gains, while computational cost grows exponentially.

The paper attributes this to three properties that documents have, and natural images mostly don’t:

• Procedural reasoning. Answering a question about a document isn’t one action — it’s decomposing the question, forming a strategy, locating the right region, and synthesizing an answer. A single forward pass compresses all of that into one shot.
• Cognitive overload. Layout parsing, fine-grained text extraction, logical inference, and arithmetic are different skills. Forcing one set of weights to be good at all of them trades off depth for breadth.
• Vulnerability to factual errors. A single misread table cell or a truncated paragraph can invalidate the whole answer, and a monolithic feed-forward model has no way to catch its own mistake mid-reasoning.

The proposed fix is a shift from monolithic scaling to what the authors call procedural scaling: decompose the workflow into functional roles, and scale each role independently, in a way that the role actually benefits from more compute.

Four Agents, One Pipeline

MACT decomposes document understanding into four collaborative agents, each handling one stage of the workflow:

Planning Agent (𝒜_plan). Given the question and the document, this agent doesn’t just write one plan — it first generates several relevant reference plans for similar hypothetical tasks (inspired by analogical prompting), then uses those as scaffolding to produce the actual execution plan for the real question. The plan stays high-level: it states targets and requirements per step but doesn’t commit to specific tools or execution details, so it doesn’t box in the agent that has to actually do the work.

Execution Agent (𝒜_exe). This agent walks through the plan step by step, treating each step as a unit with its own definition, expected output, and access to results from prior steps. It pulls tools from a shared tool library to actually extract information — OCR, cropping, table parsing, and so on — and passes along the full trace of what it did and found.

Judgment Agent (𝒜_judg). This is the self-correction layer, and its design choice is the most interesting part of the paper. Rather than having the planning or execution agent grade its own homework, MACT uses a separate, independent judge that only assesses correctness — it never attempts a fix itself. If it finds a mistake, it flags which step failed, describes the mistake, and routes the problem back to whichever agent (planning or execution) is responsible, up to three correction rounds.

Answer Agent (𝒜_ans). Once a plan-execution pair has passed judgment, this agent synthesizes the final answer — deliberately keeping both the corrected trace and the earlier incorrect attempts in context, so it can see exactly what changed rather than silently discarding the error-prone parts.

The reason for a separate judge is worth sitting with. The paper compares three self-correction designs: a) let the same agent correct itself internally, b) use one extra agent that both judges and corrects, and c) MACT’s approach — a judge that only judges. Options a) and b) share a weakness: whatever component is responsible for correction is being trained, via RL, toward “pass verification” as its objective. That creates an incentive to produce vague or incomplete fixes that look right on the surface but aren’t actually right. Separating judgment from correction removes that incentive from the judge, at the cost of needing four coordinated components instead of one or two.

Agent-Wise Adaptive Test-Time Scaling

The second pillar of MACT is that test-time compute isn’t spent the same way for every agent — each one gets a scaling strategy suited to its actual job:

• The planning agent scales by generating multiple relevant reference plans in parallel, increasing the odds that at least one reasoning path lines up with what the document actually contains.
• The execution agent scales at the step level: for each execution step, it produces several candidate outputs, scores them with a pretrained reward model, and keeps only the top-scoring one before moving to the next step — rejecting the rest.
• The judgment agent uses budget forcing: a minimum thinking-token budget is enforced, so if the agent tries to wrap up its reasoning too early, it’s pushed to keep reasoning until it hits the floor. Since judgment is the step most sensitive to catching subtle mistakes, more deliberation pays off disproportionately here.
• The answer agent, whose job is mostly synthesis rather than search or verification, gets little benefit from extra test-time compute, so the paper doesn’t scale it aggressively.

This matters because it directly counters a real failure mode of naive test-time scaling: pouring the same extra compute into every component regardless of whether that component actually benefits from it. The paper’s ablations show this agent-specific allocation beats parallel, sequential, hybrid, and internal (budget-forcing-everywhere) scaling strategies applied uniformly across the whole pipeline.

Does the Decomposition Actually Help?

The paper trains three variants of MACT on different base model families — Qwen2.5-VL, MiMo-VL, and InternVL3 — each kept under 30B total parameters, and evaluates on 15 benchmarks spanning document types (text, webpage, chart, table) plus general and mathematical reasoning, to check that the framework doesn’t just help documents at the expense of everything else.

The headline result: despite being smaller than 30B parameters, all three MACT variants outperform every open-source and closed-source model under 100B parameters that the paper tested, and beat their own base models by 9.9–11.5% on average across the 15 benchmarks. Compared against much larger monolithic models from the same family — Qwen2.5-VL-72B and InternVL3–78B — the MACT variants still come out ahead by 3.7–6.6% on average, with the widest margins showing up on the longest-context and most reasoning-heavy benchmarks.

The Bigger Picture

MACT’s core bet is that for a domain like document understanding — where the task is inherently a multi-step procedure rather than a single perceptual judgment — the right place to add capacity isn’t a bigger model, it’s more roles, and more targeted compute per role. That’s a different axis than the “just scale parameters” story that has dominated VLM progress, and the fact that a set of sub-30B-parameter agents can outperform 70–90B monolithic models on this specific class of task is a reasonable existence proof that procedural scaling is worth taking seriously — at least for tasks that are naturally sequential and self-checkable, which document QA clearly is.

References

• Yu X, Xu C, Chen Z, Zhang Y, Lu S, Yang C, Zhang J, Yan S, Hu X. Visual Document Understanding and Reasoning: A Multi-Agent Collaboration Framework with Agent-Wise Adaptive Test-Time Scaling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2026.
• Xiaomi LL, Xia B, Shen B, Zhu D, Zhang D, Wang G, Zhang H, Liu H, Xiao J, Dong J, Zhao L. MiMo: Unlocking the Reasoning Potential of Language Model — From Pretraining to Posttraining. arXiv preprint arXiv:2505.07608. 2025 May 12.
• Yang A, Li A, Yang B, Zhang B, Hui B, Zheng B, Yu B, Gao C, Huang C, Lv C, Zheng C. Qwen3 technical report. arXiv preprint arXiv:2505.09388. 2025 May 14.
• Zhu J, Wang W, Chen Z, Liu Z, Ye S, Gu L, Tian H, Duan Y, Su W, Shao J, Gao Z. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. 2025 Apr 14.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI