Author(s): Veera RS

Originally published on Towards AI.

Why a 3B AI Model Can Beat a 70B One — It’s Not About Model Size Anymore

Chances are, when you’ve interacted with a chatbot, you’ve seen it pause and say something like “thinking.” What’s going on there?

The Old Playbook: Train-Time Compute

Most people know how LLMs actually get trained. There’s a transformer architecture, and it’s fed a massive corpus of data — an absolute ton of data going in. Through next-token prediction, it learns language, it learns reasoning, it learns facts, and probably learns how to code. And all of this gets compressed into the weights of the model.

For years, the playbook for making that model smarter has been to scale up. That means more parameters. That means more training data, more FLOPs (Floating Point Operations — it’s a measure of how much raw computational work a model performs) during pre-training, and the scaling laws have shown that this holds up.

This approach is called train-time compute, and it is a fixed cost. You spend months of computing time and probably millions of dollars training a model, and then those weights are frozen.

From that point on, whether somebody asks the model to summarize an email or to solve a graduate-level physics problem, it does the same thing: one forward pass through the network, token by token.

And every token is something of a commitment, because the model picks the statistically most likely next token, emits it, and at that point it’s kind of locked in — there’s no going back to reconsider.

It’s a forward pass, which is, well, always moving forward. So if that first token sends the response down the wrong path, the model just keeps going with it. That’s actually one of the reasons LLMs can hallucinate so convincingly.

What If We Give the Model a Compute Budget?

But what if we give the model a compute budget? This time, we’re going to give the model a budget — specifically, this budget is not spent during training but at inference time, when the model runs. And it decides how to spend that budget.

Well, that is test-time compute. That’s what that “thinking” message is all about. And the research coming out over the last couple of years shows this might be just as important a scaling axis as model size.

So what’s actually going on during that thinking time? There are a few distinct mechanisms, and they can actually be combined.

Mechanism #1: Chain of Thought

The most visible one of these is just a chain of thought. Anybody can invoke a chain of thought just by prompting it — by telling the model to think step by step. But there’s a newer class of models, and those models are called reasoning models.

These reasoning models have been trained through reinforcement learning to do this automatically. So the RL process bakes a chain of thought into the model itself.

During RL training, the model learns that producing intermediate reasoning tokens — breaking down a problem and working through the logic step by step — tends to get a higher reward. So it does it more often.

Essentially, we can say it is generating what we’ll call thinking tokens as it goes through this process. Thinking tokens are generated before the actual response.

Now, what’s special about thinking tokens? On the face of it, not much. They’re still real output tokens, they cost real compute — but they change what that forward pass is being used for.

Remember, in a standard response, the model is committing to final answer tokens from the very first word. But with thinking tokens, those early commitments are really just like a scratchpad of work.

The model can explore an approach. It can realize, “Ah, this is not working,” and then try a different angle. All before it commits to a single word of the actual answer returned to the user.

So instead of going straight from a query to the answer we give back to the user, we now have an intermediate step. We go from query → reasoning (where the thinking tokens are generated) → answer.

So that’s the chain of thought.

Mechanism #2: Search

The second mechanism is search. In standard inference, the model does greedy or near-greedy decoding, and that one-and-done forward pass is what I’m talking about there. It picks the most likely next token, and then it just moves on. But with test-time compute, you can do something more like a tree search.

The model in a tree search starts a reasoning chain, and then it branches. So it tries different branches kind of off the tree. And at some point, it needs to pick one of these branches to go down. To do that, it uses a verifier to score which branch is most promising before continuing down it.

Mechanism #3: Self-Consistency

The third mechanism is self-consistency. This is to run the same problem n times at a really high temperature, so you get n different reasoning paths. Then you take a majority vote on the final answer.

So if, let’s say, 7 out of 10 independent chains have landed on the same answer, you’ve got some pretty decent confidence. There’s no verifier model needed in this case, because it’s using the statistical distribution of the model’s own outputs as the signal.

To be clear, all three of these mechanisms are trading something. They are trading compute for accuracy. There are more FLOPs per query, but there’s also a higher probability of getting a good response.

Does This Actually Work?

A 2024 paper from Google DeepMind on scaling test-time compute found that test-time compute follows its own scaling law. Click here to read the paper.

If we think of the axes here as performance on reasoning benchmarks, the performance actually goes up slowly, smoothly, and predictably as you increase inference compute.

In fact, researchers showed that if you take a kind of tiny parameter model — let’s say a 3 billion parameter model — and you use test-time search strategies, it would outperform a much bigger model, a 70 billion model, on hard math problems. So that’s a model that is over 20 times smaller, beating the big one — and it’s doing it just by thinking longer.

The Trade-Offs: Latency, Cost, and Overthinking

But there are definitely some trade-offs here. More thinking time — well, the obvious thing that means is more latency. If every single chatbot query takes 45 seconds while the model works through a search tree, well, users might have a bit of a bad time. And those thinking tokens are billed as regular old output tokens. So a response that burns through 10,000 of them for a single response is more expensive to run.

But it’s not just latency and expense. Another problem is just, basically, overthinking. Forcing a reasoning model to deliberate on simple questions can actually degrade performance.

The model kind of second-guesses itself into the wrong answer. And if you’ve ever talked yourself out of the right answer on an exam — yeah, it’s the same thing. And that is an analogy I have some first-hand experience of.

CAPEX vs OPEX: The Economics of Thinking

From an economics perspective, training compute — that is, considered a capital expense, or CAPEX — is paid once, regardless of the query volume or inference time. But test-time compute, that is OPEX, operational expense. It’s paid per query, and you can choose how much per query as well.

The Adaptive Approach

With these trade-offs in mind, the best approach is actually one that is considered adaptive — so it changes based upon the request that comes in. We can route easy queries to the fast, single-pass inference. And then we can route harder queries through the full reasoning pipeline. That’s how many chatbots work today. ChatGPT, for example, uses a picker to route queries between reasoning and non-reasoning models.

Two Axes, One Future

So we’ve been scaling AI by making models bigger and bigger at training time — and test-time compute is the second axis. It’s letting the model spend more compute on the problems that need it.

AI models, they’re getting bigger, they’re getting faster — but they’re also learning when to slow down and think.

If you found this helpful, consider clapping👏 so others can find it too and follow me for more amazing AI content!

Continue Reading

1. Prompt Caching Explained: How to Slash LLM Costs and Latency Without Sacrificing Quality
2. SLM vs LLM vs Frontier Models: Which One Should You Actually Use?
3. A Simple Markdown File Is Teaching AI Agents How to Think!
4. AI Isn’t Replacing Developers — It’s Doing Something Far More Interesting!

References

[1] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — https://arxiv.org/html/2408.03314v1

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI