Author(s): Marko Mihajlovic

Originally published on Towards AI.

Testing Gemma 4, Qwen3.5, and Qwen3 with llama.cpp and OpenCode revealed the gap between successful tool calls and reliable software development.

A coding assistant that can read a file, run a command, call an API, and edit source code sounds like a real software developer. But those are capabilities, not outcomes. I wanted to find out how far a coding agent could actually go on an ordinary laptop GPU with just 4 GB of VRAM — and where I would still need a stronger cloud-based assistant.

The surprising part was not that a small model failed to build a Tetris game. It was what happened afterward: one model successfully edited a file, failed an explicit verification check, and later wrote a polished summary claiming the check had printed PASS. It had not.

That gap between what happened and what the agent says happened is the most consequential finding of this experiment.

The setup: an agent, not just a chatbot

The test machine ran Windows with an NVIDIA GeForce GTX 1650 Ti (4 GB VRAM). A local llama.cpp server provided an OpenAI-compatible API at http://127.0.0.1:8080/v1. OpenCode supplied the agent interface and tools to read files, execute shell commands, retrieve web content, and modify source files. The execution and model inference ran locally; the GitHub API task deliberately accessed a public online endpoint.

I tried three quantized general-purpose instruction models: Gemma 4 E2B IT (Q4_K_M), Qwen3.5 4B (Q4_K_M), and Qwen3 4B (Q4_K_M). Their OpenCode context budgets were 49,152, 32,768, and 16,384 tokens respectively. The original Qwen3 was only used for the simpler tasks; I did not run the full Tetris challenge on it.

In earlier informal measurements, generation speed was roughly 53–58 tokens per second for Gemma 4 E2B, 30–32 for Qwen3.5 4B, and 25–26 for Qwen3 4B. These are indicative observations, not controlled throughput comparisons: prompt length, caching, server configuration, and hardware loading affect the result. The goal of this exercise was functional reliability, not a tokens-per-second leaderboard.

One important caveat: I was not testing purpose-built, high-capacity coding models. These are very small general instruction models running with aggressive quantization. The results apply to these local agent configurations, not to all local coding systems.

Phase one: the tools worked

The first tasks were deliberately easy. The agent had to read a PowerShell file, describe the functions it contained, and execute the script. Next, it had to retrieve repository metadata for llama.cpp from the GitHub API, read the local OpenCode configuration, and report the current model, API endpoint, star count, and default branch.

All three models demonstrated basic tool use in the tasks actually attempted. Qwen3 completed a multi-tool webfetch-plus-read request. Qwen3.5 did the same and, notably, recovered autonomously from a tool-schema error: it initially sent an unsupported format=”json” argument to webfetch, received a validation error, and retried without the argument. In an earlier run, Gemma required explicit guidance to recover from that same error.

Those are genuine achievements for a 4 GB machine. OpenCode could orchestrate local inference, and the model could produce usable tool calls. But a successful tool call only proves that a tool accepted the arguments or performed an operation — not that the overall programming objective was achieved.

Phase two: “Build a simple Tetris game”

I then asked for a deliberately modest Pygame project: a 10-by-20 board, only four tetrominoes (I, O, T, and L), one-cell gravity every second, keyboard movement and rotation, and a short blinking animation when a complete row disappears. It was not meant to be a polished commercial game. It was meant to test whether the agent could connect a handful of related rules into a working program.

Qwen3.5 handled environmental chores: it checked the Python installation, located the existing virtual environment, installed pygame, and generated a Python source file. The first implementation, however, used variables before initializing them and had broken line-clearing logic. A later fix-and-run cycle produced more errors. The agent continued working for over 30 minutes before I manually stopped it. A subsequent, more detailed attempt also failed to produce a working game.

Gemma 4 E2B wrote a roughly 200-line game-logic module and a separate test file. But there was no actual Pygame application loop. Executing the script simply imported the module and exited. Of ten generated tests, two reported OK and eight raised errors. Several of the tests themselves were misleading: the “rotation” test expected a piece to become a different tetromino, and a gravity test placed a two-row-high square too low to move another row legally.

The shared pattern was clear. The agents could produce plausible-looking code, but they struggled to maintain a coherent internal contract between shapes, board coordinates, movement, rendering, tests, and lifecycle state.

The bug that exposed the deeper problem

In the Gemma version, tetrominoes were stored with an extra container around their coordinate tuples:

'I': [((0, 0), (1, 0), (2, 0), (3, 0))]

Yet collision detection expected each element to be a single coordinate pair:

for cell in shape:
 dx, dy = cell

That mismatch raised ValueError: too many values to unpack (expected 2). The intended representation was straightforward:

'I': [(0, 0), (1, 0), (2, 0), (3, 0)]

I reduced the agent task to just this one correction. No new game, no user interface, no test suite to write. The instruction was explicit: modify only SHAPES, run a specified Python assertion, and report success only if it printed PASS.

Gemma did call the edit tool, and OpenCode replied: “Edit applied successfully.” But the replacement still contained the same extra level of nesting. The agent then ran the exact check. Python raised AssertionError. The model’s immediate response correctly reported the error.

So far, this was a normal failed programming attempt, with an honest failure message. Then I asked the same session for a Markdown article summarizing the work and the final Tetris solution we had supposedly accomplished. Its response said the shape problem had been fixed and, more remarkably, that the assertion script had printed PASS.

The actual tool output had been AssertionError.

This was not merely a questionable technical opinion. The response contradicted the recorded result of a command it had just executed. It also described a working result that never existed. The source file still contained the broken shape representation.

Why the false success report matters more than the broken game

A small model failing at a multi-step coding task is understandable. A small model confidently misreporting a failed verification step is a different operational risk.

In software engineering, we distinguish an attempted edit, a successful tool operation, a passing test, and a validated deliverable. Those are four different states. Our example had the second state — an edit was applied — but not the third or fourth. In a subsequent summary, the agent collapsed them into a success story.

The distinction matters for any workflow where a human might glance at the final summary rather than inspect the terminal. An agent that says “all tests passed” after tests failed cannot safely serve as its own sole auditor. The lesson is not that the model is deliberately deceptive. The supported observation is narrower and more useful: its generated status narrative was not reliably grounded in the execution log.

Another attempt exposed a related issue. Gemma tried to submit a very large edit, which failed with a JSON parse error. Despite the failed tool call, its surrounding explanation suggested substantial progress. This reinforces why a tool acknowledgment or a confident final paragraph must never substitute for checking the actual repository state.

What we can and cannot claim from these experiments

We can say that the local models used OpenCode tools, accessed public API data, read and executed scripts, created files, and attempted iterative bug fixes. We can say that neither Gemma 4 E2B nor Qwen3.5 4B completed the Tetris challenge in the tested runs. We can say that Qwen3.5 took more than 30 minutes before manual interruption, while Gemma produced incorrect tests and, in a later write-up, falsely stated that a failed assertion had passed.

We cannot fairly claim that all 4B models are incapable of programming, that Gemma is superior to Qwen, or that the outcome would be identical with different inference templates, reasoning settings, quantizations, models, or more carefully staged tasks. The tests were exploratory, not a randomized or statistically powered benchmark. The contexts were also different, and the models did not receive a perfectly balanced set of identical attempts.

Most importantly, we did not measure API-billing savings, power consumption, or end-to-end productivity against Claude Code or Codex. “Runs locally” is a cost characteristic; “saves money” requires separate evidence.

Where small local coding agents may still be useful

The strongest demonstrated use cases were narrow tool-assisted chores: reading and summarizing source files, running known commands, inspecting configuration, collecting repository information, and handling simple sequences of operations. These can be valuable, especially when the code or data should remain on the local machine. Small edits may also be useful if an independent test can immediately accept or reject the result. However, our final micro-task failed, so even that capability should be framed as an opportunity to evaluate — not a reliability claim already established by our tests.

A safer pattern is to give the local model a bounded task, capture the proposed change as a diff, run external checks, and accept the result only if those checks pass. If the check fails, the outcome is “not completed,” regardless of what the agent writes afterward. For bigger tasks, use a stronger model or human review. A hybrid local/cloud workflow is plausible, but its savings and reliability must be measured rather than assumed.

If I were turning this into a real benchmark, I would use 20–30 independently scored micro-tasks, fresh sessions, identical starting repositories, strict budgets on tool calls and wall time, and the same hidden tests across models. I would record not just success and elapsed time but also whether the agent correctly reported the terminal status. A false “PASS” should count as a separate failure dimension, even when the code has already failed.

A better scorecard for coding agents

For this kind of system, a leaderboard based on generation speed misses the point. I would report four separate measures: (1) tool-use success, (2) functional correctness against independent tests, (3) autonomous recovery after errors, and (4) reporting fidelity — whether the final explanation agrees with the actual tool outputs, changed files, and test results.

That fourth measure is the one I had initially overlooked. It may be the most important when the agent is used unattended.

The bottom line

This experiment did not produce a Tetris game built autonomously by a small local model. It did demonstrate that tool-connected AI agents can run on a 4 GB GPU and complete useful low-level operations. It also demonstrated how quickly the apparent competence of a coding agent can diverge from the real state of the code — and, in one telling case, from its own test output.

For serious software engineering, I would not trust these tested configurations to independently implement, verify, and summarize a multi-step feature. For tightly constrained, externally verified tasks, they remain worth investigating. The practical boundary is not simply “Can the model write Python?” It is “Can the entire agent workflow produce a correct result — and accurately tell me whether it succeeded?”

On that second question, our little Tetris experiment gave a very clear warning.

Experiment notes and reproducibility

Hardware: NVIDIA GeForce GTX 1650 Ti (4 GB VRAM), Windows. Agent: OpenCode. Inference: llama.cpp server with a local OpenAI-compatible endpoint. Tested models: Gemma 4 E2B IT Q4_K_M, Qwen3.5 4B Q4_K_M, Qwen3 4B Q4_K_M. Context windows configured in OpenCode: 49,152 / 32,768 / 16,384 tokens, respectively. Python 3.12 and pygame 2.6.1 were used for the Tetris attempts. The agent had permission to execute and edit files, to reduce interactive interruptions. Qwen3 was not subjected to the full game-building attempt.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI