Author(s): Yokeswaran

Originally published on Towards AI.

I Trained Two CNNs on 101 Food Categories. The Gap Was 42%

I had read the advice a hundred times.

“Don’t train from scratch. Use pretrained weights.”

Every course. Every tutorial. Every textbook.

And every time, I nodded — because it made sense in theory. But I never felt it. I never saw the gap with my own eyes. I never understood exactly where pretrained knowledge helps and where it still quietly fails.

So I stopped nodding. And I built the experiment myself.

The Setup

Two CNN models. One dataset. One fair fight.

Model 1 — TinyVGG. A lightweight 3-block CNN I built and trained completely from scratch. 12.6 million parameters. Zero prior knowledge. It starts knowing nothing — and has to learn everything from the data alone.

Model 2 — ResNet50. A 50-layer deep network pretrained on 1.2 million ImageNet images. I loaded the pretrained weights, froze the entire backbone, and added a custom classifier head on top. Only that head — about 2.1 million parameters out of 25.6 million — was actually trained on Food-101.

The dataset: Food-101. 101 food categories. 101,000 real-world photographs. Pizza, sushi, ramen, escargots, bibimbap — everything. With 101 classes, random guessing gives you 1%.

Same data. Same pipeline. Same hardware. Same optimizer. Same batch size.

The only thing that changed was the model.

Epoch 1 Told Me Everything

I want to tell you about one specific moment in this experiment.

After the very first training epoch — one pass through 75,000 images —

TinyVGG hit 2% test accuracy.

ResNet50 hit 52%.

Before it had seen a single Food-101 image with its backbone.

Before it had learned anything about what pizza looks like, or ramen, or sushi.

Fifty-two percent.

I stared at that number for a while. Because what it means is that ResNet50 arrived at this task already knowing things. Its backbone — trained on a million ImageNet images — had already built internal representations for curved surfaces, circular shapes, uniform textures, green pods, structured rolls.

It didn’t need to learn what a macaron looks like. It already understood circular pastel objects. It didn’t need to learn sushi. It already understood structured cylindrical forms.

TinyVGG had to discover all of that from scratch. Every edge detector. Every texture filter. Every shape representation. Built from zero, one gradient step at a time.

30 epochs later, TinyVGG reached 23%.

ResNet50 finished at 65%+.

That’s the gap. 42 percentage points.

And the backbone was frozen the entire time.

What “42%” Actually Means Across 101 Classes

One number hides a lot.

When I broke down the accuracy class by class — comparing both models across all 101 food categories — the picture became much more interesting than the overall score.

Some classes improved by 50–60 percentage points.

Edamame. Macarons. Sushi. Caesar salad.

These are visually unambiguous. Clear shapes. Distinct color profiles. Strong structural identity. ResNet50’s pretrained backbone already had representations for exactly these kinds of features — and it leveraged them immediately.

Other classes improved by only 10–15 points.

Steak vs filet mignon. Hot and sour soup vs clam chowder. Ramen vs pho.

These are genuinely hard. Similar colors. Similar plating. Similar textures. Even with pretrained knowledge, the model found them difficult — because the visual signal separating them is subtle enough that it requires far more specific training data or targeted fine-tuning to overcome.

This is what per-class analysis gives you that overall accuracy never can.

It tells you where your model is confident and where it’s still guessing. It tells you exactly where to focus your effort next. The 10 hardest classes are a roadmap — they’re saying “this is where more data, better augmentation, or a different strategy would actually help.”

Overall accuracy tells you how you did. Per-class accuracy tells you what to do next.

Two Times the Parameters. Three Times the Accuracy.

Here’s a number I want you to sit with.

ResNet50 has 2× the parameters of TinyVGG.

But it achieved nearly 3× the accuracy.

If parameters were the main driver, the ratio should be roughly equal. It’s not.

What this shows is that raw parameter count is the wrong lens entirely. What matters is how those parameters are organized — residual connections that prevent gradients from vanishing through 50 layers, batch normalization that stabilizes training at depth, hierarchical feature representations that build from edges to textures to shapes to objects — and critically, what those parameters already know before training begins.

A smaller model with prior knowledge will consistently beat a larger model starting from nothing.

Not because of the numbers. Because of the head start.

The Training Wasn’t a Straight Line

I want to be honest about something.

The path to 65% accuracy had a wall in it.

Around epoch 20–22, test accuracy stopped moving. It sat in a band between 64.5% and 65.0%, oscillating, going nowhere. The model wasn’t getting worse. It just… stopped improving.

The problem was the learning rate. It was still relatively high, and the optimizer was bouncing around a good minimum instead of settling into it.

My original scheduler — CosineAnnealingLR — drops the learning rate on a fixed curve regardless of what the model is doing. That works well when you're training everything from scratch. But for frozen-backbone training, where only the head is learning, you need something that reacts.

Switching to ReduceLROnPlateau changed things immediately.

Every time accuracy stalled for 3 consecutive epochs, the learning rate dropped by 70%. Within a few epochs of each drop, accuracy pushed to a new high.

The lesson: the training strategy has to match the architecture strategy.

Frozen backbone training has different learning dynamics than full training. Treating them the same way costs you accuracy.

Beyond the Model — Building the Whole System

This is where I want to spend a moment.

A lot of deep learning projects end with a notebook, a trained model, and an accuracy number.

Mine didn’t stop there.

I deployed both models as a live interactive Gradio app on HuggingFace Spaces. Running entirely on CPU. No GPU required. No sign-up. Anyone can upload a food image right now and watch both models predict in real time.

🔗 Try it here → huggingface.co/spaces/Yoki051202/food101-classifier

The app shows the predicted class, confidence score, Top-5 predictions, inference time — and you can run both models simultaneously and see them compete side by side.

But the part I’m most proud of isn’t the UI.

The .pth weight files — the actual trained models — live in a private HuggingFace model repository. The public Space contains only code. At startup, the app downloads the weights securely using a token stored as a Space secret.

Users can interact with the app. They can inspect the code. But they cannot access the weights.

Public Space (code only)
 │
 │ hf_hub_download() + HF_TOKEN secret
 ▼
Private Model Repo (weights only)
 │
 ▼
CPU Inference → Result shown to user

This is how production ML systems protect their intellectual property. Building it taught me more about real-world MLOps than any tutorial I’ve read.

Three Small Decisions That Moved the Numbers

Not everything that mattered was about architecture.

Label smoothing replaced hard 1.0 / 0.0 training targets with soft 0.9 / 0.001 distributions. For a 101-class problem where some categories genuinely look similar, this stopped the model from becoming overconfident during training. Small change. Real impact.

Mixed precision training via torch.amp.autocast cut training time noticeably with zero accuracy cost. If you're not using this on a GPU, you're leaving speed on the table for no reason.

Stronger augmentation for the stronger model. ResNet50 got random resized crops, color jitter, random grayscale. TinyVGG got gentler augmentation. A pretrained backbone is already robust to visual perturbations — it can handle harder inputs. TinyVGG, learning from nothing, needed room to breathe.

None of these decisions are revolutionary. But each one contributed. Together they closed the gap between what the architecture was theoretically capable of and what the training actually achieved.

What I’d Do Next

Fine-tune the backbone.

Freezing it was the right choice for this experiment — it isolated the effect of pretrained features cleanly. But it left performance on the table.

Training all 25.6 million parameters with a very low learning rate for the backbone (1e-5) and a higher rate for the head (1e-4) would likely push accuracy toward 78–83%. That’s where the real ceiling of this architecture lives.

I’d also add Grad-CAM visualizations to the demo — heatmaps showing which regions of the image each model focuses on when making a prediction. Accuracy tells you what the model does. Attention maps tell you how it thinks.

That’s the next version.

The Code

Everything is on GitHub — training notebooks for both models, the insights analysis, all visualizations, and the full HuggingFace deployment code.

⭐ github.com/yokeswarans/TinyVgg-ResNet50-food101

If you want to discuss the approach, have questions about any part of the implementation, or are working on something similar — leave a comment or reach out on LinkedIn. I read everything.

And if you haven’t tried the demo yet:

🔗 huggingface.co/spaces/Yoki051202/food101-classifier

Upload whatever food is closest to you right now. See what both models say.

Built with PyTorch · Gradio · HuggingFace Spaces

Thanks for reading this far — it genuinely means a lot! ❤️

If you found this article helpful, I’d really appreciate your support by following me on Medium.

I’ll be sharing more content on Computer Vision, PyTorch, and building real-world AI systems from scratch.

Let’s keep learning and building together. 🙌

-Yoki

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI