Author(s): Ake

Originally published on Towards AI.

A practical, first-principles guide to the problems Kubernetes solves — and why Docker alone is not enough

Part 1 of the Kubernetes for MLOps series

TL;DR

Kubernetes exists because running one container is easy, but operating many containers across many machines is not.

• A Python service is simple, but it creates a single point of failure.
• Virtual machines improve isolation, but they are heavy, slow to start, and prone to environment drift.
• Docker makes applications portable, reproducible, and lightweight — but mainly solves the single-host problem.
• Docker Compose coordinates containers on one machine, not across an entire fleet.
• Kubernetes adds scheduling, self-healing, service discovery, scaling, and zero-downtime deployments across multiple machines.

The central idea is simple: you declare the state you want, and Kubernetes continuously works to make the real system match it.

What you will understand after this chapter: Why the industry converged on container orchestration, and what problem Kubernetes actually solves — from first principles, not marketing copy.

The Starting Point: A Fraud Detection Team

You are the sole ML engineer at a fintech startup. The payments team has trained an XGBoost model that detects fraudulent transactions with 94% precision. The model needs to run as a real-time inference service: every card swipe calls your API within 200ms and gets a fraud probability score. If the score exceeds a threshold, the transaction is blocked.

The model works. Now the infrastructure becomes your problem.

This chapter traces exactly how that problem evolves — from a Python script to a Kubernetes deployment — and at every step explains why the current approach broke down and what each new layer actually solved.

Era 1: Start with a Python Service

You start the only way an engineer should: the simplest thing that works.

# fraud_detector.py
import numpy as np
import xgboost as xgb
from fastapi import FastAPI
from pydantic import BaseModel
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

app = FastAPI(title="Fraud Detector", version="1.0.0")

# Model loaded once at startup — lives in this process's memory
model = xgb.XGBClassifier()
model.load_model("fraud_model.json")
logger.info("Model loaded successfully")

...

@app.get("/health")
def health():
 return {"status": "ok"}
...

You run it:

uvicorn fraud_detector:app --host 0.0.0.0 --port 8000 --workers 4

It works. The payments team integrates it. Transactions flow. Life is good for about six weeks.

What Breaks

Single point of failure. Your process is the only instance. When it crashes — due to a memory leak, an unexpected exception, a malformed input — every downstream payment attempt fails. At 3am on a Saturday.

No isolation. The fraud detector shares the OS, filesystem, CPU, and memory with every other process on that machine. A misconfigured apt upgrade can break your Python runtime. A different service leaking memory OOM-kills your process. You have no guarantees.

Manual deployments. Retraining the model means SSH-ing to the production server, copying a new fraud_model.json, and restarting uvicorn. Every deployment is a manual SSH session. Mistakes happen. There is no rollback.

No horizontal scaling. Transaction volume grows 5x after a marketing campaign. You cannot add capacity without significant manual intervention. The single instance becomes a latency bottleneck.

No resource limits. A bug in the feature extraction code causes a tight loop. Your process consumes 100% CPU. Other services on the same host degrade.

Era 2: Add Isolation with Virtual Machines

The first instinct is correct: isolate services. Virtual machines provide hard boundaries between workloads. The isolation story is real. A crash in VM 1 does not affect VM 2. The hypervisor enforces CPU and memory boundaries. You can snapshot, restore, and clone VMs. You have an audit trail.

What virtual machines did not solve

Resource waste at scale. A Ubuntu 22.04 minimal install consumes roughly 2GB of RAM just to exist. Your XGBoost model with a FastAPI wrapper needs about 400MB of RAM to serve traffic. The VM tax means you are paying for 2GB of RAM per instance just to run a 400MB application. Across a fleet of 50 fraud-detection VMs, that is 100GB of RAM doing nothing but running OS daemons.

Boot time. A VM takes 30–90 seconds to boot. When traffic spikes suddenly — a flash sale, a bot attack, a news event — you cannot add capacity fast enough. By the time a new VM is healthy, the spike has passed.

Environment drift. Two VMs provisioned from the same Machine imagesix months apart will differ. Security patches, library updates, and manual configuration changes accumulate. You have experienced “it works on VM 2 but not VM 3” at the worst possible time.

Slow iteration. To deploy a new model version, you build a new Machine image(10–15 minutes), launch a new instance (2–3 minutes), wait for health checks (1–2 minutes), shift traffic. A deployment takes 30 minutes minimum. Rolling back is not faster.

The dependency conflict problem. The fraud detection service needs XGBoost 2.0. A new anomaly detection service needs XGBoost 1.7 because a legacy dependency pins it. On VMs, both services share the system Python. You either containerize the environments manually (virtualenv, conda) or run each service on its own VM — amplifying the waste problem.

Virtual machines solved isolation. They created a new category of problems around density, speed, and reproducibility.

Era 3: Package the Service with Docker

Docker and Containers: The Essential Concepts

Docker did not invent containers. Linux already provided the core technologies, especially namespaces and control groups (cgroups).

Docker’s main contribution was making containers easy to build, distribute, and run consistently across different environments.

Namespaces: Process Isolation

Linux namespaces give a process its own view of system resources.

The container can also have its own hostname, filesystem, and network interface. However, it still shares the host’s Linux kernel.

cgroups: Resource Limits

Namespaces provide isolation, while cgroups control resource usage. With Docker, you can restrict how much CPU and memory a container can consume:

docker run \
 --memory="512m" \
 --cpus="1.0" \
 fraud-detector:v1.2.0

This container can use up to:

• 512 MB of memory
• One CPU core

If it exceeds its memory limit, the kernel can terminate the container’s process without directly stopping the other containers on the host.

Container Images and Layers

A container image is built from read-only filesystem layers:

Layer 4: Application code
Layer 3: Python dependencies
Layer 2: System libraries
Layer 1: Base image

For the fraud detection service, the layers might contain ( for example ):

Application: fraud_detector.py and fraud_model.json
Python packages: XGBoost, FastAPI and NumPy
System libraries: libgomp and libstdc++
Base image: python:3.11-slim

When only the application code changes, Docker can reuse the previous layers. This makes builds faster and ensures that the same application environment can run in development, testing, and production.

Docker therefore solved three important problems:

• Packaging: the application and its dependencies travel together.
• Reproducibility: the same image runs across environments.
• Isolation: each container receives its own process, network, and filesystem view.

Why Docker Is Not Enough

Docker solved packaging, reproducibility, and isolation for our fraud detector. But operating one container is very different from managing many containers across several machines.

Imagine that ten fraud-detection containers run on three virtual machines. One machine crashes, another still runs model v1.2.0, and the third already runs v1.3.0.

New questions appear:

• Who replaces containers when a machine fails?
• How do we scale during a traffic spike?
• How do we deploy a new model without interrupting requests?
• How do we keep every machine on the same version?

Docker runs containers, but it does not coordinate an entire fleet. This is the container orchestration problem.

Docker Compose Helps — on One Machine

Docker Compose can define multiple services, health checks, volumes, and restart policies in one file. It is excellent for local development and some single-host deployments.

However, it does not provide fleet-wide scheduling. If a machine fails, Compose cannot automatically move its containers to another host. For that, we need an orchestrator.

Enter Kubernetes

Kubernetes manages containerized applications across multiple machines. It provides:

• Automatic scheduling and recovery
• Horizontal scaling
• Service discovery and load balancing
• Secret and configuration management
• Controlled rolling updates

Its central idea is the reconciliation loop. Kubernetes continuously compares the state we requested with what is actually running.

If we request three replicas of the fraud detector but only two are running, Kubernetes creates a replacement.

Declarative Configuration

Instead of manually starting and replacing containers, we declare the result we want:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: fraud-detector
spec:
 replicas: 3
 selector:
 matchLabels:
 app: fraud-detector
 template:
 metadata:
 labels:
 app: fraud-detector
 spec:
 containers:
 - name: fraud-detector
 image: fraud-detector:v1.3.0
 ports:
 - containerPort: 8000
 readinessProbe:
 httpGet:
 path: /health
 port: 8000

Apply it with:

kubectl apply -f fraud-detector-deployment.yaml

Kubernetes schedules the Pods, monitors them, and replaces failed instances. The readiness probe prevents traffic from reaching a Pod before it is ready.

Docker vs. Kubernetes

Docker and Kubernetes are not competitors. Docker gives us a consistent container image; Kubernetes keeps the distributed application running.

In the next part, we will examine the Kubernetes control plane, worker nodes, API server, scheduler, controller manager, and kubelet.

Sources and Further Reading

1. Docker overview — Explains containers, images, isolation, and Linux namespaces.
   Docker: What is Docker?
2. Docker security and isolation — Covers namespaces, cgroups, and the container security model.
   Docker Engine Security
3. Docker Compose — Explains how Compose defines multi-container applications using YAML.
   Docker Compose Documentation
4. Docker Compose in production — Discusses using Compose for production and single-server deployments.
   Use Docker Compose in Production
5. Kubernetes objects and desired state — Explains how Kubernetes continuously brings the actual state closer to the desired state.
   Kubernetes Objects

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI