Author(s): Ashish Nishad

Originally published on Towards AI.

Hardware math, quantization tradeoffs, five inference engines benchmarked, and two real case studies where the numbers met reality on my own machine.

Most “run an LLM locally” guides stop at ollama pull and a happy screenshot. This one doesn’t. This is the reference I wish existed when I started actually building on local models instead of just trying them once: real hardware math, the tradeoffs between five different inference engines, what quantization actually costs you, and — because benchmarks lie by omission — two real tests I ran myself where the theory either held up or didn’t.

The article explains how to run LLMs locally in 2026 by grounding choices in hardware math (especially quantization memory costs and how context length increases KV-cache usage), then separating “engine” capabilities from simple Ollama-vs-vLLM comparisons (Ollama wraps llama.cpp with convenience but no continuous batching; llama.cpp offers low-level control; vLLM targets high-throughput multi-user GPU serving via PagedAttention and continuous batching; text-generation-webui focuses on exploration and LoRA; SGLang targets structured/agentic JSON-like generation). It challenges widely quoted benchmarks by showing that engine gaps mostly appear only under concurrent load, while single-user performance is closer and influenced heavily by quantization formats. It provides practical setup steps for Ollama, llama.cpp, and vLLM, and concludes with two real case studies on a 16GB Apple Silicon Mac—Qwen3–8B matching expectations within the “workable floor,” and GLM-4.7-Flash feeling slow because it’s running at the hardware limit—leading to a final decision framework: compute parameter+KV-cache needs, pick quantization (default Q4_K_M; use higher Q only if you have headroom), choose the engine based on single-user vs concurrent and structured-vs-chat workloads, verify the specific model variant’s actual size/file format, and test your intended tasks on your own hardware.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI