I Served the Same Model on vLLM, SGLang, and TensorRT-LLM — the Default Gives Up 29%
Last Updated on June 18, 2026 by Editorial Team
Author(s): Chew Loong Nian – AI ENGINEER
Originally published on Towards AI.
I Served the Same Model on vLLM, SGLang, and TensorRT-LLM — the Default Gives Up 29%
I ran the exact same Llama on three inference engines this week, and the one almost everyone reaches for first finished last. vLLM, the default serving stack for what feels like half the industry, pushed 12,500 tokens per second on a single H100. SGLang pushed 16,200 on the identical hardware and model. That is a 29% gap for free, and it shows up on the workload most of us actually run: agents, RAG, and multi-turn chat.
After introducing the benchmark, the article explains why serving performance matters financially and argues that the “best” engine depends on workload in 2026—especially on prefix overlap common in agentic, RAG, and multi-turn chat scenarios. It profiles the three engines (vLLM with PagedAttention and KV-cache management; SGLang with RadixAttention for aggressive shared-prefix reuse; TensorRT-LLM with NVIDIA-focused compilation and tuning). The author then details the methodology (same H100 hardware, Llama 3.1 8B plus a 70B check, chat vs agent/RAG workloads, and concurrency sweeps with median reporting) and summarizes results: on the 8B model SGLang leads throughput and latency, TensorRT-LLM can edge ahead only after a long compile step, and vLLM is most stable and easiest to deploy. The core finding is the “prefix-cache blowout,” where RadixAttention can turn a 29% gap into up to 6.4× throughput on prefix-heavy traffic, while the advantage shrinks when prefixes are unique or at larger 70B scale. Finally, it provides reproducibility commands and a recommendation framework: choose SGLang for shared-prefix workloads, vLLM for model-swapping and lowest-friction operations, and TensorRT-LLM only for long-term pinned NVIDIA deployments where the setup cost and vendor lock-in are worth it.
Read the full blog for free on Medium.
Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.
Published via Towards AI
Towards AI Academy
We Build Enterprise-Grade AI. We'll Teach You to Master It Too.
15 engineers. 100,000+ students. Towards AI Academy teaches what actually survives production.
Start free — no commitment:
→ 6-Day Agentic AI Engineering Email Guide — one practical lesson per day
→ Agents Architecture Cheatsheet — 3 years of architecture decisions in 6 pages
Our courses:
→ AI Engineering Certification — 90+ lessons from project selection to deployed product. The most comprehensive practical LLM course out there.
→ Agent Engineering Course — Hands on with production agent architectures, memory, routing, and eval frameworks — built from real enterprise engagements.
→ AI for Work — Understand, evaluate, and apply AI for complex work tasks.
Note: Article content contains the views of the contributing authors and not Towards AI.
Related posts
Recent Posts
