Author(s): Aya Mahmoud

Originally published on Towards AI.

DeepInfer ships DeepSeek’s production kernels, speculative decoding and deterministic sampling under Apache 2.0. Here is what’s actually in the repo, what you’d still have to build, and who can run it today.

DeepSeek put the inference framework behind its own API on GitHub under Apache 2.0.

After the lead, the article explains that DeepInfer is built largely on vLLM with ideas from SGLang and TensorRT-LLM, then shows what DeepSeek added (custom kernels, parallelism strategies, and small but meaningful production optimizations like DSpark speculative decoding and bounded SWA replay). It highlights product-relevant features DeepInfer includes—structured outputs via regex/JSON schema/EBNF, plus deterministic sampling and batch-invariant inference on NVIDIA hardware. The author then details what DeepInfer deliberately does not include: key “service” components like KV cache management, prefill/decode disaggregation orchestration, and replica-level load balancing—arguing these missing pieces reveal where DeepSeek still retains operational advantage. Finally, it covers who can run it (specific supported hardware and model family, with notes that Ascend support is thinner than headline claims), where to start (build and run offline inference, plus environment-variable switches for reproducibility and features), and concludes with the broader takeaway that the moat is shifting away from the raw engine and toward fleet-level caching, routing, and scheduling.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI