DeepSeek DeepInfer: Open-Source Inference Engine Explained

DeepSeek DeepInfer is the inference framework that runs DeepSeek-V4.1-Flash behind DeepSeek's own API, and the company published it on GitHub under the Apache 2.0 license on October 20, 2026. The first release serves one model, runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and leaves out the KV cache storage and load balancers a full online service needs. In April 2025 DeepSeek had said it would not release its serving engine, so this is the first time outsiders can run the code DeepSeek runs in production.

The repository's first public commit is stamped 04:09 UTC on October 20, and the Hacker News thread about it passed 2,200 points the same day. By the afternoon of October 21 (KST) the repo had about 15,000 GitHub stars.

Key takeaways

• DeepInfer is, in its README's words, "the high-performance inference framework that powers DeepSeek's large-scale production services." The open release covers offline (batch) inference with the core serving features.
• It supports one model, DeepSeek-V4.1-Flash, and loads the deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint by default.
• It runs on NVIDIA SM100 GPUs (the B200 and GB200) and Huawei Ascend 950 NPUs. Hopper GPUs such as the H100 are not on the list.
• The release has no KV cache system, no disaggregated vision-encoder, prefill and decode serving, and no load balancing across data-parallel replicas. DeepSeek says those parts are tied to its internal infrastructure.
• DeepSeek says one cluster on this stack serves 155 trillion tokens a day and earns about $38 per GPU-hour at off-peak prices.
• DeepInfer was originally built on vLLM, with ideas from SGLang and TensorRT-LLM. No head-to-head benchmarks have been published.

What is DeepSeek DeepInfer?

An inference framework is the software between a trained model and its users: it batches requests, schedules prefill (reading the prompt) and decode (generating tokens), and spreads the work across accelerators. vLLM and SGLang are the best-known open-source examples.

DeepInfer is DeepSeek's own version of that layer. The README says this release "provides offline inference with the core features needed for serving at scale." The public entry point is a Python LLM class with a generate method, plus example scripts for offline runs, benchmarks and evaluations. The repository has no HTTP API server.

The deployment Q&A that ships with the code describes the production system. DeepSeek writes that "a single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing)," and says inference code for future DeepSeek models will be released in the same repository.

DeepInfer is paired with deepseek-recipe, a Rust library with Python bindings that turns Chat Completions, Responses and Messages API requests into DeepSeek's token format. The Q&A says its logic matches the official DeepSeek API, so a third-party deployment behaves the same way at the API level.

Why DeepSeek changed course on its inference engine

In April 2025 DeepSeek published The Path to Open-Sourcing the DeepSeek Inference Engine. It had considered releasing its internal engine and decided against it. The engine was "an early fork of vLLM" heavily customized for DeepSeek models, it was tightly coupled to internal cluster management tools, and the team lacked the bandwidth to maintain a large open-source project. DeepSeek said it would contribute standalone libraries and optimizations to existing projects instead.

DeepInfer reverses that decision only in part. The README still credits vLLM: DeepInfer "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM." The pieces that stay closed are the ones DeepSeek called infrastructure-bound in 2025. According to the Q&A, the KV cache management and load balancing components "are tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced." A Zhihu post describes their design instead.

DeepInfer's kernels come from libraries DeepSeek released in the meantime: FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom kernels written in TileLang.

What DeepInfer includes

The README lists these features:

• Optimized kernels for attention, matrix multiplication and expert communication, from the libraries above.
• Parallelism. Attention can run data-parallel (DP) or with balanced context parallelism (CP). The mixture-of-experts layers use expert parallelism (EP) with an expert-parallel load balancer (EPLB) that places copies of busy experts on more GPUs.
• DSpark and confidence-scheduled verification, DeepSeek's form of speculative decoding. A draft model proposes several tokens and the main model checks them in one pass. A confidence head predicts which draft tokens will be accepted, so verification compute goes to the likely ones.
• SWA bounded replay, which rebuilds the sliding-window attention (SWA) cache by replaying a short recent slice of tokens instead of the whole context.
• Asynchronous scheduling: a "zero-overhead" batch scheduler plans the next step on the CPU while the GPU runs the current one. It is on by default.
• Batch-invariant inference and deterministic sampling, so the same input and sampling settings give the same output whatever else is in the batch. Both work on NVIDIA only.
• Structured outputs that constrain generation to a regex, a JSON schema or an EBNF grammar.

The Q&A puts numbers on two of these. DSpark's draft model proposes 5 tokens at a time in production. Verifying a variable number per request, instead of all 5, cuts verified draft tokens per step by about 30% while the average accepted length barely changes; DeepSeek calculates a 4 to 7% throughput gain at its usual load of about 60 concurrent sequences per rank. On the decoder side, bounded replay recomputes only the last 128 prompt tokens, which DeepSeek says nearly halves total prefill computation.

These features follow V4.1-Flash's architecture, a causal encoder-decoder with a heavily compressed KV cache. Our DeepSeek V4.1-Flash-1017 explainer covers that design along with the model's benchmarks and API prices.

What DeepInfer leaves out

The README lists three things a complete inference service needs that this release does not include:

1. A KV cache system, the store that keeps the attention keys and values of earlier tokens so a repeated prompt prefix is not recomputed. In DeepSeek's production setup the cache sits on a dedicated 3FS storage cluster with about 100 PiB of SSD, and the Q&A says cached prefixes are kept for 3 days for agent traffic and 7 days for chatbot traffic.
2. VE/Prefill/Decode disaggregation, which puts the vision encoder (VE), prompt processing and token generation on separate machines. The README says most of the engine functionality this needs is already in place.
3. Load balancing across data-parallel replicas for both prefill and decode.

Those gaps matter at DeepSeek's scale. Its recommended layout uses small expert-parallel groups (EP2 to EP8) for prefill and wide groups of 64 to 128 GPUs for decode, and the Q&A puts the smallest efficient deployment at 64 GPUs of prefill plus 64 GPUs of decode. An outside team gets a fast engine for one group of machines and has to build the cache store and request routing around it.

Which hardware can run DeepInfer?

The README names two accelerator families:

• NVIDIA SM100 GPUs. SM100 corresponds to compute capability 10.0, which NVIDIA's CUDA GPU list assigns to the B200 and GB200. Hopper GPUs (H100 and H200, compute capability 9.0) are not listed, and neither are the B300 and GB300, which NVIDIA lists at compute capability 10.3.
• Huawei Ascend 950 NPUs, using Huawei's CANN 9.2 software stack.

The other prerequisites are Linux, GCC 13 or newer, Python 3.12 and, on NVIDIA, CUDA 13. The default V4.1-Flash-1017 checkpoint on Hugging Face is 48 safetensors shards totaling about 510 GB, and every benchmark DeepSeek published for DeepInfer uses at least 4 SM100 GPUs. In practice that means cloud providers, large companies and research labs with Blackwell or Ascend 950 clusters.

Why the Ascend 950 path matters

vLLM runs on Ascend through a community-maintained plugin, vllm-ascend. DeepInfer supports Ascend from its first release, and one build variable, DEEPINFER_ACCELERATOR, set to nvidia or ascend, picks the platform.

On September 30, DeepSeek released open-source programming tools for Huawei's Ascend chips, including Ascend support for the TileLang kernel language and Ascend versions of its compute and communication libraries, TNW reported citing Reuters. The two companies also built a supernode of 128 Ascend 950 chips. Earlier in September, Bloomberg reported that DeepSeek plans to deploy at least 160,000 Ascend 950DT accelerators in an Inner Mongolia data center, for inference rather than training, according to TechNode.

DeepInfer is the kind of software those chips would run, and anyone with Ascend 950 hardware can now use the engine DeepSeek uses in production. Policy is moving as well; our overview of China's draft AI export controls covers Beijing's proposed rules for models, agents and chips.

The Ascend support is still unmeasured in public. All of DeepInfer's published performance charts come from NVIDIA GPUs, the two determinism features work only on NVIDIA, and DeepSeek has released no Ascend throughput numbers.

DeepInfer performance: what the charts show

DeepSeek published two Pareto charts and no table. According to the README, prefill was measured on 4 SM100 GPUs and decode on 64, each within one scale-up domain (a group of GPUs joined by the fast in-node or supernode interconnect). The script examples/sweep/run_sweep.sh reproduces them.

Read off the prefill chart, throughput climbs from roughly 60,000 tokens per second per GPU at a 4,096-token batch (about 70 ms per step) to just under 90,000 at a 65,536-token batch (about 730 ms).

In the decode chart, at 8 requests per rank each user gets about 325 tokens per second. At 256 requests per rank each GPU produces about 26,000 tokens per second while each user gets about 100. The chart assumes a 75% per-token draft acceptance rate, so these figures are modeled from benchmark runs rather than measured on live traffic.

All of these are DeepSeek's own numbers, and DeepSeek has not compared DeepInfer with vLLM or SGLang on the same hardware. The open kernels also differ slightly from the internal ones. The Q&A says the public FlashMLA lacks one hand-tuned SASS optimization, which makes its decoding kernel 5 to 10% slower, and that end-to-end performance is at most 5% apart.

DeepInfer vs vLLM and SGLang

DeepInfer shares code history with vLLM and ideas with SGLang, but it covers far less ground.

Sources: DeepInfer README, vLLM README, vLLM recipe for DeepSeek-V4.1-Flash, SGLang README, SGLang cookbook for DeepSeek-V4.1. Checked October 21, 2026.

If you need an HTTP endpoint for V4.1-Flash today, or you run H200 or AMD GPUs, vLLM is the shorter path, and SGLang has a preview build. DeepInfer shows how DeepSeek gets its production throughput, and it is where DeepSeek says inference code for its next models will appear.

How to try DeepInfer

The README and the examples guide give this path:

1. On Linux, create and activate an empty Python 3.12 environment.
2. Run export DEEPINFER_ACCELERATOR=nvidia (or ascend) and keep it set wherever you run DeepInfer.
3. Build with ./build.sh.
4. Run python examples/offline_inference.py. The first run downloads deepseek-ai/DeepSeek-V4.1-Flash-1017 into the Hugging Face cache; set DEEPINFER_HF_MODEL_DIR to use a local copy.

The examples show parallel layouts such as --ep 8, --ep 2 --dp 8 and --ep 2 --cp 8. Environment variables switch features on, for example DEEPINFER_ENABLE_MTP_VARLEN=1 for DSpark's variable-length verification and DEEPINFER_BATCH_INVARIANT=1 for batch-invariant inference. The eval folder has scripts that score Pass@1 on AIME 2026, HLE and GPQA Diamond, so you can check model quality on your own build.

Who should pay attention

The clearest audience is cloud providers and large companies serving DeepSeek models on B200, GB200 or Ascend 950 clusters, who get DeepSeek's own scheduling, speculative decoding and expert-balancing code. Researchers who need reproducible outputs on NVIDIA also get batch-invariant inference and deterministic sampling.

Everyone else is better served by the DeepSeek API, vLLM or SGLang. If you want an open model that runs on a single desktop machine, Meta's Muse Spark Open on a Mac Studio is a closer fit.

Things to watch next: Ascend 950 benchmark numbers, independent comparisons with vLLM and SGLang, whether the KV cache and load-balancing code gets released, and DeepInfer support for the next DeepSeek model.

FAQ

What is DeepSeek DeepInfer?

DeepInfer is DeepSeek's high-performance inference framework, the software that serves DeepSeek-V4.1-Flash in DeepSeek's production services. DeepSeek released it on GitHub under the Apache 2.0 license on October 20, 2026. The open version supports offline inference for one model on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

Is DeepInfer free for commercial use?

Yes. DeepInfer is released under the Apache License 2.0, which allows commercial use, modification and redistribution. The DeepSeek-V4.1-Flash-1017 weights it loads are published separately on Hugging Face under the MIT license.

Can DeepInfer run on an H100 or an RTX GPU?

Not officially. The README lists only NVIDIA SM100 GPUs (B200 and GB200, compute capability 10.0) and Huawei Ascend 950 NPUs. Hopper GPUs like the H100 and consumer RTX cards have other compute capabilities and are not on the list. vLLM's DeepSeek-V4.1-Flash recipe does list the H200.

Is DeepInfer faster than vLLM or SGLang?

There is no public answer yet. DeepSeek published throughput charts for DeepInfer on NVIDIA SM100 GPUs but no same-hardware comparison with vLLM or SGLang.

Does DeepInfer include an OpenAI-compatible API server?

No. The release provides offline inference through a Python LLM class and example scripts. It also leaves out the KV cache system, prefill/decode disaggregation and data-parallel load balancing that DeepSeek uses in production.

Sources

• DeepInfer repository and README, GitHub
• DeepSeek V4.1 Flash Inference Deployment Q&A (DeepInfer docs)
• DeepInfer examples guide
• The Path to Open-Sourcing the DeepSeek Inference Engine (April 2025)
• Hacker News: DeepInfer discussion
• NVIDIA: CUDA GPU compute capability
• DeepSeek-V4.1-Flash-1017 on Hugging Face
• vLLM recipe: DeepSeek-V4.1-Flash
• SGLang cookbook: DeepSeek-V4.1
• SiliconANGLE: DeepSeek open-sources DeepInfer
• TNW: DeepSeek open-sources Huawei chip tools as a simpler alternative to CUDA
• TechNode: DeepSeek plans 160,000 Huawei Ascend 950DT chips (Bloomberg report)