DeepSeek releases DeepInfer code for running its AI models
The inference framework currently supports only DeepSeek-V4.1-Flash and works with Nvidia SM100 GPUs and Huawei Ascend 950 chips.
Chinese AI lab DeepSeek made its production inference software, DeepInfer, publicly available on October 20, 2026. Announcing the move on X, the company said it was sharing the framework to help reduce computing costs and energy consumption. The code is available through GitHub with an Apache 2.0 license.
Inference is the process through which a trained model handles requests. DeepInfer combines model-specific optimizations with tools for distributing computation among multiple chips. Its GitHub documentation also describes speculative decoding, which prepares several tokens in advance for checking together, alongside features for consistent outputs and prescribed answer formats.
The package does not provide a complete standalone service. According to the GitHub documentation, it excludes caching, load balancing between machines and the separation of prompt processing from answer generation. DeepSeek also published deepseek-recipe, a tool that replicates its official API’s conversion between text and tokens.
DeepSeek said vLLM provided the main foundation for DeepInfer’s initial development, with contributions also drawn from SGLang and Nvidia’s TensorRT-LLM. The vLLM project said it would use ideas from DeepInfer to improve its own V4.1-Flash support.
The company’s benchmarks used four SM100 GPUs to process prompts and 64 to generate output. Bloomberg’s Saritha Rai reported that DeepInfer’s claimed gains had not yet undergone independent benchmarking.
