If your team serves large models, DeepSeek just handed you a reference design worth reading, though probably not a drop-in replacement for your current stack. DeepInfer, the inference engine DeepSeek uses behind its own API, is now open source under Apache 2.0. It offers real optimizations for throughput and cost, but it covers a narrow set of hardware and leaves out several pieces a full production service needs.
What happened
According to The New Stack, DeepSeek released DeepInfer on 20 October 2026 so developers can run its V4.1 Flash model on their own infrastructure with many of the optimizations it applies in its API. Support for future DeepSeek models is planned. The package bundles optimized kernels and communication libraries, including the previously released FlashMLA, DeepGEMM and DeepEP, plus mechanisms for spreading inference across chips. It also supports speculative decoding, asynchronous scheduling and structured outputs.
A companion library, deepseek-recipe, handles how API requests become model inputs and how outputs become responses. It exists for Rust and Python and aims to help self-hosted deployments mimic the official API’s behavior, The New Stack reports.
The release is partial. The documentation describes support for offline inference with the core features needed for serving at scale, but a KV cache management system, disaggregation of inference stages across machines and replica load balancing are missing. Hardware support is limited to Nvidia’s SM100 (Blackwell) architecture and Huawei’s Ascend 950 NPUs. Hopper is absent. Batch-invariant inference and deterministic sampling work only on Nvidia, and DeepSeek’s published benchmarks use Nvidia GPUs, with four SM100 GPUs for prefill and 64 for decoding.
Why it matters
Open weights have never guaranteed efficient serving. The engine that schedules requests and moves work between chips shapes cost per token as much as the model does. Until now, outside teams ran DeepSeek models on third-party engines such as vLLM. In 2025 DeepSeek said its own engine was too tied to internal infrastructure to release. The new scope suggests how it resolved that: ship the compute core, leave cluster plumbing out.
The hardware list is the strategic signal. Putting Huawei’s Ascend alongside Nvidia from day one fits earlier reports, cited by The New Stack, of DeepSeek planning large Ascend deployments and collaborating with Huawei on programming tools. This suggests a serving stack that treats non-Nvidia silicon as a first-class target, which matters to anyone worried about single-vendor dependence.
What it means for you
Check your hardware first. If you run Hopper or older GPUs, you cannot use DeepInfer directly. Its value to you is in the code: the kernels and scheduling ideas can inform tuning of the engine you already run.
If you run Blackwell and serve DeepSeek models, benchmark DeepInfer against your current engine on your own traffic, not DeepSeek’s 4-plus-64 GPU setup. Budget engineering time for the missing layers: KV cache handling, prefill and decode separation and load balancing. Those are where self-hosted services often win or lose on cost, so the headline efficiency may not survive without them.
If you depend on reproducible outputs, note that deterministic features are Nvidia-only. Ask any vendor offering DeepSeek hosting which engine and hardware they use, and whether results match the official API. For pricing comparisons across hosted options, the AI Tool Radar tracks current AI tool costs.
What to watch
- Whether DeepSeek fills in KV cache management, disaggregated serving and load balancing, which would make DeepInfer a full stack.
- Independent benchmarks on Ascend 950 hardware, since DeepSeek’s own numbers cover only Nvidia.
- Whether Hopper support arrives or its techniques land in vLLM and other engines.
- How quickly future DeepSeek models gain day-one support.