DeepSeek DeepInfer Explained: The Free Engine Behind DeepSeek’s $0.15 AI (2026 Guide)

On October 20, 2026, DeepSeek DeepInfer arrived on GitHub — and it is one of the most unusual open-source releases this year. DeepSeek, the Chinese AI lab famous for selling frontier-class AI at 15 cents per million tokens, published the actual production inference engine that serves its own models to millions of users. The DeepSeek DeepInfer release runs DeepSeek V4.1 Flash, ships under the permissive Apache 2.0 license, and supports two very different chip families: Nvidia’s newest SM100 (Blackwell) GPUs and Huawei’s Ascend 950 NPUs. In other words, DeepSeek just handed the world the software recipe behind the cheapest capable AI on the market.

But this is not a chatbot, and it is not a model you can download and run on a laptop. An inference engine is the software layer that serves a trained AI model to users at scale — it splits the model across chips, batches thousands of simultaneous requests, and decides how fast answers come back. Serving costs are a huge part of why AI is expensive, and DeepSeek DeepInfer is DeepSeek’s answer to that cost problem, battle-tested on its own production traffic. This guide explains what the release contains, how its technology works in plain language, what the benchmark numbers actually mean, where the Huawei angle changes the chip story, and the honest limitations nobody should skip.

In this guide

DeepSeek DeepInfer open-source inference engine running in a modern AI data center

A modern AI data center: this is the hardware environment DeepSeek DeepInfer was built to feed.

What Is DeepSeek DeepInfer?

Every AI model you use — whether through an app, a website or an API — needs a serving system between the raw model weights and your screen. When you type a prompt, the request joins a queue with thousands of others, the model is loaded across dozens of GPUs, your tokens are processed, and the answer streams back. The software managing all of that — batching, memory, scheduling, splitting work across chips — is called an inference engine (or inference framework). It is invisible to users, but it decides two things that matter enormously: how fast responses arrive and how much each response costs to serve.

DeepSeek DeepInfer is DeepSeek’s own inference engine, the one the company says powers its large-scale production services — the same services behind the DeepSeek app, website and API that millions of people use daily. Until October 20, 2026, it was a closely guarded internal tool. Now the code is public on GitHub under the Apache 2.0 license, which means anyone can read it, study it, and even build commercial products on top of it without paying DeepSeek anything.

There is a twist worth noting right away. In April 2025, DeepSeek publicly said it would not release its full serving engine, because the software was too tightly coupled to its internal systems and the team was too small to support an open release. Eighteen months later, it did exactly that. The reversal tells you something about how much the industry has changed: open infrastructure is now a competitive weapon, not just a side project.

The release also includes a companion tool called deepseek-recipe — an MIT-licensed tokenizer component that converts text into tokens and back in exactly the way DeepSeek’s official API does. Think of it as the matching adapter that lets the engine speak DeepSeek’s dialect of tokenization.

What the first release actually covers

Despite the grand framing, the first DeepSeek DeepInfer release is deliberately narrow. Understanding its boundaries matters more than memorizing its features:

  • One model: the release supports a single model — DeepSeek V4.1 Flash (the 1017 build, with its 1-million-token context window, as listed on the model card).
  • Two chip families: Nvidia SM100 GPUs (the Blackwell generation, with CUDA 13) and Huawei Ascend 950 NPUs (with CANN 9.2). Both are data-center hardware — this is not consumer-GPU software.
  • Offline inference only: the release handles batch/offline processing. The live-serving pieces — a KV cache storage system, prefill/decode disaggregation and load balancing across replicas — are not included. DeepSeek says those components are too tied to its internal infrastructure to publish.

DeepSeek has said inference code for future models will land in the same repository, so the supported-model list should grow. For now, though, treat DeepSeek DeepInfer as a production-proven toolkit for one model, not a general-purpose serving engine.

Why This Release Matters (More Than It Looks)

Open-source model weights are now routine — dozens of capable models ship every month. But production serving stacks almost never go open. The inference layer is where an AI company’s cost advantage lives: the custom kernels, the scheduling tricks, the hard-won knowledge of which configurations waste GPU time. Frontier labs treat this as crown-jewel intellectual property. DeepSeek DeepInfer is a rare window into how a frontier lab actually serves models at scale — the equivalent of a championship racing team publishing its engine blueprints.

The timing adds a second layer. The release landed one day after the U.S. government ordered federal agencies to remove DeepSeek’s models from their systems. DeepSeek’s official announcement framed the release as community service — sharing its production stack “to help the community lower compute costs, reduce energy use, and make AI more affordable.” The community noticed: the GitHub repository drew thousands of stars within hours and crossed roughly 8,000 stars in its first day or so, with hundreds of forks. One widely liked reply to DeepSeek’s announcement put the contrast bluntly: OpenAI, “supposed to be open for the benefit of humanity, ended up staying closed.”

There is a third reason this release matters, and it is the one most relevant to anyone building with AI in the United States: the hardware list. Few frontier labs have ever treated a Chinese chip as a first-class citizen in a serving framework alongside Nvidia’s latest. That support arrives in the same month DeepSeek said it plans to pre-train on Huawei silicon by the end of 2026. The chip war and the AI race are converging — and DeepSeek DeepInfer is where they meet in open source.

Inside DeepSeek DeepInfer: The Technology Stack

DeepSeek’s GitHub README credits the engine’s lineage directly: DeepSeek DeepInfer was “originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM” — the open-source serving engines that already power much of the world’s AI infrastructure. On top of that base, DeepSeek stacked its own recent systems research into operational form. Here is what the stack contains, explained without a computer-science degree.

Custom kernels: the hand-tuned math at the bottom

At the lowest level, inference is matrix math. Kernels are small, hand-optimized programs that execute that math on a GPU as efficiently as possible — the difference between a good kernel and a naive one can be the difference between profitable and money-losing serving. DeepSeek DeepInfer bundles the kernels from DeepSeek’s earlier open systems projects:

  • FlashMLA — an optimized attention kernel for DeepSeek’s multi-head latent attention design. Attention is the most expensive operation in a Transformer; making it faster is the single biggest lever in serving.
  • DeepGEMM — a library for fast general matrix multiplication on Nvidia hardware, the bread-and-butter operation of neural networks.
  • DeepEP — communication primitives for expert parallelism (more on that below), letting chunks of a mixture-of-experts model exchange data across chips quickly.
  • TileKernels and DeepSelect — additional specialized kernels, plus custom kernels written in TileLang, a language for writing high-performance GPU kernels more productively.

One detail from the deployment Q&A is refreshingly honest: DeepSeek’s internal FlashMLA decoding kernel is 5–10% faster than the public one in this release, and the company puts the end-to-end gap at 5% or less. So the open release is very close to — but not quite — the absolute bleeding edge of what DeepSeek runs internally.

DSpark: speculative decoding that guesses ahead

The headline technique in DeepSeek DeepInfer is DSpark, DeepSeek’s version of speculative decoding — and it deserves a plain-English explanation, because it is genuinely clever.

Normally, a language model generates text one token at a time: it produces a word, feeds it back in, produces the next word, and so on. Each step requires a full pass through the model, which is slow. Speculative decoding works like a smart assistant relationship: a small, fast draft model guesses several tokens ahead (DeepInfer drafts up to 5 tokens per step, configurable via the DEEPINFER_ENABLE_MTP_VARLEN setting), and then the big model verifies all the guesses in a single pass. Correct guesses are kept; wrong ones are discarded. Because verification is much cheaper than generation, this is usually a big win.

DSpark adds a twist: confidence-scheduled verification. A “Confidence Head” predicts how likely each drafted token is to be accepted, and the scheduler verifies only the most valuable prefix — it skips verifying guesses that are probably wrong anyway. Per DeepSeek’s Q&A, this cuts the number of verified draft tokens by about 30% (from 5.00 down to between 3.2 and 4.4) while the accepted length barely moves (3.29 vs 3.20 tokens per step). In other words: it verifies less and keeps almost the same output quality.

The measured payoff is a 4–7% throughput gain at around 60 sequences per rank versus fixed verification — though DeepSeek notes it can be slightly slower (about 1.5%) at very small batch sizes, where there is less to parallelize.

Parallelism: spreading a giant model across chips

A model like DeepSeek V4.1 Flash — roughly 763 billion total parameters — does not fit on one GPU. Serving it means splitting the work across many chips, and how you split it determines performance. DeepSeek DeepInfer uses two complementary strategies:

  • Attention data parallelism (with balanced context parallelism): the attention computation is distributed across chips, with the context window divided so each chip handles a fair share.
  • Mixture-of-experts expert parallelism (EP) with EPLB: DeepSeek’s models are mixture-of-experts designs — only a fraction of the model’s “experts” fire for each token. Expert parallelism assigns different experts to different chips, and the expert-parallel load balancer (EPLB) keeps the work evenly spread so no chip sits idle while others drown.

The scale is striking: in DeepSeek’s published configurations, decode runs at EP64 to EP128 — that is, the decoding stage is spread across 64 to 128 expert-parallel workers inside a supernode — while prefill uses the smallest EP that memory allows, typically EP2 to EP8. A published EP8-vs-EP64 chart shows the wide configuration ahead at every batch size. The deployment Q&A notes the smallest efficient production unit is 64 prefill GPUs plus 64 decode GPUs — this is data-center-scale software, full stop.

SWA bounded replay and asynchronous scheduling: the quiet efficiency wins

Two more features do unglamorous but important work:

  • SWA bounded replay: DeepSeek V4.1 Flash uses sliding-window attention in some layers. When the engine gets a cache hit, instead of recomputing the entire attention state, it recomputes only the last 128 tokens — making recomputation cost constant regardless of how long the context is. In the decoder layers, DeepSeek says this nearly halves total prefill compute.
  • Asynchronous scheduling: CPU batch scheduling overlaps with GPU execution, so the GPU never sits idle between steps waiting for the next batch plan. It is on by default.

Structured outputs and deterministic sampling

For developers building real applications, two more features matter:

  • Structured outputs: generation can be constrained to a regex, JSON schema or EBNF grammar — the model is forced to emit valid JSON, for example, instead of JSON-adjacent prose that breaks your parser. Anyone who has shipped an AI feature that consumes model output knows how valuable this is.
  • Deterministic sampling and batch-invariant inference: the engine can produce identical outputs for identical inputs and sampling parameters — run the same prompt twice, get the same answer. That sounds simple, but on GPUs it requires real engineering. One caveat: these two features are available on Nvidia hardware only in this release.

Nvidia SM100 GPU hardware supported by DeepSeek DeepInfer for high-speed AI inference

Nvidia’s Blackwell-class GPUs are one of two supported chip families in the DeepSeek DeepInfer release.

How Fast Is DeepSeek DeepInfer? The Benchmarks

DeepSeek published Pareto curves for DeepSeek V4.1 Flash at 131,072-token context, and — to its credit — the methodology is visible: prefill ran on 4 SM100 GPUs, decode on 64. Every number below is DeepSeek’s own, self-reported, read from those charts. No independent lab has reproduced them yet, so treat them as the vendor’s best case — still useful, because they describe the shape of the performance, not just a single headline number.

Decode performance

Decoding — generating the output tokens — is where users feel speed. At the wide EP64 configuration with speculative decoding enabled:

  • Above 25,000 output tokens per second per GPU at 256 requests per rank — the headline throughput figure.
  • Near 100 tokens per second per user at that same high load — comfortably faster than any human reads.
  • At just 8 requests, per-user speed passes 300 tokens per second — the kind of latency that makes a chatbot feel instant.

Prefill performance

Prefill — processing the input prompt — ran at EP4 with 140-token bounded replay:

  • About 60,000 non-cached tokens per second per GPU at 4,096 tokens per rank.
  • Throughput nears 90,000 tokens per second per GPU at 65,536 tokens, at roughly 730 milliseconds per step.

For production context, DeepSeek’s deployment Q&A reports figures that describe the scale this engine handles internally: 155 trillion tokens per day per cluster, a 96% input cache hit rate, average time-to-first-token of around 400 milliseconds, and 230 tokens per second (up to 300 for coding workloads). It also reports overall revenue of US$38 per GPU-hour at “valley pricing” — a rare glimpse at the unit economics of a frontier AI service. Again: DeepSeek’s numbers, unverified by third parties.

How to read these numbers honestly

Two caveats before anyone copies these figures into a pitch deck. First, Pareto curves show throughput-vs-latency tradeoffs, not single-point supremacy — the “25,000 tokens/s” figure holds at a specific, very high concurrency. Second, these are the numbers for one model on two specific chip types. They do not tell you how the engine would perform on a different model — and since only DeepSeek V4.1 Flash is supported right now, you cannot test that question either. Impressive, but provisional.

The Huawei Angle: First-Class Support for Ascend 950

This is the part of the DeepSeek DeepInfer story with implications far beyond one GitHub repository. The release ships with first-class support for Huawei’s Ascend 950 NPUs (via CANN 9.2) right alongside Nvidia’s SM100 GPUs (via CUDA 13). Few frontier AI labs have ever treated a Chinese chip as an equal to Nvidia’s newest in a serving framework — AI Weekly called it the first time a frontier lab has done this.

Why it matters:

  • Supply-chain reality: access to Nvidia’s latest GPUs is constrained for Chinese companies by export controls. A production-proven inference engine that runs equally well on domestic silicon is a strategic asset, not just a technical one.
  • Timing: DeepSeek said just last week that it plans to pre-train on Huawei silicon by the end of 2026. DeepSeek DeepInfer reads as the serving half of that same strategy — train on Ascend, serve on Ascend.
  • Signal to the ecosystem: when a lab of DeepSeek’s caliber writes first-class Ascend support into open code, it lowers the risk for every other Chinese company considering the same path. Open source is also standards-setting.

AI processor chip close-up — DeepSeek DeepInfer also supports Huawei Ascend 950 NPUs alongside Nvidia GPUs

The silicon story: DeepSeek DeepInfer runs on Nvidia’s Blackwell GPUs — and, unusually, on Huawei’s Ascend 950 NPUs too.

For US readers, the practical takeaway is simpler: the world of AI infrastructure is splitting into at least two hardware ecosystems, and the software to serve models on both now exists in the open. The chip war just got a reference implementation.

DeepSeek DeepInfer vs vLLM vs SGLang vs TensorRT-LLM

DeepSeek DeepInfer did not appear from nowhere — it stands on the shoulders of the open serving engines that came before it. Here is how the first release compares, using DeepSeek’s own README and the projects’ public documentation:

Feature DeepSeek DeepInfer vLLM SGLang TensorRT-LLM
Maintainer DeepSeek Community (UC Berkeley origins) Community (LMSYS/UC Berkeley origins) Nvidia
License Apache 2.0 Apache 2.0 Apache 2.0 Apache 2.0
Model coverage 1 (DeepSeek V4.1 Flash) 200+ architectures LLMs, vision models, diffusion Broad (Nvidia support matrix)
Hardware Nvidia SM100, Huawei Ascend 950 Nvidia, AMD, Intel, CPUs + plugins Nvidia, AMD, Intel, Apple, Ascend Nvidia GPUs
Speculative decoding DSpark (confidence-scheduled) n-gram, EAGLE, Medusa variants SpecForge drafts Yes
Online serving No — offline inference only Yes (OpenAI-compatible server) Yes Yes
P/D disaggregation Not in this release Yes Via llm-d / Dynamo Yes
Deterministic sampling Yes (Nvidia only) Limited Limited Limited

The honest reading of this table: DeepSeek DeepInfer is not trying to replace vLLM or SGLang. It is narrower in model coverage, narrower in hardware, and lacks the online-serving machinery those engines ship with. Its advantage is different — it is the actual production configuration one specific frontier lab uses, with that lab’s kernels and that lab’s deployment data. For researchers and infrastructure teams, that production provenance is the entire point.

Why This Is the Secret Behind DeepSeek’s $0.15 Pricing

Here is where DeepSeek DeepInfer connects to something every AI user has felt in their wallet. DeepSeek V4.1 Flash — the model this engine serves — is priced at $0.15 per million input tokens off-peak ($0.30 at peak hours), $0.60 per million output tokens, and a near-zero $0.003 per million cached tokens. That is roughly one-fortieth of what flagship-tier models charge per token. One outlet calculated that the model completes a full coding task for about $0.20 — with benchmark scores that rival models costing 40x more per token.

How can that possibly be sustainable? Part of the answer was already known: DeepSeek’s companion research paper on KV cache compression showed how the model squeezes a 1-million-token context into less GPU memory — memory, not raw compute, being the binding constraint on long-context serving. But the model is only half the equation. The other half is the serving software: batching thousands of requests without waste, guessing tokens ahead with DSpark, recomputing only 128 tokens on cache hits, overlapping scheduling with execution, and spreading experts across 64+ GPUs with balanced loads.

Developer deploying DeepSeek DeepInfer's inference engine on server infrastructure

Behind every “cheap AI” price tag is infrastructure like this — and DeepSeek just open-sourced its version.

DeepSeek DeepInfer is the missing half of the $0.15 story. Cheap inference is a systems problem as much as a models problem, and DeepSeek has now published both halves: the compressed model and the engine that serves it. For the broader industry, the message is pointed: the moat around cheap AI is engineering, and that engineering is now public. Expect every inference startup to be reading this code very carefully.

The same logic explains why DeepSeek can afford generosity elsewhere. The lab also opened a Search API at $0.75 per 1,000 requests — undercutting Brave ($5) and Exa ($4) by more than 5x — because the same cost discipline applies across its stack. When your serving costs approach zero, you can price like it.

The Honest Limitations

Every honest guide needs this section, and DeepSeek DeepInfer needs it more than most — because the hype framing (“DeepSeek open-sources its production engine!”) is bigger than what the code currently delivers. Here is what the release does not do:

  • Offline inference only. There is no online serving stack — no KV cache storage system, no prefill/decode disaggregation, no load balancing across replicas. If you want a live API endpoint, you will build the serving layer yourself around it. For most teams, that means DeepSeek DeepInfer is a component, not a solution.
  • One model. Only DeepSeek V4.1 Flash is supported. You cannot point this engine at Llama, Qwen, or any other model — and the published benchmarks tell you nothing about how it would behave on them.
  • Data-center hardware only. Builds require Linux, GCC 13 or later, Python 3.12, and CUDA 13 or CANN 9.2. The smallest efficient deployment is 64 prefill GPUs plus 64 decode GPUs. This is not software you run on a workstation to “try out” — it is software for organizations that already operate GPU clusters.
  • Self-reported numbers. Every performance figure in this article comes from DeepSeek’s own charts and Q&A. No independent lab has reproduced them. Treat them as the vendor’s best case.
  • The 5% gap. DeepSeek’s internal FlashMLA kernel is 5–10% faster than the public one; the company puts the end-to-end gap at 5% or less. The open release is excellent — but DeepSeek still keeps a small edge for itself.
  • Geopolitics are real. The release arrived a day after a U.S. federal ban on DeepSeek in government systems, and Huawei hardware support is central to the story. Enterprises with U.S. government contracts or strict vendor policies will need legal review before touching any of this — the code is Apache 2.0, but the context is complicated.

None of this makes the release unimportant. But it does mean the audience is narrower than the headlines suggest: infrastructure researchers, GPU-cloud operators, and systems teams — not app developers, and not end users.

Who Should Actually Use DeepSeek DeepInfer

Let’s be practical about who gets value from this release today:

  • AI infrastructure researchers: this is the primary audience. A frontier lab’s real serving stack, with its kernels, scheduling logic and deployment data, is research gold. If you work on inference optimization, this is required reading.
  • GPU cloud providers and neoclouds: operators running large Nvidia or Huawei fleets can study how DeepSeek configures expert parallelism and speculative decoding at 64+ GPU scale — and potentially adapt the techniques.
  • Teams serving DeepSeek V4.1 Flash at scale: if you self-host the model’s MIT-licensed weights (about 510 GB), DeepSeek DeepInfer is now the reference serving stack for it.
  • Students of systems engineering: the README, deployment Q&A and Pareto curves are a free masterclass in production LLM serving.

And who should not bother? Almost everyone else. If you are a developer who just wants cheap, fast AI in an app, the DeepSeek API at $0.15 per million tokens — or OpenRouter’s listing of the same model — is the sane route. If you want to self-host an open model on modest hardware, models like FLUX 3 show the self-hosting formula working at a much smaller scale. DeepSeek DeepInfer is infrastructure for the few, priced in GPU clusters rather than dollars.

Getting Started in 5 Steps

If you are in the target audience, here is the practical path:

  1. Check your hardware. You need Nvidia SM100 (Blackwell) GPUs with CUDA 13, or Huawei Ascend 950 NPUs with CANN 9.2 — on Linux, with GCC 13+ and Python 3.12. Realistically, plan for dozens of GPUs, not one.
  2. Read the README first. DeepSeek’s GitHub README documents the architecture, configuration flags (like DEEPINFER_ENABLE_MTP_VARLEN for DSpark and DEEPINFER_USE_NEXTN_DECODE for draft length), and the Pareto curves. The deployment Q&A alongside it answers the production questions.
  3. Grab the model weights. The engine serves DeepSeek V4.1 Flash — the weights are on Hugging Face under an MIT license (roughly 510 GB). Pair them with the deepseek-recipe tokenizer so tokenization matches DeepSeek’s official API.
  4. Start with offline batches. This release covers offline inference — run batch jobs first, validate the structured outputs (regex, JSON schema, EBNF) against your use case, and measure your own throughput before assuming DeepSeek’s numbers.
  5. Plan the serving layer separately. You will still need a KV cache system, request routing, and load balancing to turn this into a live service. Budget engineering time for that — it is the part DeepSeek kept.

Engineers maintaining server racks powered by DeepSeek DeepInfer inference software

Running DeepSeek DeepInfer at production scale means operating clusters like this — 64 prefill GPUs plus 64 decode GPUs at minimum.

FAQs

Is DeepSeek DeepInfer really free?

Yes — the code is under the Apache 2.0 license, one of the most permissive open-source licenses. You can use it, modify it and build commercial products on it without paying DeepSeek. The hardware to run it is anything but free: expect a data-center GPU cluster.

Should I use DeepSeek DeepInfer instead of vLLM?

Probably not, unless you are specifically serving DeepSeek V4.1 Flash at large scale. vLLM supports 200+ model architectures, runs on far more hardware, and ships a production-ready online server. DeepSeek DeepInfer’s value is its production provenance — it is DeepSeek’s actual stack — not its generality.

Can I run DeepSeek DeepInfer on my own computer?

No. The release targets Nvidia SM100 data-center GPUs or Huawei Ascend 950 NPUs, requires a Linux build environment, and is designed for clusters of 64+ GPUs. For personal use, the DeepSeek app or API is the way to experience the model this engine serves.

Why does Huawei chip support matter?

Because it is the first time a frontier AI lab has given a Chinese chip first-class, equal support in a production serving framework alongside Nvidia’s latest. Combined with DeepSeek’s stated plan to pre-train on Huawei silicon by end of 2026, it signals a serious two-ecosystem future for AI infrastructure — and gives every other Chinese AI company a lower-risk path to follow.

Are the 25,000 tokens/second benchmarks real?

They are DeepSeek’s own measurements, published with visible methodology (4 GPUs for prefill, 64 for decode, 131K context). They have not been independently reproduced. Treat them as credible but provisional — the vendor’s best case, not a third-party verdict.

Will DeepSeek DeepInfer support more models?

DeepSeek says inference code for future models will land in the same repository. For now, only DeepSeek V4.1 Flash is supported — treat any roadmap promises as intentions, not commitments.

Our Verdict

DeepSeek DeepInfer is the most interesting kind of open-source release: not a product, but a peek behind the curtain. It will not change what AI tools you use tomorrow. What it changes is the industry’s understanding of how the cheapest frontier-class AI on Earth actually gets served — the kernels, the speculation, the parallelism, the scheduling — and it hands the Huawei ecosystem a production-grade serving framework at exactly the moment that ecosystem needs one.

For the efficiency story of October 2026 — Liquid AI’s tiny-but-mighty LFM2.5, the open-weight wave, the $0.15 model — DeepSeek DeepInfer is the infrastructure chapter. Models get the headlines; serving stacks decide the economics. DeepSeek just published its economics textbook. The industry’s homework starts now.

All performance figures in this article are DeepSeek’s self-reported numbers from its GitHub README and deployment Q&A, published October 20, 2026. Technical reporting via MarkTechPost and AI Weekly. If you are comparing the broader efficiency landscape, our guides to DeepSeek V4.1 Flash and Liquid AI LFM2.5 are the natural companions to this one.

Leave a Comment

Your email address will not be published. Required fields are marked *