NVIDIA said its full-stack inference software has cut token costs on the DeepSeek V4 model by up to 5x in a single month on the Blackwell platform, the clearest signal yet that software — not silicon refreshes — is now driving the economics of running frontier models. The company published the figures in a June 30, 2026 post by Amr Elmeleegy, citing benchmark data from SemiAnalysis InferenceX on GB300 NVL72 and GB200 NVL72 systems. Stacking four optimizations — disaggregated serving, large expert parallelism over NVLink, NVFP4 precision and multi-token prediction — delivers up to 20x throughput per GPU versus an unoptimized baseline.
The framing matters: NVIDIA is explicitly redefining how customers evaluate AI infrastructure. Peak FLOPS no longer wins the deal; cost per token at a target latency does. That shift moves the competitive battleground from chip launch cycles to how fast a vendor can land software optimizations across model runtimes, kernels, and communication libraries — a game NVIDIA has been quietly compounding for years.
Customer proof points are stacking up. Baseten used NVIDIA TensorRT-LLM to serve DeepSeek V4 Pro on Blackwell, layering proprietary runtime optimizations to deliver up to 50% more tokens per second on reasoning, coding and long-context workloads. Cognition is running NVIDIA Dynamo to manage inference GPUs for reinforcement learning workloads without building the orchestration layer itself. Deep Infra serves frontier open-source models on Blackwell from day zero, and Together AI used TensorRT-LLM on Blackwell to help Cursor push model optimizations into production endpoints for its real-time coding experience.
Key facts
- 01NVIDIA's inference software stack cut DeepSeek V4 token costs by up to 5x on Blackwell in one month.
- 02Stacked optimizations — disaggregated serving, large expert parallelism over NVLink, NVFP4 and multi-token prediction — deliver up to 20x throughput per GPU.
- 03Baseten reports up to 50% more tokens per second serving DeepSeek V4 Pro on Blackwell via TensorRT-LLM.
- 04DFlash speculative decode delivers up to 15x more throughput on existing hardware; FastVideo generates 1080p video in under five seconds.
- 05vLLM and SGLang improved DeepSeek V4 performance on Blackwell roughly 5x in about a month, cutting token costs to one-fifth of prior levels.
The technical claim is that agentic AI broke the assumptions of traditional web-scale serving. A single agent request can spawn hundreds of subagents, thousands of tasks and multiple large language models running across GPUs, CPUs, DPUs and storage. Without coordinated scheduling and memory management, that complexity becomes wasted capacity. NVIDIA's stack tries to solve this across three layers — production operation, application acceleration, and infrastructure access — so individual optimizations compound rather than collide.
The 20x throughput figure is the headline result of that compounding. Each of the four techniques helps on its own; together, on Blackwell with NVLink-connected expert parallelism and NVFP4 precision, they multiply. NVIDIA's argument is that capturing those gains in production requires every layer of the stack to cooperate, which is harder to replicate piecemeal.
“Lower cost per token comes from turning individual optimizations into system-level performance”— Amr Elmeleegy, NVIDIA
The open-source flywheel is the other half of the story. PyTorch, launched in 2016 with native CUDA support, has co-evolved with NVIDIA architectures, surfacing Tensor Cores, Transformer Engine and NVFP4 to developers through a familiar API. New research lands in PyTorch and runs on NVIDIA hardware immediately — DFlash speculative decode arrived with up to 15x more throughput on existing GPUs, and FastVideo can generate 1080p video in under five seconds.
That same dynamic showed up the moment DeepSeek V4 dropped. The leading open-source inference frameworks vLLM and SGLang shipped day-zero deployment recipes for Blackwell, making the model immediately runnable across millions of GPUs. Performance then improved by up to 5x within about a month across both frameworks, cutting token costs to roughly one-fifth of prior levels — without any hardware change.
The caveat is that NVIDIA's published comparisons are NVIDIA-on-NVIDIA: Blackwell with the latest software versus Blackwell with older software. The numbers say nothing direct about competitive positioning against AMD's MI355, Google's TPU v6, or AWS Trainium2 on the same models. SemiAnalysis InferenceX data lends third-party credibility to the throughput claims, but customers evaluating alternatives will need to run their own workloads. The 50% Baseten gain and the 20x stacked figure also assume engineering teams that can actually deploy disaggregated serving and expert parallelism — non-trivial work that smaller shops may not capture in full.
The broader implication is that NVIDIA is building a moat that's harder to see than H100 supply constraints but possibly more durable. Each PyTorch release, each new frontier open model with day-zero CUDA support, each customer optimization fed back into TensorRT-LLM or Dynamo widens the gap between Blackwell economics and anything else on the market. Competitors can match peak FLOPS; matching a decade of CUDA-native framework integration is a different problem. For inference providers deciding where to spend the next billion dollars of capex, the question NVIDIA wants them asking is no longer which chip is fastest — it's which platform will be 5x cheaper per token a month from now.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
