NVIDIA's GB300 NVL72 platform runs up to 20x more AI agents per megawatt than the company's prior-generation HGX H200, according to the first published results of AgentPerf, a new benchmark from Artificial Analysis designed specifically for agentic workloads. The June 12 results put Blackwell Ultra on top of the inaugural ranking and give buyers their first apples-to-apples view of how accelerated computing systems handle agents rather than single chat completions. The benchmark measures concurrent agents at service-level objectives of 20 and 60 tokens per second per agent, two thresholds meant to capture real production responsiveness.
The test workload is DeepSeek V4 Pro, a frontier mixture-of-experts model representative of the systems powering today's most capable agents. AgentPerf runs it against real coding-agent trajectories sourced from public repositories spanning more than 12 programming languages, with the agent reading files, writing and editing code, executing commands, and iterating on results. Tool calls are simulated using representative CPU processing time so the scores reflect accelerated compute performance only.
The distinction between agentic and conversational inference is the entire point of the new benchmark. A chat completion is one model call and one response; an agent chains dozens to hundreds of model calls together, each passing growing context to the next, with tool calls like code compilation, database search, and web browsing at every handoff.
“Agentic AI is a fundamentally different workload than conversational AI. A single chat completion is a sprint: one large language model (LLM) call, one response. An agent functions more like a relay: It breaks a goal into many steps and keeps going until the task is done.”— Shruti Koparkar, NVIDIA
Key facts
- 01NVIDIA GB300 NVL72 runs up to 20x more agents per megawatt than the HGX H200 on AgentPerf's DeepSeek V4 Pro workload.
- 02AgentPerf, from Artificial Analysis, is the first benchmark built specifically for agentic AI rather than single LLM calls.
- 03GB300 NVL72 connects 72 GPUs into one rack-scale system to distribute mixture-of-experts model execution.
- 04The benchmark measures concurrent agents at service-level objectives of 20 and 60 tokens per second per agent.
- 05Baseten, DeepInfra, and Together AI are already serving DeepSeek V4 Pro agentic workloads on Blackwell.
Existing inference benchmarks measure how fast a model responds to a single request and how many simultaneous requests a box can handle. They miss the chained calls, growing context windows, and tool-call latencies that define what an agent actually does in production. That gap is what AgentPerf is trying to close, and it is the reason Hopper-era numbers can look very different once the workload is structured as a relay rather than a sprint.
NVIDIA attributes the 20x gain to rack-scale codesign rather than any single chip improvement. GB300 NVL72 connects 72 GPUs into one system, letting a large mixture-of-experts model like DeepSeek V4 Pro distribute execution across experts without paying a heavy coordination tax. CUDA kernels overlap communication and compute so the cost of routing between experts is absorbed rather than added to latency.
“The complexity isn't additive; it's multiplicative.”— Shruti Koparkar, NVIDIA
NVIDIA TensorRT LLM handles the software side, separating input processing from output generation so each can be optimized independently as concurrent agent sessions scale. The result, per NVIDIA's published charts, is that GB300 NVL72 supports far more concurrent agents per megawatt than H200 at both the 20 and 60 tokens-per-second thresholds. The efficiency gap widens as more agents are stacked onto the same infrastructure.
The economic argument NVIDIA is making is straightforward: per-megawatt agent throughput translates directly into how much productive work a data center investment can deliver. For enterprises building agent fleets, that number matters more than peak single-query latency, because agents are long-running, multi-step processes that consume capacity for the duration of a task, not a single response. Power, not silicon, is increasingly the binding constraint on agent deployments at scale.
Three inference providers — Baseten, DeepInfra, and Together AI — are already serving DeepSeek V4 Pro and other frontier models on Blackwell for production agentic applications. Together AI powers real-time inference for Cursor, the AI coding platform whose agents debug issues, generate features, and execute refactors while developers continue working. DeepInfra runs Pam.ai, an AI workforce platform for car dealerships that books service appointments, handles calls, and runs outbound sales campaigns entirely on Blackwell.
AgentPerf is the first round, and NVIDIA has an obvious interest in a benchmark on which its newest rack-scale system tops the chart against its own prior generation. The real competitive test will come when AMD's MI400-class systems and custom silicon from hyperscalers post numbers on the same benchmark, and when Artificial Analysis expands the workload set beyond coding agents. The H200 comparison also flatters the headline figure — a generation-over-generation gain at the rack level is expected to be large.
The launch lands as NVIDIA pushes into its next architecture. The company says Vera Rubin is now in full production, and CEO Jensen Huang is scheduled to keynote GTC Taipei on June 1. If AgentPerf becomes the standard buyers use to size agent fleets, the metric NVIDIA cares about shifts from tokens per second on a single query to concurrent agents per megawatt — a framing that favors rack-scale integration and full-stack software optimization, which is exactly where NVIDIA's moat is widest. Expect every competing accelerator vendor to either run the benchmark or argue publicly that it measures the wrong thing.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
