Nvidia published research on Friday showing that a custom software wrapper around Claude Opus 5 lifted the model to a 100% score on ARC-AGI-3, an interactive reasoning benchmark where the same model scores 30% on its own. The finding reframes what actually determines agent performance: not the model weights, but the harness — the scaffolding of tools, memory management, and control logic that sits between the model and the task.
ARC-AGI-3 is a set of 2D games with no instructions. The model has to figure out how to play and win, similar to how a human would approach a new game cold. A 100% score means the agent solves the games at human level. OpenAI's models scored less than 10% on the same benchmark before the company published its own harness research last month, which tripled scores by tweaking two settings — but still fell far short of Nvidia's result.
The distinction matters because most users still treat an agent as an API call to a model. Adel El Hallack, vice president of product in Nvidia's AI unit, argues that framing misses where the work actually happens.
Key facts
- 01Nvidia's custom harness pushed [Claude](/claude) Opus 5 to a 100% score on ARC-AGI-3, up from 30% with no harness.
- 02OpenAI models scored less than 10% on the same benchmark before harness tweaks tripled their results.
- 03Nvidia's harness, called Agentic Variation Operators, adds a supervisor agent that redirects the main agent when it stalls.
- 04Databricks research in July showed that swapping harnesses on the same model can 2x inference costs.
- 05Microsoft tested 19 LLMs on long-horizon document tasks in April and found every frontier model produced error-filled work.
Long-horizon tasks — stringing many decisions together over hours or days to produce completed work — are the current frontier for agent research. Microsoft tested 19 LLMs on long-horizon document editing in April and found that every model, including frontier ones, filled the documents with errors. Agents running unsupervised have also been documented deleting user files, wiping databases, and pursuing objectives through collusion or hacking when left to their own devices.
Nvidia's harness, called Agentic Variation Operators, addresses that drift with a second agent layer. Where tools like Claude Code, Codex, and Hermes typically run a single agent loop, AVO adds a supervisor that watches the primary agent and intervenes when it gets stuck or heads down a dead end.
The supervisor pattern isn't entirely new, but Nvidia's benchmark result is the strongest public evidence yet that it scales. AVO is not a productized offering — Nvidia ships harness components under its open Nemo brand, some commercial and some open source.
The cost implications are the second story. Databricks published research in July showing that harness choice, not model choice, can double the cost of running an agent. CEO Ali Ghodsi put the point bluntly.
That has downstream consequences for how buyers evaluate AI spend. A model priced as premium can behave as budget if paired with a well-tuned harness, and a cheap model can burn through tokens if the wrapper mismanages context. The pricing tier on a model card no longer maps cleanly to the cost of getting work done.
El Hallack framed Nvidia's broader argument around openness. Closed model providers control the harness as much as the weights, which limits how much users can tune for their specific workload. Open harnesses expose more knobs — memory policies, tool selection, supervisor logic — and give buyers a path to raise accuracy without waiting for the next base-model release.
He also tied the argument to security. OpenAI has slowed model training in part because of security issues arising from model outputs. An open agent stack, in Nvidia's telling, lets operators isolate and audit each layer of the system rather than trusting a single vendor's black box.
The counterweight is that Nvidia has an obvious commercial interest in this framing. If the harness is where the value lives, then the infrastructure layer — GPUs, runtimes, orchestration tooling — captures a larger share of the agent economy than the model providers do. Anthropic, OpenAI, and Google have every reason to argue the opposite, and OpenAI's own harness research last month suggests the frontier labs are quietly reaching similar conclusions about where the gains come from.
For AI buyers, the practical takeaway is that model benchmark leaderboards are becoming a poor guide to real-world agent performance. A 30% score and a 100% score on the same benchmark with the same model, separated only by scaffolding, means procurement decisions based on model choice alone are leaving most of the accuracy — and most of the cost variance — on the table. Expect the next 12 months of agent tooling to shift decisively toward supervisor architectures, open harness components, and pricing conversations that finally treat the wrapper as a first-class line item.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
