Nvidia's competitive moat in AI is not its chips. It is CUDA, the software platform that sits between every major machine-learning framework and Nvidia's silicon, and that Jensen Huang has called his most precious treasure. In 2026, with a single frontier training run costing on the order of a hundred million dollars, the difference between a well-tuned CUDA kernel and a generic GPU instruction is the difference between a viable model and a wasted quarter.
The numbers that matter are not transistor counts. A matrix multiplication that takes three lines in PyTorch balloons to more than 50 lines in raw CUDA, and PTX — Nvidia's lower-level assembly-like layer — demands even finer control. DeepSeek's engineers famously dropped down to PTX to wring out extra performance, an option that exists precisely because Nvidia exposes that depth and rivals largely do not.
CUDA stands for Compute Unified Device Architecture, though no one expands the acronym. Its core trick is parallelization. Filling out a 9-by-9 multiplication table requires 81 operations on a single-core machine; a nine-core GPU can split it column by column for a ninefold speed gain, and a GPU that recognizes commutativity can collapse the workload to 45 operations. Stacked across billions of tensor ops, those nanosecond-level savings compound into the throughput that makes large-model training economically possible.
Key facts
- 01CUDA, not Nvidia's chips, is what Jensen Huang calls his most precious 'treasure' and the durable moat in AI.
- 02A matrix multiplication takes three lines in PyTorch but 50-plus lines in raw CUDA — and PTX, the underlying assembly, is harder still.
- 03Independent benchmarks show AMD's MI300X underperforms Nvidia's H100 despite more cores and memory, because frameworks are built on CUDA.
- 04DeepSeek's engineers bypassed CUDA entirely and wrote in PTX to squeeze extra performance from Nvidia GPUs.
- 05Rival stacks — AMD's ROCm, Intel's oneAPI, the Apple/AMD/Qualcomm-backed OpenCL — have all failed to dislodge CUDA.
The platform's origin story sits in the early 2000s, when Stanford PhD student Ian Buck built a GPU programming language called Brook, joined Nvidia, and led CUDA's development with John Nickolls. The hardware they targeted was originally designed to render video-game graphics at 60 frames per second. Nvidia's bet — that the same parallel architecture could serve general high-performance computing — is now the substrate of nearly every frontier AI lab, including OpenAI, Anthropic and Google.
“A matrix multiplication that takes three lines in PyTorch took 50-plus lines in CUDA — the kind of friction that explains why AMD's MI300X underperforms the Nvidia H100 even with better specs on paper.”— Jaeden Schafer
CUDA itself has grown beyond a language. It is a nested bundle of hand-tuned libraries, each function shaving nanoseconds off specific mathematical operations. Modern Nvidia GPUs ship with specialized units — tensor cores, streaming multiprocessors, layered cache hierarchies — that CUDA libraries know how to address directly. A competing GPU with more raw cores and more memory still has to route work through a less mature software stack, which is why benchmarks keep landing the way they do.
Independent researchers comparing the Nvidia H100 and AMD MI300X have found the AMD part underperforms despite better specs on paper. The gap is not silicon. It is that PyTorch, JAX and every other dominant framework target CUDA first and AMD's ROCm second, when at all. ROCm's own user community has documented persistent bugs and compatibility issues, and the broader open alternative — OpenCL, backed years ago by Apple, AMD and Qualcomm — never reached critical mass.
Intel's oneAPI has fared no better. The chipmaker's struggles in 2026 are as much a software story as a manufacturing one, and oneAPI has not pulled significant developer share from CUDA. The most credible challenger may be Modular, the company founded by Chris Lattner, the language designer behind Apple's Swift and the LLVM compiler infrastructure. Modular is attacking the kernel-portability problem directly, but it is still early.
The deeper barrier is talent. Writing GPU kernels well is a niche, grindsome discipline, and most AI researchers — even very good ones — cannot write a single line of C++. The pool of skilled GPU kernel engineers is small, and a large share of them work at Nvidia. Coding agents have started to help with kernel code but still hobble through it, which means the human bottleneck protects the moat for now.
DeepSeek is the obvious counterexample, and worth taking seriously. Its team's willingness to write directly in PTX showed that a sufficiently skilled outside group can extract Nvidia-grade performance without leaning on every CUDA convenience. But DeepSeek's PTX work also reinforces the point: the lower you go, the more specialized the labor, and PTX still runs only on Nvidia hardware. Going around CUDA does not free you from Nvidia.
The pattern that fits Nvidia best is not AMD or Intel — it is Apple. Apple's moat was never the iPhone alone; it was iOS, the App Store, and a developer base that kept building there. Nvidia's hardware is excellent, but the reason customers keep paying its prices is the surrounding software: CUDA, the libraries, the kernel engineers, the framework integrations that ship CUDA-first by default.
For the AI market, the implication is that the chip-supply conversation is mis-framed. The constraint on training capacity is not just fab output or HBM availability; it is the cost of leaving the CUDA ecosystem. Until a rival stack reaches feature parity and a comparable talent pool — neither close in 2026 — Nvidia's pricing power on data-center GPUs holds, and the labs paying those prices will keep paying them. The companies that will pressure Nvidia are not the ones building faster chips. They are the ones building a credible second software stack.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
