Microsoft AI says its MAI-DxO diagnostic system correctly diagnosed 83% of complex New England Journal of Medicine case reports, compared with 36% for physicians working with textbooks and internet access. The results were published October 22 in Science, and Microsoft is now taking the system into a prospective study with Beth Israel Deaconess Medical Center in Boston.
MAI-DxO is an orchestration layer rather than a single model. Instead of producing one immediate answer, it coordinates several AI agents that act like a panel of virtual physicians: they question a simulated patient, order tests and debate a diagnosis before committing to one. The benchmark, SDBench, includes 304 complex NEJM cases plus a set of emergency department presentations from the MIMIC-IV-ED dataset.
Without external resources, physicians in the study reached 22% accuracy on the NEJM cases, making MAI-DxO nearly four times as accurate; with resources, it was more than twice as accurate. Microsoft said MAI-DxO reached those results at half to one-third of the cost, ordering fewer tests and procedures, and that one cost-optimized configuration hit 81% on NEJM cases at less than half the cost of comparable systems.
Key facts
- 01MAI-DxO correctly diagnosed 83% of complex NEJM cases; physicians with textbooks and internet access reached 36%, and 22% without them.
- 02On emergency department cases from MIMIC-IV-ED, Microsoft reported 80% accuracy for MAI-DxO versus 47% for physicians.
- 03Paired with DxO, GPT-5.5 rose from 72% to nearly 87% on complex cases; Gemini 2.5 Flash rose from 46.4% to nearly 69%.
- 04The results were published October 22 in Science; Beth Israel Deaconess Medical Center will run a prospective study.
- 05Microsoft is working with Mayo Clinic on a healthcare foundation model planned for release next year.
On common US emergency department presentations, Microsoft reported MAI-DxO at 80% accuracy versus 47% for physicians, or 1.7 times more accurate. "Across both datasets, no other human or AI approach was both more accurate and less costly than MAI-DxO," the company's MAI Health Team wrote.
Microsoft tested the harness with models from OpenAI, Google, Anthropic, Meta and DeepSeek, with the strongest results coming from OpenAI's GPT-5 and GPT-5.5. GPT-5.5 rose from 72% to nearly 87% on complex cases with DxO, at slightly lower cost. Gemini 2.5 Flash went from 46.4% alone to nearly 69%. GPT-5 alone scored 78.9% at about $8,400 per case; DxO cut roughly $3,000 from that and added about four accuracy points.
Microsoft said the gains held and improved as newer models arrived, without changes to the orchestration method. The study's data also indicates that much of the improvement comes from the underlying models, with DxO adding gains on top. On emergency department cases the orchestration added little: GPT-5.5 scored about 85% with or without it, at similar cost.
The comparison has a stated limit. NEJM cases this complex are usually handled by specialists or teams of physicians, and those groups were not part of the study. The human baseline was primary care doctors and general internists working alone.
Microsoft framed the benchmark as a first step. "Benchmarks matter, but the true test is performance in real clinical environments," the MAI Health Team wrote. In the Beth Israel study, patients who consent will receive DxO recommendations alongside standard care, and those recommendations will be compared with the treating physicians' independent assessments.
Microsoft is also working with Mayo Clinic on a healthcare foundation model that combines frontier AI, clinical expertise and longitudinal multimodal data, with release planned for next year. The company has open-sourced SDBench for select NEJM cases, including the benchmark, its Gatekeeper and Judge components and the cost-estimation code.
Mustafa Suleyman, Microsoft AI's CEO, called the result a step toward "medical superintelligence" and predicted that healthcare models will get "an order of magnitude better" within 24 to 36 months. Dominic King, Microsoft AI's vice president of health, described the growing mismatch between demand for medical expertise and its supply as the biggest challenge in healthcare delivery.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
