|
 |
 |
A Brief Communication in Nature Medicine made the rounds last month under a headline that is hard to misread: “General-purpose large language models outperform specialized clinical AI tools on medical benchmarks.” It generated a lot of commentary, a combative public response from one of the named companies, and a request to the journal for a retraction.
The study is fine. The methods are reasonable and the statistics are careful. The problem is the title. It states a general conclusion that the study did not test and does not support. The body of the paper, to the authors’ credit, is far more careful than the headline suggests.
A familiar pattern: titles that outrun the evidence
This isn’t a problem specific to AI. Nutrition research has been living with it for decades.
Consider a 2011 paper titled “Intake of added sugars is not associated with weight measures in children 6 to 18 years.” The title states a sweeping null result. The study behind it was a cross-sectional analysis of NHANES data using a single 24-hour dietary recall per child: a one-day snapshot of self-reported eating. A design like that genuinely cannot establish that sugar is “not associated” with weight: it can’t address reverse causation (heavier kids who have already started cutting back), it can’t capture habitual intake from one recalled day, and it can’t speak to causation at all. The finding may be real within its narrow frame. The title claims something the frame can’t carry.
You can find this pattern repeatedly. A widely cited 2008 meta-analysis concluded that the association between sugar-sweetened beverages and children’s BMI was “near zero”, and even flagged its own evidence of publication bias, before later and larger syntheses found a clear positive association. Reviewers have since documented that industry-funded reviews of sugar and weight were roughly five times more likely to report no association than independent ones.
Two lessons travel from nutrition to medical AI: the scope of a title should match the scope of the evidence, and it always matters who is asking the question and how.
What the paper measured: single-turn general medical knowledge
So what did this study test? Two commercial clinical tools, OpenEvidence and Wolters Kluwer’s UpToDate Expert AI, against three frontier models, GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6. The evaluation had three stages:
• 500 MedQA questions: USMLE-style, multiple-choice medical knowledge.
• 500 HealthBench items: single-turn, free-response prompts graded for alignment with clinicians (a benchmark built by OpenAI).
• 100 “real clinical queries” (RCQ): de-identified single-turn questions that physicians had typed into NYU Langone’s HIPAA-compliant GPT instance, scored blind by 12 clinicians, producing 1,800 annotations.
The frontier models won all three. On MedQA, Gemini hit 97.4% versus OpenEvidence’s 89.6% and UpToDate’s 88.4%. On HealthBench, GPT led at 88.0 with the two clinical tools around 62. On the real-query benchmark, the three frontier models formed the top tier (3.5–3.6 on a 1–4 scale) while the clinical tools (3.17–3.24) came out about even with Google’s free auto-generated Search AI Overview (3.27).
Now look at what these three stages have in common. Every one of them is the same task: answering a single-turn, general medical question, in isolation, with no patient record attached. MedQA is an exam. HealthBench is exam-adjacent. RCQ is a real but still single-turn question. The study measured one capability three times: general medical question answering. That is a legitimate thing to measure. It is not “medical benchmarks,” and OpenEvidence and UpToDate are not all “specialized clinical AI tools.” Two products, one task family.
To the authors’ credit, the body says as much:
• They flag that MedQA and HealthBench may have leaked into training data
• They note HealthBench was built by OpenAI and that GPT-5.2 may benefit from “benchmark-developer overlap”
• They call the RCQ the primary evidence and HealthBench merely supplementary
• They frame the whole thing as “a snapshot of a rapidly evolving landscape,” adding that “deeply subspecialized medical tasks may favor more sophisticated, domain-specific adaptation.”
All of that careful hedging lives under a title that hedges nothing.
One task is not “medicine”
Why does the single-task issue matter so much? Because real clinical work is not a quiz. It’s summarizing a chart, drafting a discharge instruction, extracting a tumor stage from a pathology report, reconciling a medication list, catching an error in a note, turning a clinician’s question into a database query.
This is exactly the gap that MedHELM, the Stanford-led, open-source benchmark published in Nature Medicine, was built to close. MedHELM organizes clinical AI into a clinician-validated taxonomy of 5 categories, 22 subcategories, and 121 distinct tasks, and evaluates models across roughly three dozen benchmarks. Many of these benchmarks are deliberately private or gated, drawn from clinical operations rather than exam material, precisely so the test set can’t be memorized. Its whole premise is that near-perfect exam scores tell you very little about deployment.
You can see the same philosophy in the benchmark suite we publish, which compares John Snow Labs’ medical language models against GPT-5.4, Gemini 3.1 Pro, and Claude Opus 4.6 across 13 clinical and biomedical tasks, which overlaps heavily with MedHELM. A few tasks from these suites, with an example of what each actually asks:
• MEDEC: medical error detection and correction in clinical notes. Given a note, flag whether a sentence contains an error (wrong diagnosis, wrong drug, wrong causal organism), point to the sentence, and rewrite it. On the original benchmark, the best models scored around 70% versus about 80% for physicians.
• MedCalc-Bench: clinical calculations from a patient vignette. For example: given a note, compute the patient’s Cockcroft-Gault creatinine clearance, which requires pulling the right values from the right encounter and applying the right formula. This is the hardest task in the set: direct prompting tops out around 35% on the verified version.
• EHRSQL: turning a clinician’s plain-English question into SQL over a hospital database (”How many patients were prescribed warfarin in the last month?”). Built from questions posed by 200+ real hospital staff, it also includes unanswerable questions to test whether the model knows when to abstain.
• ACI-Bench: generating a structured visit note from a doctor–patient conversation transcript. The input is the messy back-and-forth of a real clinical encounter; the output is the note.
• MedAlign: following a clinician’s instruction against a real, longitudinal patient chart, e.g. “Summarize this patient’s past symptoms, examinations, treatments, and surgeries.” Answering it means synthesizing dozens of notes across years of one real record – something no amount of memorized literature helps with. The data is gated behind a research agreement, so it can’t be crawled, and even GPT-4 got roughly a third of these instructions wrong. The difficulty persists for newer models: a 2025 preprint, TIMER, found the strongest system it tested correct on only about 4