Medical AI · Generative AI & LLMs · Lesson 3 of 7

Medical LLMs and their benchmarks

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on 2026-10-18 · Last updated 2026-10-18

Intermediate
Clinical focus
About 30 minutes
Includes exercise

Before this lesson: How large language models work, Diagnostic accuracy metrics: sensitivity, specificity, PPV, ROC and calibration

In short

Medical LLMs are usually compared on benchmarks: exam-style multiple-choice questions such as MedQA, rubric-graded conversations such as HealthBench, and task suites such as MedHELM. As of 2026, many general-purpose models score above 90% on MedQA, so the exam benchmark no longer separates them well. More importantly, a randomised study found that people using LLMs did no better than a control group at identifying conditions, even though the models alone answered well. Benchmark scores are a starting point, not evidence of clinical benefit.

You will learn to

Describe the main types of medical LLM benchmark and what each measures
Explain benchmark saturation, contamination and model-graded evaluation
Calculate a confidence interval for benchmark accuracy and compare two models fairly
Critically appraise a claim that an LLM 'performs at expert level'

What medical benchmarks measure

As of October 2026. Benchmarks are updated and superseded often; check the current version.
Benchmark	Format	What it tests	Notes
MedQA (2021)	Multiple-choice questions in the style of the US medical licensing exam; the US English test set has 1,273 questions	Recall and applied knowledge	Widely reported; now near saturation
MultiMedQA (2023)	A collection of question sets plus consumer health questions	Knowledge, and the quality of long-form answers rated by clinicians	Introduced with Med-PaLM
MedXpertQA (2025)	Harder expert-level questions, including images	Advanced reasoning	Built to separate models that saturate MedQA
HealthBench (2025)	5,000 multi-turn conversations, graded against rubrics written by 262 physicians	Open-ended answers to patients and clinicians, including safety behaviours	Grading is done by a language model
MedHELM (2026)	A taxonomy of 121 clinician-validated tasks and a suite of 37 benchmarks	Real clinical work beyond exams: documentation, decision support, administration	Combines exact-match and model-graded scoring

Saturation: when the exam stops being informative

In 2023, Med-PaLM 2, a medically tuned model, reached 86.5% on MedQA (Singhal et al., 2025). By 2026 many general-purpose models, without medical fine-tuning, score above 90%. When most models are close to the ceiling, small differences mostly reflect noise, question errors or memorisation rather than meaningful capability.

Contamination is a related problem: if benchmark questions, or close paraphrases, appeared in a model's training data, its score partly reflects memory. Because training data are rarely disclosed, contamination is hard to rule out for public benchmarks.

Exam performance is not patient benefit

Bean and colleagues (Nature Medicine, 2026) randomised 1,298 adult members of the public to use one of three LLMs, or whatever sources they would normally use, to work through ten medical scenarios. Tested alone, the models identified relevant conditions in 94.9% of cases. Participants using the same models identified relevant conditions in under 34.5% of cases and chose the right course of action in under 44.2%, no better than the control group.

Users gave incomplete information, misread the models' answers or ignored good suggestions. The gap between a model's knowledge and the outcome for the person using it is exactly what benchmarks do not measure.

What would show clinical value

Evaluation on real tasks, with the intended users, in the intended setting.
Outcomes that matter: correct decisions, time saved, errors caught or introduced, patient outcomes.
Comparison with current practice, ideally in a prospective or randomised study.

Uncertainty in benchmark scores

A benchmark score is an estimate from a sample of questions, so it needs a confidence interval. Many published comparisons use small subsets.

python

import numpy as np

def wilson_interval(correct, n, z=1.96):
    p = correct / n
    centre = (p + z**2 / (2 * n)) / (1 + z**2 / n)
    half = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / (1 + z**2 / n)
    return centre - half, centre + half

for correct, n in [(88, 100), (1120, 1273)]:
    lo, hi = wilson_interval(correct, n)
    print(f"{correct}/{n} correct = {correct / n:.1%}  (95% CI {lo:.1%} to {hi:.1%})")

# 88/100 correct = 88.0%  (95% CI 80.2% to 93.0%)
# 1120/1273 correct = 88.0%  (95% CI 86.1% to 89.7%)

Wilson 95% confidence intervals for the same 88% accuracy on 100 and 1,273 questions.

On 100 questions, an 88% score is compatible with anything from about 80% to 93%. Claims that one model 'beats' another by a few points on a small set are usually not supported.

Comparing two models on the same questions

When two models answer the same questions, only the questions where they disagree tell you which is better. McNemar's test uses exactly those.

python

import numpy as np
from scipy.stats import binomtest

rng = np.random.default_rng(9)
n = 1273
difficulty = rng.normal(size=n)                  # some questions are harder for every model
model_a = rng.random(n) < 1 / (1 + np.exp(-(2.0 - 1.5 * difficulty)))
model_b = rng.random(n) < 1 / (1 + np.exp(-(2.1 - 1.5 * difficulty)))
print(f"Model A {model_a.mean():.1%}, model B {model_b.mean():.1%}")

a_only = int(np.sum(model_a & ~model_b))
b_only = int(np.sum(~model_a & model_b))
p = binomtest(a_only, a_only + b_only, 0.5).pvalue
print(f"A right, B wrong: {a_only};  B right, A wrong: {b_only};  McNemar p = {p:.2f}")

# Model A 81.3%, model B 83.3%
# A right, B wrong: 128;  B right, A wrong: 154;  McNemar p = 0.14

Two simulated models on 1,273 questions; an exact McNemar test on the disagreements.

Model B was built to be slightly better, and scores 2 points higher, but on 1,273 questions the difference is not statistically clear (p = 0.14).
Results also depend on prompts, settings and model versions. A fair comparison fixes these and reports them.

Reading a benchmark claim

Which benchmark, which version, how many questions, and with what confidence interval?
Could the test questions have been in the training data?
Who or what graded open-ended answers? Model-graded scores need checking against clinician ratings.
Were prompts and settings the same for every model compared?
Does the task resemble the intended use, with the intended users?
Is there any evidence beyond benchmarks: user studies, workflow evaluations, outcomes?

Vendor neutrality

Models are named in this course for identification only. Rankings change with every release and are not endorsements. Choose tools through your organisation's evaluation and governance process.

Exercise

Analyse benchmark results like a reviewer: quantify uncertainty, compare models properly and look for warning signs.

Data: Simulated benchmark answers generated in the notebook (Generated in the notebook)

Reproduce the confidence intervals and find how many questions are needed for a ±2-point interval at 85% accuracy.
Reproduce the McNemar comparison, then increase the gap between the models until the difference becomes clear.
Simulate contamination: mark 20% of questions as 'seen in training' with 99% accuracy, and show how the overall score changes.
Simulate a model-graded benchmark in which the grader agrees with clinicians 85% of the time, and estimate how much this blurs a 3-point difference.
Write a five-sentence critical appraisal of a fictional press release claiming 'expert-level' performance.

The runnable notebook and worked solution will be published soon. You can work through the steps with the code on this page in the meantime.

Key takeaways

Exam benchmarks such as MedQA are near saturation as of 2026.
Newer benchmarks test conversations and real tasks, often with model-based grading.
High model scores did not translate into better decisions by users in a randomised study.
Report confidence intervals and compare models on the same questions.

Terms used: Benchmark contamination, Hallucination, Large language model (LLM), LLM-as-a-judge

FAQ

Do LLMs pass medical licensing exams?

Many recent models score well above typical pass marks on exam-style benchmarks such as MedQA. Exam questions are not the same as clinical practice, and passing does not show safe or useful performance with real patients.

What is HealthBench?

A benchmark of 5,000 multi-turn health conversations, released in 2025, graded against rubrics written by 262 physicians. A language model checks each rubric criterion.

What is benchmark contamination?

When test questions, or close versions of them, were included in a model's training data, so the score partly reflects memorisation rather than ability.

Are medically fine-tuned models better than general models?

Not necessarily. As of 2026, general-purpose models often match or exceed earlier medical models on benchmarks. What matters is performance on your task, evaluated properly.

What evidence should I look for before using an LLM tool clinically?

Evaluation on the intended task with intended users, comparison with current practice, error analysis, and ideally prospective studies, in addition to regulatory status and data protection.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.