Skip to content
Dr. Ram KumarDiabetes & metabolic health

Medical AI · Foundations · Lesson 6 of 6

Diagnostic accuracy metrics: sensitivity, specificity, PPV, ROC and calibration

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on · Last updated

  • Beginner
  • Clinical focus
  • About 25 minutes
  • Includes exercise

In short

Sensitivity and specificity describe how a test or model behaves in people with and without a condition. Predictive values tell you what a result means for the person in front of you, and they change with how common the condition is. AUROC measures how well a model ranks patients, and calibration checks whether its predicted probabilities can be taken at face value. A clinical AI model needs all of these, not one headline number.

You will learn to

  • Build a 2×2 confusion matrix and calculate sensitivity, specificity, PPV and NPV from it
  • Explain why the same model has a different PPV in a screening clinic and a specialist clinic
  • Read an ROC curve and say what AUROC does and does not tell you
  • Check whether a model's predicted probabilities are calibrated

The 2×2 table behind every metric

Every classification metric starts from the same table. Each patient is placed in one of four cells by two facts: whether they really have the condition (by the reference standard), and whether the test or model says they do.

An AI model usually outputs a probability, not a yes or no. It only fills this table once you choose a threshold, such as 'flag anyone above 0.5'. Change the threshold and every number below changes with it.

The confusion matrix. Rows are what the test says; columns are the truth.
Condition presentCondition absent
Test or model positiveTrue positive (TP)False positive (FP)
Test or model negativeFalse negative (FN)True negative (TN)

Sensitivity, specificity and the predictive values

Sensitivity and specificity are read down the columns: they start from people whose true status is known. Predictive values are read across the rows: they start from the test result, which is the position a clinician is in.

Machine-learning libraries call sensitivity 'recall' and PPV 'precision'. They are the same quantities.
MetricFormulaThe question it answers
Sensitivity (recall)TP / (TP + FN)Of people with the condition, how many does the test catch?
SpecificityTN / (TN + FP)Of people without it, how many does the test correctly clear?
Positive predictive value (PPV, precision)TP / (TP + FP)If the test is positive, how likely is the condition?
Negative predictive value (NPV)TN / (TN + FN)If the test is negative, how likely is it truly absent?

Why accuracy alone misleads

  • If 2% of patients have a condition, a model that calls everyone negative is 98% accurate and clinically useless: its sensitivity is zero.
  • Always report sensitivity and specificity (or PPV and NPV) alongside accuracy, and say what the prevalence was.

Why PPV depends on prevalence

Take a test with 90% sensitivity and 80% specificity, applied to 1,000 people. If 10% have the condition, 90 of the 100 are caught and 180 of the 900 healthy people are wrongly flagged, so only 90 of 270 positives are real: a PPV of 33%. In a population where 1% have the condition, the same test gives a PPV of about 4%.

The test has not changed; the population has. This is why a model validated in a hospital, where the condition is common, can produce mostly false alarms when moved to community screening.

python
def ppv(sensitivity, specificity, prevalence):
    true_pos = sensitivity * prevalence
    false_pos = (1 - specificity) * (1 - prevalence)
    return true_pos / (true_pos + false_pos)

for prevalence in [0.01, 0.05, 0.10, 0.30]:
    print(f"prevalence {prevalence:>4.0%}  PPV {ppv(0.90, 0.80, prevalence):.1%}")

# prevalence   1%  PPV 4.3%
# prevalence   5%  PPV 19.1%
# prevalence  10%  PPV 33.3%
# prevalence  30%  PPV 65.9%
PPV for a test with 90% sensitivity and 80% specificity at different prevalences.
Same test (sensitivity 90%, specificity 80%), three populations.
PrevalencePPVNPV
1% (community screening)4.3%99.9%
10%33.3%98.6%
30% (specialist clinic)65.9%94.9%

ROC curves and AUROC

A receiver operating characteristic (ROC) curve plots sensitivity against 1 − specificity at every possible threshold. The area under it (AUROC, or the C-statistic) is the probability that the model gives a randomly chosen patient with the condition a higher score than a randomly chosen patient without it. 0.5 is a coin toss; 1.0 is perfect ranking.

AUROC is threshold-free, which makes it useful for comparing models, but it is also its weakness: it says nothing about how the model performs at the one threshold you will actually use, and nothing about whether its probabilities are right. For rare outcomes, a high AUROC can sit alongside a low PPV, so the precision–recall curve is often reported as well.

python
import numpy as np
from sklearn.metrics import confusion_matrix, roc_auc_score

rng = np.random.default_rng(42)
n = 2000
y_true = rng.binomial(1, 0.10, size=n)  # 10% prevalence

# Patients with the condition get higher scores on average
logit = rng.normal(loc=np.where(y_true == 1, 1.5, -1.0), scale=1.0)
y_prob = 1 / (1 + np.exp(-logit))

y_pred = (y_prob >= 0.5).astype(int)
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()

print(f"Sensitivity {tp / (tp + fn):.2f}   Specificity {tn / (tn + fp):.2f}")
print(f"PPV         {tp / (tp + fp):.2f}   NPV         {tn / (tn + fn):.2f}")
print(f"AUROC       {roc_auc_score(y_true, y_prob):.3f}")

# Sensitivity 0.92   Specificity 0.85
# PPV         0.41   NPV         0.99
# AUROC       0.958
Simulate a model's scores for 2,000 patients at 10% prevalence, then compute the metrics at a 0.5 threshold.

Calibration: can the probabilities be trusted?

A model is well calibrated if, among patients it gives a 20% risk, about 20% go on to have the outcome. Discrimination (AUROC) and calibration are separate properties: a model can rank patients perfectly and still overstate everyone's risk two-fold.

Calibration matters whenever a probability is shown to a clinician or patient, or used to decide treatment. It is also the property most likely to break when a model is moved to a new hospital or population, so it should be re-checked at every external validation.

python
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss

observed, predicted = calibration_curve(y_true, y_prob, n_bins=10, strategy="quantile")
for p, o in zip(predicted, observed):
    print(f"predicted {p:.2f}   observed {o:.2f}")

print(f"Brier score {brier_score_loss(y_true, y_prob):.3f}")  # lower is better

# predicted 0.07   observed 0.00
# predicted 0.13   observed 0.00
# ...
# predicted 0.49   observed 0.04
# predicted 0.62   observed 0.20
# predicted 0.85   observed 0.76
# Brier score 0.120
Continuing from the previous block: compare predicted and observed risk in ten groups, and compute the Brier score.
  • The simulated model ranks patients very well (AUROC 0.958), yet in most groups its predicted risk is far higher than the observed risk: patients it puts at 40–50% risk almost never have the condition.
  • Used as it is, this model would frighten patients and could drive unnecessary tests. Good discrimination does not guarantee trustworthy probabilities.

What to look for in a published model

  • Discrimination: AUROC with a confidence interval.
  • Calibration: a calibration plot, calibration slope and intercept, not just a single test statistic.
  • Clinical usefulness: sensitivity, specificity and PPV at the threshold that will actually be used, ideally with a decision curve.
  • The prevalence in the validation population, and whether it matches yours.

A clinical example: point-of-care nerve testing

The same reasoning applies to any diagnostic tool, AI or not. For example, the manufacturer of one point-of-care nerve conduction device reports a pooled sensitivity of about 88% and specificity of about 82% for detecting peripheral neuropathy. What a positive result means for an individual still depends on how likely neuropathy was before the test, which is why results are always interpreted alongside symptoms and examination.

Exercise

Explore how threshold and prevalence change the metrics of a simulated risk model, and check its calibration.

Data: Synthetic patients generated in the notebook (no real patient data) (Generated in the notebook)

  1. Run the simulation above and record sensitivity, specificity, PPV and NPV at thresholds 0.3, 0.5 and 0.7.
  2. Re-run the simulation at 2% and 30% prevalence without changing the model. Which metrics move, and which stay roughly the same?
  3. Plot the ROC curve with sklearn.metrics.RocCurveDisplay and mark your three thresholds on it.
  4. Draw a calibration plot. Is the model over- or under-estimating risk? Recalibrate it with logistic (Platt) scaling and compare Brier scores.
  5. Write two sentences you would put in a report for clinicians describing this model's performance.

The runnable notebook and worked solution will be published soon. You can work through the steps with the code on this page in the meantime.

Key takeaways

  • Every metric depends on a chosen threshold; report which one you used.
  • Sensitivity and specificity travel with the test; PPV and NPV depend on prevalence.
  • AUROC measures ranking, not whether probabilities are correct.
  • Calibration must be checked separately, and re-checked in every new population.

Terms used: AUROC (area under the ROC curve), Brier score, Calibration, Confusion matrix, Positive predictive value (PPV), Sensitivity (recall), Specificity

Related: Machine learning track, Medical AI glossary, DPN Check nerve conduction testing

FAQ

Is precision the same as positive predictive value?

Yes. Machine-learning libraries use 'precision' and 'recall'; clinical papers use 'positive predictive value' and 'sensitivity'. The formulas are identical.

What is a good AUROC for a clinical model?

There is no universal cut-off. An AUROC of 0.8 may be useful for one decision and inadequate for another. What matters is performance at the chosen threshold, calibration, and whether using the model improves decisions compared with current practice.

Why did my model's PPV drop when we used it in a different clinic?

Most likely because the condition is less common there. PPV falls as prevalence falls, even if sensitivity and specificity are unchanged. Calibration may also have shifted, so check both.

Should I use accuracy to report a clinical model?

Not on its own. With rare outcomes, accuracy can be high for a model that misses every case. Report sensitivity, specificity, predictive values, AUROC and calibration together.

What is the Brier score?

The mean squared difference between predicted probabilities and actual outcomes (0 or 1). Lower is better. It reflects both discrimination and calibration, so it is a useful single summary alongside AUROC and a calibration plot.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.