Skip to content
Dr. Ram KumarDiabetes & metabolic health

Medical AI · Machine learning · Lesson 8 of 9

Calibration and decision-curve analysis

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on · Last updated

  • Intermediate
  • Clinical focus
  • About 30 minutes
  • Includes exercise

Before this lesson: A logistic-regression risk score, Diagnostic accuracy metrics: sensitivity, specificity, PPV, ROC and calibration

In short

Calibration asks whether predicted risks match what actually happens; decision-curve analysis asks whether using the model would lead to better decisions than treating everyone or no one. When a model moves to a new hospital, it often keeps its ranking ability (AUROC) but its risks become too high or too low. Recalibration with a small local sample can fix this, and net benefit shows whether the model is worth using at the thresholds that matter clinically.

You will learn to

  • Assess calibration with grouped observed-versus-predicted risk, calibration slope and intercept
  • Explain why calibration often fails when a model is moved to a new population
  • Recalibrate a model using a sample from the new setting
  • Calculate and interpret net benefit across clinically relevant thresholds

Why calibration fails at new sites

Van Calster and colleagues call calibration the Achilles heel of predictive analytics. A model developed where an outcome is common will tend to overestimate risk where it is rarer, even if the relationships between predictors and outcome are the same. Differences in measurement, coding, case mix and care quality add further drift.

The example reuses the synthetic diabetes-risk model, developed at one site and applied to a second site whose baseline risk is lower.

python
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

predictors = ["age", "bmi", "fasting_glucose", "family_history", "hypertension"]

def simulate(n, baseline, seed):
    rng = np.random.default_rng(seed)
    d = pd.DataFrame({
        "age": rng.integers(30, 75, n),
        "bmi": rng.normal(27, 4.5, n).round(1),
        "fasting_glucose": rng.normal(100, 10, n).round(),
        "family_history": rng.binomial(1, 0.3, n),
        "hypertension": rng.binomial(1, 0.35, n),
    })
    logit = (baseline + 0.03 * d.age + 0.10 * d.bmi + 0.11 * d.fasting_glucose
             + 0.6 * d.family_history + 0.4 * d.hypertension)
    d["diabetes_5y"] = rng.binomial(1, 1 / (1 + np.exp(-logit)))
    return d

development = simulate(5000, baseline=-17.5, seed=5)
new_site = simulate(5000, baseline=-18.5, seed=6)        # lower baseline risk
model = LogisticRegression(penalty=None, max_iter=1000)
model.fit(development[predictors], development.diabetes_5y)

p_new = model.predict_proba(new_site[predictors])[:, 1]
print(f"New site AUROC {roc_auc_score(new_site.diabetes_5y, p_new):.3f}")
print(f"Mean predicted {p_new.mean():.1%}, observed {new_site.diabetes_5y.mean():.1%}")

groups = pd.qcut(p_new, 5, labels=False)
table = pd.DataFrame({"predicted": p_new, "observed": new_site.diabetes_5y}).groupby(groups).mean()
print((table * 100).round(1).rename_axis("risk fifth"))

# New site AUROC 0.806
# Mean predicted 18.9%, observed 9.4%
#             predicted  observed
# risk fifth
# 0                 2.6       1.0
# 1                 6.8       2.8
# 2                12.8       5.6
# 3                23.0       9.9
# 4                49.3      27.8
Develop at one synthetic site; apply to another with lower baseline risk.
  • Discrimination is preserved (AUROC 0.806), so the model still ranks patients well.
  • Every risk group is overestimated, roughly two-fold. A patient told their risk is 49% faces about 28% in reality. Decisions based on thresholds would flag far too many people.

Recalibrating with local data

Instead of building a new model, the existing one can be updated. Logistic recalibration refits only two numbers, an intercept and a slope applied to the model's log-odds, using a modest sample from the new site. The ranking of patients does not change.

python
recal_part, eval_part = new_site.iloc[:2000], new_site.iloc[2000:]

def log_odds(d):
    p = model.predict_proba(d[predictors])[:, 1]
    return np.log(p / (1 - p)).reshape(-1, 1)

recalibrator = LogisticRegression(penalty=None).fit(log_odds(recal_part), recal_part.diabetes_5y)
p_before = model.predict_proba(eval_part[predictors])[:, 1]
p_after = recalibrator.predict_proba(log_odds(eval_part))[:, 1]
print(f"Recalibration: slope {recalibrator.coef_[0][0]:.2f}, "
      f"intercept {recalibrator.intercept_[0]:.2f}")
print(f"Observed {eval_part.diabetes_5y.mean():.1%}; mean predicted before "
      f"{p_before.mean():.1%}, after {p_after.mean():.1%}")
print(f"AUROC before {roc_auc_score(eval_part.diabetes_5y, p_before):.3f}, "
      f"after {roc_auc_score(eval_part.diabetes_5y, p_after):.3f}")

# Recalibration: slope 0.93, intercept -1.01
# Observed 9.1%; mean predicted before 18.7%, after 9.6%
# AUROC before 0.812, after 0.812
Continuing from the previous block: recalibrate on 2,000 patients from the new site, evaluate on the other 3,000.
  • AUROC here (0.812) differs slightly from the full-site figure (0.806) because it is measured on the 3,000-patient evaluation half only.
  • The intercept of about −1 recovers the one-unit drop in baseline risk built into the simulation, and predicted risk now matches observed risk.
  • AUROC is identical before and after: recalibration changes the risks, not the ranking.
  • Isotonic regression and Platt scaling are alternatives, mostly used for machine-learning models whose probabilities are distorted in more complex ways.

Net benefit: is the model worth using?

Choosing a risk threshold, say treating or investigating anyone above 10%, implies a judgement about harms: at 10%, a clinician accepts nine unnecessary interventions to catch one case. Net benefit puts true and false positives on the same scale using that threshold: net benefit = (true positives − false positives × threshold / (1 − threshold)) / n.

A decision curve plots net benefit across a range of reasonable thresholds and compares the model with two default strategies: intervene for everyone, and intervene for no one (net benefit zero).

python
y = eval_part.diabetes_5y.to_numpy()

def net_benefit(p, threshold):
    flagged = p >= threshold
    tp = np.sum(flagged & (y == 1)) / len(y)
    fp = np.sum(flagged & (y == 0)) / len(y)
    return tp - fp * threshold / (1 - threshold)

prevalence = y.mean()
print("threshold  treat-all  model-before  model-after")
for t in [0.05, 0.10, 0.20, 0.30]:
    treat_all = prevalence - (1 - prevalence) * t / (1 - t)
    print(f"{t:>8.0%}  {treat_all:>9.3f}  {net_benefit(p_before, t):>12.3f}  "
          f"{net_benefit(p_after, t):>11.3f}")

# threshold  treat-all  model-before  model-after
#       5%      0.043         0.053        0.058
#      10%     -0.010         0.028        0.040
#      20%     -0.136         0.002        0.020
#      30%     -0.299        -0.009        0.012
Continuing from the previous block: net benefit of the model before and after recalibration.
  • After recalibration, the model beats both default strategies at every threshold shown.
  • Before recalibration, the overestimated risks flag too many people: at a 30% threshold the model does worse than intervening for no one.
  • A net benefit of 0.040 at 10% means the same as finding 4 extra true cases per 100 patients with no unnecessary interventions.

Reading a decision curve

  • Focus on the range of thresholds that clinicians would plausibly use for this decision; performance at other thresholds is irrelevant.
  • A model is useful at a threshold only if its net benefit exceeds both 'treat all' and 'treat none'.
  • Small differences in net benefit can matter at population scale; uncertainty intervals help judge whether they are real.
  • Decision curves need well-calibrated predictions: miscalibration translates directly into lower net benefit.

What to report for any clinical prediction model

  • Discrimination (AUROC or C-index, with confidence interval).
  • Calibration (plot, slope and intercept) in each validation population.
  • Clinical utility (net benefit over a justified threshold range).
  • Any updating or recalibration done for the new setting, as TRIPOD+AI recommends.

Exercise

Detect miscalibration at a new site, recalibrate, and judge clinical usefulness with a decision curve.

Data: Synthetic sites generated in the notebook (no real patient data) (Generated in the notebook)

  1. Reproduce the new-site calibration table and draw a smoothed calibration curve.
  2. Recalibrate with only the intercept (slope fixed at 1) and with intercept and slope; compare.
  3. Vary the size of the recalibration sample (200, 500, 2,000) and see how stable the result is.
  4. Plot full decision curves from 1% to 40%, with 'treat all' and 'treat none', before and after recalibration.
  5. Simulate a third site where one predictor's effect is weaker, and check whether recalibration alone is enough.

The runnable notebook and worked solution will be published soon. You can work through the steps with the code on this page in the meantime.

Key takeaways

  • A model can keep its AUROC at a new site while its risks become badly wrong.
  • Recalibration with a modest local sample fixes the risks without changing the ranking.
  • Net benefit weighs true and false positives using the clinical threshold.
  • Report discrimination, calibration and net benefit for every validation.

Terms used: AUROC (area under the ROC curve), Calibration, Dataset shift, Decision-curve analysis, External validation

Related: Diagnostic accuracy metrics, A logistic-regression risk score, External validation and dataset shift (Evaluation track)

FAQ

What is the difference between discrimination and calibration?

Discrimination is how well a model separates patients with and without the outcome (AUROC). Calibration is whether its predicted risks match observed risks. A model needs both.

What is a calibration slope?

The slope obtained by regressing the outcome on the model's log-odds. A slope of 1 is ideal; below 1 means predictions are too extreme, typically from overfitting.

When should I recalibrate a model?

Whenever you apply it to a population that differs from the development population and you have local outcome data. Check calibration first; update the intercept, or the intercept and slope, if needed.

What is net benefit?

A measure of clinical usefulness: true positives minus false positives weighted by the odds of the risk threshold, per patient. It allows a model to be compared with treating everyone or no one.

How do I choose a threshold for a decision curve?

From the clinical decision itself: how many unnecessary interventions are acceptable to catch one case. Plot a range of plausible thresholds rather than a single value.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.