Medical AI · Evaluation, safety & deployment · Lesson 1 of 5

External validation and dataset shift

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on 2026-10-18 · Last updated 2026-10-18

Intermediate
Technical focus
About 30 minutes
Includes exercise

Before this lesson: Calibration and decision-curve analysis, Building a cohort and avoiding data leakage

In short

External validation tests a model on data from a different hospital, population or period than the one it was built on. Dataset shift, such as different patients, a different lab assay or a different baseline risk, often leaves a model's ranking ability (AUROC) roughly intact while making its predicted risks wrong. Plan external validation with enough outcome events, report calibration as well as discrimination, and be ready to recalibrate or update the model for each new setting.

You will learn to

Distinguish internal, temporal and external validation
Name the main types of dataset shift with clinical examples
Simulate how case-mix, measurement and baseline-risk shifts affect AUROC and calibration
Plan an external validation with an adequate sample and the right metrics

Kinds of validation

The 2024 BMJ series by Collins, Riley and colleagues on evaluating clinical prediction models recommends external validation in each population where a model will be used, with calibration as a central measure.

Validation	Data used	What it shows
Internal	A held-out part of the development data, or bootstrap / cross-validation	Performance on similar patients; corrects for optimism
Temporal	Later patients from the same setting	Robustness to changes over time
External (geographic)	A different hospital, region or country	Transportability to new settings
Prospective / in use	Patients as the model is actually used	Real-world performance, including workflow effects

Types of dataset shift

Finlayson and colleagues described real examples in clinical AI, from software updates changing model inputs to new patient populations, and argued that clinicians are often the first to notice that a model has drifted.

Shift	What changes	Clinical example
Case mix (covariate shift)	The distribution of patient characteristics	An older or sicker population
Measurement shift	How a predictor is recorded	A new creatinine assay, a different scanner, new coding rules
Prior (prevalence) shift	The baseline rate of the outcome	A community clinic instead of a specialist centre
Concept shift	The relationship between predictors and outcome	A new treatment changes who goes on to have the outcome

Simulating four settings

The model below predicts a synthetic three-year kidney outcome from age, eGFR, HbA1c and albuminuria. It is developed at one hospital, then tested internally and at three other synthetic sites: an older population, a laboratory whose eGFR assay reads 15 units higher, and a population with lower baseline risk.

python

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

predictors = ["age", "egfr", "hba1c", "albuminuria"]

def site(n, seed, age_mean=58, assay_bias=0.0, baseline=-2.0):
    """Synthetic cohort: 3-year risk of a kidney outcome. assay_bias shifts measured eGFR only."""
    rng = np.random.default_rng(seed)
    d = pd.DataFrame({"age": rng.normal(age_mean, 10, n).round(),
                      "true_egfr": rng.normal(70, 20, n).clip(15, 120),
                      "hba1c": rng.normal(7.8, 1.3, n).round(1),
                      "albuminuria": rng.binomial(1, 0.3, n)})
    logit = (baseline + 0.02 * (d.age - 58) - 0.04 * (d.true_egfr - 70)
             + 0.3 * (d.hba1c - 7.8) + 1.0 * d.albuminuria)
    d["outcome"] = rng.binomial(1, 1 / (1 + np.exp(-logit)))
    d["egfr"] = (d.true_egfr + assay_bias).round()          # what the lab reports
    return d

dev = site(6000, seed=1)
model = LogisticRegression(max_iter=1000).fit(dev[predictors], dev.outcome)

def evaluate(name, d):
    p = model.predict_proba(d[predictors])[:, 1]
    recal = LogisticRegression(penalty=None).fit(np.log(p / (1 - p)).reshape(-1, 1), d.outcome)
    print(f"{name:<28} n={len(d):>5}  outcome {d.outcome.mean():5.1%}  "
          f"AUROC {roc_auc_score(d.outcome, p):.3f}  predicted {p.mean():5.1%}  "
          f"slope {recal.coef_[0][0]:.2f}")

evaluate("Internal (same hospital)", site(3000, seed=2))
evaluate("Older population", site(3000, seed=3, age_mean=70))
evaluate("Different eGFR assay (+15)", site(3000, seed=4, assay_bias=15))
evaluate("Lower baseline risk", site(3000, seed=5, baseline=-2.8))

# Internal (same hospital)     n= 3000  outcome 18.7%  AUROC 0.734  predicted 18.8%  slope 0.90
# Older population             n= 3000  outcome 23.0%  AUROC 0.757  predicted 23.5%  slope 1.02
# Different eGFR assay (+15)   n= 3000  outcome 18.4%  AUROC 0.759  predicted 12.0%  slope 1.03
# Lower baseline risk          n= 3000  outcome 11.3%  AUROC 0.759  predicted 19.4%  slope 0.99

Develop at one synthetic site; test under three kinds of shift.

Older population: risk rises, but age is in the model, so predictions rise with it and stay calibrated (23.5% predicted, 23.0% observed).
Different assay: patients are no healthier, but their eGFR reads 15 units higher, so the model underestimates risk by a third (12.0% predicted, 18.4% observed). Nothing about the patients changed; only the measurement did.
Lower baseline risk: the model overestimates risk (19.4% predicted, 11.3% observed).
AUROC stays between about 0.73 and 0.76 everywhere. A validation reporting only AUROC would have missed both failures.

Small validation samples are uncertain

External validation needs enough outcome events. Riley and colleagues recommend formal sample-size calculations for validation studies; a common rule of thumb is at least 100, and preferably 200 or more, events.

python

small = site(400, seed=6)
p = model.predict_proba(small[predictors])[:, 1]
rng = np.random.default_rng(0)
boot = []
for _ in range(1000):
    i = rng.integers(0, len(small), len(small))
    if small.outcome.iloc[i].nunique() == 2:
        boot.append(roc_auc_score(small.outcome.iloc[i], p[i]))
lo, hi = np.percentile(boot, [2.5, 97.5])
print(f"External sample of {len(small)} patients, {small.outcome.sum()} events: "
      f"AUROC {roc_auc_score(small.outcome, p):.3f} (95% CI {lo:.3f} to {hi:.3f})")

# External sample of 400 patients, 84 events: AUROC 0.750 (95% CI 0.690 to 0.807)

Continuing from the previous block: bootstrap confidence interval for AUROC in a 400-patient sample.

With 84 events, the AUROC could plausibly be anywhere from about 0.69 to 0.81. Calibration estimates are even less precise in small samples.

What to do when a model does not transport

Model updating should itself be validated and documented (Steyerberg; TRIPOD+AI).
Finding at the new site	Usual response
Good discrimination, calibration off on average	Recalibrate the intercept with local data
Predictions too extreme or too modest	Recalibrate intercept and slope
Specific predictors behave differently (for example a new assay)	Fix the measurement, or re-estimate those coefficients
Poor discrimination	Consider a different model, or developing one for the setting

Ask before adopting any model

Has it been externally validated in a population like ours, with calibration reported?
Are our measurements (assays, codes, equipment) the same as those used in development?
Do we have local data to check and, if needed, recalibrate it before use?

Exercise

Explore how different kinds of dataset shift affect a clinical prediction model, and fix what can be fixed.

Data: Synthetic sites generated in the notebook (no real patient data) (Generated in the notebook)

Reproduce the four-site comparison and draw calibration plots for each site.
Add a concept-shift site where albuminuria matters half as much, and check AUROC and calibration.
Recalibrate the model at the assay-shift and low-risk sites using 500 local patients, and evaluate on the rest.
Repeat the bootstrap for samples of 200, 400 and 2,000 patients and plot how the interval narrows.
Write the validation plan you would require before deploying this model at a new hospital.

The runnable notebook and worked solution will be published soon. You can work through the steps with the code on this page in the meantime.

Key takeaways

Validate externally in every population where a model will be used.
Shift often preserves AUROC while breaking calibration; report both.
Measurement changes, such as a new assay, can shift a model without any change in patients.
Use enough outcome events, and recalibrate or update models with local data.

Terms used: AUROC (area under the ROC curve), Calibration, Dataset shift, External validation

FAQ

What is external validation?

Testing a prediction model on data from a different setting, population or time period than the one used to develop it, to see whether it still performs well.

What is dataset shift?

A difference between the data a model was developed on and the data it is used on, such as different patients, measurements, outcome rates or relationships.

Why can AUROC stay the same when a model fails at a new site?

AUROC measures ranking. If all predictions shift up or down together, patients keep the same order, so AUROC is unchanged while the predicted risks become wrong.

How many patients are needed for external validation?

It depends on the outcome rate and the precision needed. Formal sample-size methods exist; at least 100, and preferably 200 or more, outcome events is a common minimum.

Can a model be fixed for a new hospital?

Often, by recalibrating with local data. If predictors behave very differently or discrimination is poor, more extensive updating or a new model may be needed.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.