Skip to content
Dr. Ram KumarDiabetes & metabolic health

Medical AI · Machine learning · Lesson 4 of 9

Gradient-boosted trees for readmission risk

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on · Last updated

  • Intermediate
  • Technical focus
  • About 35 minutes
  • Includes exercise

Before this lesson: A logistic-regression risk score, Building a cohort and avoiding data leakage

In short

Gradient boosting builds many small decision trees in sequence, each correcting the errors of the ones before, and often performs very well on tabular data. It can find non-linear relationships and interactions without being told about them. For hospital readmission, as for many clinical problems, the gain over a well-specified logistic regression is often small, so compare them fairly, tune with early stopping, and check calibration as well as AUROC.

You will learn to

  • Explain how gradient boosting builds a model from many small trees
  • Train a histogram gradient-boosting classifier with early stopping in scikit-learn
  • Compare boosting fairly with logistic regression on AUROC and Brier score
  • Interpret permutation importance and its limits

How gradient boosting works

A single decision tree splits patients into groups by asking yes/no questions ('more than two admissions last year?'). Alone, a tree is unstable and rough. Gradient boosting adds trees one at a time: each new small tree is fitted to the errors left by the trees so far, and its contribution is scaled down by a learning rate. After tens or hundreds of trees, the sum is a flexible, accurate model.

Because trees split on thresholds, boosting naturally captures U-shaped risks (both low and high sodium) and interactions (heart failure is worse for someone who lives alone) without anyone specifying them. Popular implementations include scikit-learn's HistGradientBoosting, XGBoost and LightGBM; they differ mainly in speed and features.

SettingWhat it controlsTypical starting point
Learning rateHow much each tree contributes0.05–0.1
Number of treesHow long boosting continuesSet high and use early stopping
Tree size (leaves or depth)How complex each tree can be15–31 leaves, or depth 3–6
Minimum samples per leafPrevents tiny, overfitted groups20 or more for clinical data

A synthetic readmission dataset

Readmission risk is a common target for hospital models, and a systematic review found most published readmission models had only modest discrimination. The simulation below builds 12,000 synthetic discharges with deliberately non-linear risk: a U-shaped sodium effect, age mattering only after 65, and an interaction between heart failure and living alone.

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(8)
n = 12000
df = pd.DataFrame({
    "age": rng.integers(20, 95, n),
    "length_of_stay": rng.gamma(2.0, 2.5, n).round(1),
    "prior_admissions": rng.poisson(0.8, n),
    "sodium": rng.normal(138, 4, n).round(),
    "heart_failure": rng.binomial(1, 0.15, n),
    "lives_alone": rng.binomial(1, 0.25, n),
})
logit = (-2.6 + 0.35 * np.minimum(df.prior_admissions, 4)
         + 0.04 * np.maximum(df.age - 65, 0)
         + 0.9 * (np.abs(df.sodium - 139) > 6)        # risk at both low and high sodium
         + 0.6 * df.heart_failure
         + 0.8 * df.heart_failure * df.lives_alone    # interaction
         + 0.05 * df.length_of_stay)
df["readmit_30d"] = rng.binomial(1, 1 / (1 + np.exp(-logit)))
print(f"{n} synthetic discharges, 30-day readmission {df.readmit_30d.mean():.1%}")

# 12000 synthetic discharges, 30-day readmission 18.8%
Synthetic discharges with non-linear effects and an interaction built in.

Boosting versus logistic regression

python
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss, roc_auc_score
from sklearn.model_selection import train_test_split

features = ["age", "length_of_stay", "prior_admissions", "sodium",
            "heart_failure", "lives_alone"]
train, test = train_test_split(df, test_size=0.3, random_state=0, stratify=df.readmit_30d)

models = {
    "Logistic regression": LogisticRegression(max_iter=2000),
    "Gradient boosting": HistGradientBoostingClassifier(
        learning_rate=0.05, max_iter=500, max_leaf_nodes=15,
        early_stopping=True, validation_fraction=0.2, random_state=0),
}
for name, model in models.items():
    model.fit(train[features], train.readmit_30d)
    p = model.predict_proba(test[features])[:, 1]
    print(f"{name:<20} AUROC {roc_auc_score(test.readmit_30d, p):.3f}   "
          f"Brier {brier_score_loss(test.readmit_30d, p):.4f}")
print(f"Boosting stopped after {models['Gradient boosting'].n_iter_} trees")

# Logistic regression  AUROC 0.646   Brier 0.1446
# Gradient boosting    AUROC 0.655   Brier 0.1438
# Boosting stopped after 58 trees
Continuing from the previous block: both models on the same split; boosting uses early stopping.
  • Boosting does better than a plain logistic regression, because it picks up the U-shaped sodium effect and the interaction without being told.
  • Early stopping held back 20% of the training data and stopped at 58 trees, when validation performance stopped improving. Without it, boosting easily overfits.
  • The gain is small. Readmission depends heavily on things rarely in the record, such as social support and access to follow-up, which limits every model.

A fair comparison includes a well-specified regression

Logistic regression only looks weak when it is given straight-line terms for relationships that are curved. Adding clinically sensible terms, such as abnormal sodium, age above 65 and the heart-failure-by-living-alone interaction, is a fairer comparison.

python
def add_terms(d):
    return d.assign(
        sodium_abnormal=(np.abs(d.sodium - 139) > 6).astype(int),
        age_over_65=np.maximum(d.age - 65, 0),
        hf_alone=d.heart_failure * d.lives_alone,
    )
engineered = features + ["sodium_abnormal", "age_over_65", "hf_alone"]
lr2 = LogisticRegression(max_iter=2000).fit(add_terms(train)[engineered], train.readmit_30d)
p = lr2.predict_proba(add_terms(test)[engineered])[:, 1]
print(f"Logistic + clinical terms AUROC {roc_auc_score(test.readmit_30d, p):.3f}   "
      f"Brier {brier_score_loss(test.readmit_30d, p):.4f}")

# Logistic + clinical terms AUROC 0.674   Brier 0.1408
Continuing from the previous block: logistic regression with clinical terms added.
  • With the right terms, logistic regression beats boosting here. This is partly an unfair advantage: we wrote the simulation, so we know the true shapes. In real data, splines and clinical knowledge get you some of the way, and boosting can reveal shapes you had not thought of.
  • A fair comparison gives both methods the same data, the same split and comparable tuning effort, and reports calibration as well as discrimination.

What the model relies on: permutation importance

Permutation importance shuffles one feature at a time in the test set and measures how much performance drops. A large drop means the model depends on that feature.

python
from sklearn.inspection import permutation_importance

gb = models["Gradient boosting"]
result = permutation_importance(gb, test[features], test.readmit_30d,
                                scoring="roc_auc", n_repeats=10, random_state=0)
importance = pd.Series(result.importances_mean, index=features)
print(importance.sort_values(ascending=False).round(3))

# age                 0.063
# sodium              0.023
# heart_failure       0.023
# prior_admissions    0.020
# length_of_stay      0.006
# lives_alone         0.002
Continuing from the 'Boosting versus logistic regression' block.
  • Living alone appears almost unimportant, yet in the simulation it more than doubles the effect of heart failure. It matters only for the 15% with heart failure, so shuffling it barely changes overall AUROC. Importance scores can hide interactions.
  • Importance describes the model, not causes. A feature can matter to the model because it is a proxy for something else, as the 'Prediction or causation?' lesson showed.

Practical checklist

  • Start with a logistic-regression baseline, including sensible non-linear terms or splines.
  • Tune boosting with early stopping on a validation set taken from the training data only, never the test set.
  • Compare AUROC, Brier score and calibration plots; boosting can be poorly calibrated and may need recalibration.
  • Prefer the simpler model when performance is similar: it is easier to validate, explain and maintain.
  • Check performance in subgroups (age, sex, ethnicity, site), not just overall.

Exercise

Compare gradient boosting with logistic regression for 30-day readmission, fairly and on several metrics.

Data: Synthetic discharges generated in the notebook (no real patient data) (Generated in the notebook)

  1. Reproduce the three models and their AUROC and Brier scores.
  2. Add natural cubic splines (sklearn SplineTransformer) for age and sodium to the plain logistic regression and compare.
  3. Tune the boosting model's learning rate and leaf count with cross-validation on the training set only.
  4. Draw calibration plots for all models; recalibrate the boosted model with isotonic regression if needed.
  5. Compute SHAP values for the boosted model and check whether they reveal the heart-failure-by-living-alone interaction that permutation importance missed.

The runnable notebook and worked solution will be published soon. You can work through the steps with the code on this page in the meantime.

Key takeaways

  • Gradient boosting adds small trees in sequence and captures non-linearity and interactions automatically.
  • Use early stopping on training-set validation data to avoid overfitting.
  • Compare against a well-specified logistic regression, not a straw man.
  • Importance scores describe the model and can hide interactions.

Terms used: AUROC (area under the ROC curve), Brier score, Gradient boosting, Logistic regression

Related: A logistic-regression risk score, Missing data and class imbalance, Building a cohort and avoiding data leakage

FAQ

Is gradient boosting better than logistic regression?

Sometimes, especially with many predictors, strong interactions or large datasets. For many clinical problems the difference is small once the regression includes sensible non-linear terms.

What is the difference between XGBoost, LightGBM and scikit-learn's HistGradientBoosting?

They are implementations of the same idea. They differ in speed, memory use, handling of categorical and missing values, and extra features. Results are usually similar with comparable tuning.

What is early stopping?

Stopping the addition of trees when performance on a held-out validation set stops improving. It prevents overfitting and removes the need to guess the number of trees.

Can gradient boosting handle missing values?

scikit-learn's HistGradientBoosting, XGBoost and LightGBM handle missing values natively by learning which branch to send them down. That can exploit informative missingness, which has its own risks.

Why are readmission models usually only moderately accurate?

Readmission depends on many factors that are rarely recorded, such as social support, access to follow-up care and adherence, as well as on care received after discharge.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.