Skip to content
Dr. Ram KumarDiabetes & metabolic health

Medical AI · Machine learning · Lesson 7 of 9

Explaining models with SHAP

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on · Last updated

  • Intermediate
  • Technical focus
  • About 30 minutes
  • Includes exercise

Before this lesson: Gradient-boosted trees for readmission risk

In short

SHAP (SHapley Additive exPlanations) breaks a model's prediction for one patient into contributions from each input, which add up to the prediction. Averaged across patients, it shows which inputs the model relies on most, and it can reveal interactions that simpler importance scores miss. SHAP describes how the model behaves, not why the patient is at risk, and its explanations become unreliable when inputs are strongly correlated.

You will learn to

  • Explain what a SHAP value represents for an individual prediction
  • Compute global and local SHAP explanations for a tree model in Python
  • Use SHAP to detect an interaction that permutation importance hides
  • Describe the limits of post-hoc explanations in clinical use

What a SHAP value is

SHAP borrows Shapley values from cooperative game theory. Think of the inputs as players who together produce the prediction. A feature's SHAP value is its average contribution across all the ways features could be added to the model. For each patient, the baseline (the model's average output) plus all the SHAP values equals that patient's prediction.

For tree models such as gradient boosting, TreeSHAP computes exact values quickly. For other models, SHAP uses approximations that can be slow and less reliable.

Global explanation: what the model relies on

This lesson reuses the synthetic readmission data from the gradient-boosting lesson, in which living alone matters only for patients with heart failure.

python
import shap
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split

features = ["age", "length_of_stay", "prior_admissions", "sodium",
            "heart_failure", "lives_alone"]
train, test = train_test_split(df, test_size=0.3, random_state=0, stratify=df.readmit_30d)
model = HistGradientBoostingClassifier(learning_rate=0.05, max_iter=500, max_leaf_nodes=15,
                                       early_stopping=True, random_state=0)
model.fit(train[features], train.readmit_30d)

explainer = shap.TreeExplainer(model)
sv = explainer(test[features])            # SHAP values on the log-odds scale
global_importance = pd.Series(np.abs(sv.values).mean(axis=0), index=features)
print(global_importance.sort_values(ascending=False).round(3))

# age                 0.270
# heart_failure       0.164
# prior_admissions    0.151
# length_of_stay      0.124
# sodium              0.110
# lives_alone         0.027
Run the data-generation block from the gradient-boosting lesson first. Install with: pip install shap
  • Mean absolute SHAP value is a common global importance measure. shap.plots.beeswarm(sv) shows the full distribution, including whether high values push risk up or down.
  • As with permutation importance, living alone ranks last overall, because it matters for only a minority of patients.

Local explanation: one patient

python
i = test.reset_index(drop=True).query("heart_failure == 1 and lives_alone == 1").index[0]
patient = sv[i]
print(test[features].iloc[i].to_string())
print(f"\nBaseline log-odds {patient.base_values:.2f}")
for name, value in sorted(zip(features, patient.values), key=lambda t: -abs(t[1])):
    print(f"{name:<18} {value:+.2f}")
print(f"Model log-odds    {patient.base_values + patient.values.sum():.2f}")

# age                  78.0
# length_of_stay        7.8
# prior_admissions      1.0
# sodium              145.0
# heart_failure         1.0
# lives_alone           1.0
#
# Baseline log-odds -1.55
# heart_failure      +0.65
# length_of_stay     +0.36
# lives_alone        +0.25
# age                +0.20
# sodium             +0.05
# prior_admissions   +0.00
# Model log-odds    -0.03
Continuing from the previous block: explain the prediction for one synthetic patient with heart failure who lives alone.
  • The contributions add up exactly: −1.55 plus the six SHAP values gives −0.03, a predicted risk of about 49% (versus an average of about 18%).
  • For this patient, living alone adds +0.25, a large amount for a feature that looked unimportant globally. shap.plots.waterfall(patient) draws this as a chart.

Finding an interaction

python
t = test.reset_index(drop=True)
alone = pd.Series(sv[:, "lives_alone"].values)
print("Mean SHAP value of lives_alone = 1:")
print(alone[t.lives_alone == 1].groupby(t.heart_failure[t.lives_alone == 1]).mean().round(2).to_string())

# Mean SHAP value of lives_alone = 1:
# heart_failure
# 0    0.03
# 1    0.19
Continuing from the previous block: the contribution of living alone, split by heart-failure status.
  • Living alone adds about six times as much to predicted risk for patients with heart failure as for those without. This is the interaction built into the simulation, which permutation importance hid.
  • shap.plots.scatter(sv[:, 'lives_alone'], color=sv[:, 'heart_failure']) shows the same pattern visually.

When features are correlated

SHAP shares credit between inputs that carry the same information. If age is recorded twice, from two systems with slightly different values, the model can use either copy.

python
rng2 = np.random.default_rng(1)
train2 = train.assign(age_registry=train.age + rng2.normal(0, 1, len(train)))
test2 = test.assign(age_registry=test.age + rng2.normal(0, 1, len(test)))
features2 = features + ["age_registry"]
model2 = HistGradientBoostingClassifier(learning_rate=0.05, max_iter=500, max_leaf_nodes=15,
                                        early_stopping=True, random_state=0)
model2.fit(train2[features2], train2.readmit_30d)
sv2 = shap.TreeExplainer(model2)(test2[features2])
imp2 = pd.Series(np.abs(sv2.values).mean(axis=0), index=features2)
print(imp2[["age", "age_registry"]].round(3).to_string())

# age             0.105
# age_registry    0.193
Continuing from the 'Global explanation' block: add a near-duplicate of age and re-explain.
  • Age alone had an importance of 0.270. With a near-copy, the credit is split between the two, and neither looks as important as age really is.
  • In clinical data, correlated inputs are everywhere: creatinine and eGFR, weight and BMI, several blood-pressure readings. Group or remove redundant inputs before reading importances.

What explanations can and cannot do

Explanations are useful for developers and auditors: to spot leakage (a suspicious feature at the top of the list), check that the model uses sensible information, and investigate errors. They are less useful as reassurance at the bedside. Ghassemi and colleagues argued that current explanation methods cannot guarantee that an individual decision is correct, and that rigorous validation matters more than explanations for building trust.

Good usesMisuses
Finding leakage or proxies during developmentTreating SHAP values as causes of the patient's risk
Checking the model behaves plausibly across subgroupsUsing an explanation to justify trusting a single prediction
Discovering interactions worth investigatingRecommending a patient change a feature because it has a large SHAP value
Auditing for bias against protected groupsReplacing external validation with explanations

Interpretable by design

  • Rudin argues that for high-stakes decisions, inherently interpretable models such as sparse logistic regression or scoring systems should be preferred over explaining a black box, when they perform similarly.

Exercise

Use SHAP to explain a readmission model, find its interaction, and test how explanations change with correlated inputs.

Data: Synthetic discharges generated in the notebook (no real patient data) (Generated in the notebook)

  1. Reproduce the global ranking and draw a beeswarm plot.
  2. Draw waterfall plots for three patients with very different predicted risks.
  3. Use SHAP interaction values (explainer.shap_interaction_values) to confirm the heart-failure-by-living-alone interaction.
  4. Add a leaky feature (for example a 'follow-up booked after readmission' flag) and show how SHAP exposes it.
  5. Compare SHAP importances with the logistic-regression coefficients from the gradient-boosting lesson and explain the differences.

The runnable notebook and worked solution will be published soon. You can work through the steps with the code on this page in the meantime.

Key takeaways

  • SHAP values add up to each prediction and show what the model relied on.
  • Local SHAP can reveal effects and interactions that global rankings hide.
  • Correlated inputs split credit; group them before interpreting.
  • Explanations describe the model, not causes, and do not replace validation.

Terms used: Data leakage, Gradient boosting, SHAP

Related: Gradient-boosted trees for readmission risk, Prediction or causation?, Bias and fairness (Evaluation track)

FAQ

What is a SHAP value?

The contribution of one input to one prediction, relative to the model's average output. All of a patient's SHAP values plus the baseline add up to the model's prediction for that patient.

Does a large SHAP value mean the feature causes the outcome?

No. It means the model used that feature heavily for that prediction. The feature may be a proxy or a consequence of the outcome rather than a cause.

Is SHAP better than permutation importance?

They answer slightly different questions. Permutation importance measures how much performance depends on a feature; SHAP explains individual predictions and can reveal interactions. Both struggle with correlated features.

Should clinicians see SHAP explanations for each prediction?

Possibly, but with caution. Explanations can increase trust in wrong predictions. Their value at the bedside should be tested, not assumed.

Does SHAP work for deep-learning models?

Yes, through approximate methods such as DeepSHAP or KernelSHAP, but these are slower and less reliable than TreeSHAP for tree models.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.