Survival analysis: time-to-event models

Why time matters

A synthetic kidney-failure cohort

import numpy as np
import pandas as pd

rng = np.random.default_rng(21)
n = 3000
df = pd.DataFrame({
    "age": rng.integers(40, 80, n),
    "egfr": rng.normal(60, 18, n).clip(15, 120).round(),   # mL/min/1.73m²
    "uacr_high": rng.binomial(1, 0.35, n),                  # albuminuria
    "sglt2i": rng.binomial(1, 0.4, n),
})
hazard = 0.02 * np.exp(0.02 * (df.age - 60) - 0.04 * (df.egfr - 60)
                       + 0.8 * df.uacr_high - 0.3 * df.sglt2i)
event_time = rng.exponential(1 / hazard)
censor_time = rng.uniform(1, 8, n)           # end of follow-up varies by patient
df["years"] = np.minimum(event_time, censor_time).round(2)
df["kidney_failure"] = (event_time <= censor_time).astype(int)
print(f"{df.kidney_failure.sum()} events among {n} patients; "
      f"{(1 - df.kidney_failure.mean()):.0%} censored")

# 368 events among 3000 patients; 88% censored

Kaplan-Meier: risk over time

from lifelines import KaplanMeierFitter

naive = df[df.years <= 5].kidney_failure.sum() / n
print(f"Naive 'events / everyone' by 5 years: {naive:.1%}")

kmf = KaplanMeierFitter().fit(df.years, df.kidney_failure)
print(f"Kaplan-Meier risk by 5 years:        {1 - kmf.predict(5):.1%}")

# Naive 'events / everyone' by 5 years: 10.9%
# Kaplan-Meier risk by 5 years:        13.8%

• •The naive count treats everyone followed for less than five years as if they were event-free for the full five, so it underestimates risk (10.9% versus 13.8%).
• •kmf.plot_survival_function() draws the curve with a confidence band; plotting groups separately (for example by albuminuria) is a common first look at the data.

Cox proportional-hazards regression

from lifelines import CoxPHFitter

cph = CoxPHFitter().fit(df, duration_col="years", event_col="kidney_failure")
print(cph.summary[["exp(coef)", "exp(coef) lower 95%", "exp(coef) upper 95%"]].round(2))
print(f"\nConcordance index {cph.concordance_index_:.3f}")

#            exp(coef)  exp(coef) lower 95%  exp(coef) upper 95%
# covariate
# age             1.02                 1.01                 1.02
# egfr            0.96                 0.95                 0.96
# uacr_high       2.77                 2.25                 3.41
# sglt2i          0.72                 0.58                 0.89
#
# Concordance index 0.737

• •The hazard ratios are close to the values built into the simulation (age 1.02, eGFR 0.96 per mL/min/1.73m², SGLT2 inhibitor 0.74). For albuminuria the estimate (2.77) is somewhat above the true 2.23, whose value falls just below the 95% interval: with one sample, an interval misses the truth about 1 time in 20.
• •The concordance index (C-index) is the survival version of AUROC: the probability that, of two patients, the one who has the event first had the higher predicted risk.
• •In real observational data, the SGLT2-inhibitor hazard ratio would not be a reliable treatment effect: who receives a medicine is confounded, as the 'Prediction or causation?' lesson explains. Treatment effects come from trials.

From hazard ratios to individual risk

patients = pd.DataFrame({"age": [55, 70], "egfr": [80, 35],
                         "uacr_high": [0, 1], "sglt2i": [0, 0]})
risk_5y = 1 - cph.predict_survival_function(patients, times=[5]).iloc[0]
print(risk_5y.round(3).to_string())

# 0    0.034
# 1    0.559

• •A 55-year-old with good kidney function and no albuminuria has a predicted 3.4% five-year risk in this synthetic model; a 70-year-old with eGFR 35 and albuminuria has 55.9%.
• •Real, validated tools exist for this task, for example the Kidney Failure Risk Equation, which uses age, sex, eGFR and albuminuria. Teaching models like this one must not be used for patient care.

Competing risks and other cautions

• •Check the proportional-hazards assumption (lifelines: cph.check_assumptions(df)); effects that change over time need other approaches.
• •Censoring must be unrelated to the risk of the event. If sicker patients are lost to follow-up more often, estimates are biased.
• •Machine-learning survival methods exist, such as random survival forests and gradient-boosted survival models (for example in scikit-survival), and are evaluated with the C-index and time-dependent calibration.

Key takeaways

• •Censoring makes simple event counts underestimate risk; survival methods use each patient's follow-up.
• •Kaplan-Meier estimates risk over time; Cox models relate the hazard to characteristics.
• •Report absolute risk at a chosen time, not only hazard ratios.
• •Common competing events, such as death, need competing-risk methods.

Terms used: C-index (concordance index), Censoring, Competing risks, Hazard ratio

Related: A logistic-regression risk score, Prediction or causation?, Diabetic kidney disease

FAQ