A logistic-regression risk score

Why start with logistic regression

A synthetic cohort

import numpy as np
import pandas as pd

rng = np.random.default_rng(5)
n = 5000
df = pd.DataFrame({
    "age": rng.integers(30, 75, n),
    "bmi": rng.normal(27, 4.5, n).round(1),
    "fasting_glucose": rng.normal(100, 10, n).round(),   # mg/dL
    "family_history": rng.binomial(1, 0.3, n),
    "hypertension": rng.binomial(1, 0.35, n),
})
logit = (-17.5 + 0.03 * df.age + 0.10 * df.bmi + 0.11 * df.fasting_glucose
         + 0.6 * df.family_history + 0.4 * df.hypertension)
df["diabetes_5y"] = rng.binomial(1, 1 / (1 + np.exp(-logit)))
print(f"{n} synthetic adults, 5-year incidence {df.diabetes_5y.mean():.1%}")

# 5000 synthetic adults, 5-year incidence 19.4%

Fitting the model and reading odds ratios

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

predictors = ["age", "bmi", "fasting_glucose", "family_history", "hypertension"]
train, test = train_test_split(df, test_size=0.3, random_state=0, stratify=df.diabetes_5y)

model = LogisticRegression(penalty=None, max_iter=1000)
model.fit(train[predictors], train.diabetes_5y)
odds_ratios = pd.Series(np.exp(model.coef_[0]), index=predictors)
print(odds_ratios.round(2))

# age                1.03
# bmi                1.09
# fasting_glucose    1.12
# family_history     2.04
# hypertension       1.65

• •Odds ratios are per unit: each extra year of age multiplies the odds by 1.03, each extra mg/dL of fasting glucose by 1.12. Per unit, continuous predictors look small; per 10 mg/dL of glucose the odds ratio is about 1.12¹⁰ ≈ 3.1.
• •penalty=None gives the classical unpenalised model. scikit-learn applies L2 penalisation by default, which shrinks coefficients; that is often sensible for prediction but changes the odds ratios.
• •As the previous lesson explained, these odds ratios describe associations within this model, not the effect of changing a patient's BMI or glucose.

Checking discrimination and calibration

from sklearn.metrics import roc_auc_score

p = model.predict_proba(test[predictors])[:, 1]
print(f"Test AUROC {roc_auc_score(test.diabetes_5y, p):.3f}")

# Calibration slope and intercept: regress the outcome on the model's log-odds
log_odds = np.log(p / (1 - p)).reshape(-1, 1)
recal = LogisticRegression(penalty=None).fit(log_odds, test.diabetes_5y)
print(f"Calibration slope {recal.coef_[0][0]:.2f}, intercept {recal.intercept_[0]:.2f}")
print(f"Mean predicted risk {p.mean():.1%}, observed {test.diabetes_5y.mean():.1%}")

# Test AUROC 0.793
# Calibration slope 0.99, intercept 0.04
# Mean predicted risk 18.7%, observed 19.4%

• •A calibration slope near 1 means predictions are neither too extreme nor too modest; below 1 usually signals overfitting.
• •Here the model is well calibrated because the test data come from the same simulation. In a new population, calibration often shifts and may need updating.
• •The usual intercept-only calibration check fixes the slope at 1. This simple two-parameter version is enough for learning; the BMC Medicine paper in the sources gives the full approach.

From model to points score

coef = pd.Series(model.coef_[0], index=predictors)
steps = {"age": 10, "bmi": 5, "fasting_glucose": 10, "family_history": 1, "hypertension": 1}
unit = coef["age"] * 10
points = {k: round(coef[k] * s / unit) for k, s in steps.items()}
print(points)

score = ((test.age - 30) / 10 * points["age"]
         + (test.bmi - 25) / 5 * points["bmi"]
         + (test.fasting_glucose - 90) / 10 * points["fasting_glucose"]
         + test.family_history * points["family_history"]
         + test.hypertension * points["hypertension"])
print(f"Points score AUROC {roc_auc_score(test.diabetes_5y, score):.3f}")

# {'age': 1, 'bmi': 2, 'fasting_glucose': 4, 'family_history': 2, 'hypertension': 2}
# Points score AUROC 0.793

• •Rounding to integer points cost nothing measurable in AUROC here, because the rounded weights keep almost the same ranking of patients.
• •A points total ranks patients but is not a probability. To report risk, map each total back to the model's predicted probability or to observed risk in a validation set.
• •Real scores often also bin continuous predictors (for example BMI 25–29.9, 30–34.9), which loses more information; check AUROC and calibration after any simplification.

Planning before modelling

• •Choose candidate predictors from clinical knowledge and the literature, defined as they will be available at time zero, rather than by screening many variables for significance.
• •Check the sample size. Riley and colleagues' method calculates the number of participants and outcome events needed for a given number of predictor parameters and expected performance; the older rule of '10 events per variable' is often too low.
• •Keep continuous predictors continuous where possible, and consider splines for non-linear relationships, rather than splitting them into categories early.
• •Plan how missing data will be handled, and how the model will be validated internally (bootstrap or cross-validation) and externally.
• •Report the model following TRIPOD+AI, including the full equation so others can validate it.

Key takeaways

• •Logistic regression is the baseline for tabular clinical prediction; more complex models must beat it.
• •Odds ratios are per unit and describe associations within the model, not causal effects.
• •Always check calibration as well as AUROC, and expect calibration to shift in new populations.
• •Choose predictors and sample size before modelling, and report with TRIPOD+AI.

Terms used: AUROC (area under the ROC curve), Calibration, Cross-validation, Logistic regression, Odds ratio

Related: Building a cohort and avoiding data leakage, Diagnostic accuracy metrics, Type 2 diabetes

FAQ