Skip to content
Dr. Ram KumarDiabetes & metabolic health

Medical AI · Applied case studies: diabetes & metabolic health · Lesson 2 of 5

Diabetic retinopathy screening

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on · Last updated

  • Intermediate
  • Clinical focus
  • About 30 minutes
  • Includes exercise

Before this lesson: CNNs for chest X-ray and retinal images, Diagnostic accuracy metrics: sensitivity, specificity, PPV, ROC and calibration

In short

Diabetic retinopathy screening is one of the most established uses of medical AI: systems grade retinal photographs and refer people with referable disease to eye specialists. Deploying one is about more than accuracy: the operating threshold sets how many people are referred and how many cases are missed, ungradable images need another route, and a change of camera can shift performance. Real-world studies have found wide differences between commercial systems, so each programme needs local validation and monitoring.

You will learn to

  • Describe how AI is used in diabetic retinopathy screening programmes
  • Choose an operating threshold for a target sensitivity and see its consequences
  • Translate accuracy into referrals, missed cases and ungradable images per 1,000 screened
  • Explain why camera changes and local validation matter

AI in retinopathy screening

Deep learning detected referable diabetic retinopathy in retinal photographs with high accuracy in development studies (Gulshan et al., 2016), and in 2018 an autonomous system became the first AI diagnostic authorised by the US FDA, after a pivotal trial in primary care reporting 87.2% sensitivity and 90.7% specificity (Abràmoff et al., 2018). Systems using smartphone-based fundus cameras have also been studied in India, with high sensitivity for sight-threatening retinopathy (Rajalakshmi et al., 2018).

Real-world results vary. A head-to-head study of seven commercial systems on the same images found sensitivities ranging from about 51% to 86% (Lee et al., 2021), and field studies show that image quality, connectivity and clinic workflow strongly affect the benefit (Beede et al., 2020).

Simulating a screening programme

The simulation gives each person a true retinopathy grade, an AI score that rises with severity, and a 5% chance of an ungradable image. The threshold is chosen on a development set to give 90% sensitivity for referable disease (moderate or worse), then checked on new people.

python
import numpy as np

rng = np.random.default_rng(3)
GRADES = ["none", "mild", "moderate", "severe", "proliferative"]
PREVALENCE = [0.70, 0.18, 0.08, 0.025, 0.015]     # an illustrative screening population
REFERABLE = {"moderate", "severe", "proliferative"}
SCORE_MEAN = {"none": -1.5, "mild": -0.6, "moderate": 1.0, "severe": 1.8, "proliferative": 2.2}

def screening_population(n, camera_shift=0.0):
    grades = rng.choice(GRADES, n, p=PREVALENCE)
    scores = np.array([rng.normal(SCORE_MEAN[g], 1.0) for g in grades]) + camera_shift
    ungradable = rng.random(n) < 0.05               # poor images: cataract, small pupils, blur
    return grades, scores, ungradable

def summarise(grades, scores, ungradable, threshold, label):
    gradable = ~ungradable
    referable = np.isin(grades, list(REFERABLE))
    flagged = scores >= threshold
    sens = np.mean(flagged[gradable & referable])
    spec = np.mean(~flagged[gradable & ~referable])
    ppv = np.mean(referable[gradable & flagged])
    per_1000 = 1000 / len(grades)
    print(f"{label}: sensitivity {sens:.1%}, specificity {spec:.1%}, PPV {ppv:.1%}")
    print(f"  per 1,000 screened: {np.sum(gradable & flagged) * per_1000:.0f} referred, "
          f"{np.sum(gradable & referable & ~flagged) * per_1000:.1f} referable cases missed, "
          f"{np.sum(ungradable) * per_1000:.0f} ungradable (need another route)")

# Choose the threshold on a development set so that sensitivity is at least 90%
dev_g, dev_s, dev_u = screening_population(20000)
ref_scores = dev_s[~dev_u & np.isin(dev_g, list(REFERABLE))]
threshold = np.quantile(ref_scores, 0.10)
print(f"Operating threshold chosen for 90% sensitivity: {threshold:.2f}\n")
summarise(*screening_population(20000), threshold, "Validation, same camera")

# Operating threshold chosen for 90% sensitivity: -0.04
#
# Validation, same camera: sensitivity 88.3%, specificity 88.8%, PPV 51.7%
#   per 1,000 screened: 194 referred, 13.2 referable cases missed, 49 ungradable (need another route)
A synthetic screening population, threshold selection and programme-level results.
  • On new people, sensitivity came out at 88.3%, below the 90% target, because a threshold chosen on one sample does not transfer exactly. Programmes set thresholds with a margin and confirm them on validation data.
  • Per 1,000 people screened: 194 referrals, about 13 people with referable disease missed, and 49 needing another route because their images could not be graded.
  • PPV is only about 52%: half of referrals turn out not to need specialist care, a workload the eye service must be able to absorb.

A new camera

Now the programme changes camera, and the images it produces lead to scores that are, on average, half a unit lower.

python
summarise(*screening_population(20000, camera_shift=-0.5), threshold, "New camera, scores 0.5 lower")

# New camera, scores 0.5 lower: sensitivity 78.9%, specificity 95.3%, PPV 70.1%
#   per 1,000 screened: 131 referred, 24.5 referable cases missed, 50 ungradable (need another route)
Continuing from the previous block: the same threshold with a camera that lowers scores.
  • Fewer referrals and a higher PPV look like an improvement, yet missed referable cases nearly double, from about 13 to 25 per 1,000. Only monitoring sensitivity, or regular audits against human graders, would reveal it.
  • Ungradable rates, referral rates and the share of AI 'negatives' confirmed by graders should all be tracked over time, especially after equipment or software changes.

Teaching models, not clinical tools

  • Data in these case studies are simulated, and the models are fitted to them for learning. They must not be used to make decisions about real patients.
  • Validated tools exist for several of these problems; any tool used in care must be validated for its population and meet regulatory requirements.

Exercise

Plan the operating point and monitoring for an AI retinopathy screening programme.

Data: Synthetic screening population in the notebook; optional open retinal image datasets (for example MedMNIST RetinaMNIST) (Generated in the notebook; MedMNIST under CC BY 4.0)

  1. Plot sensitivity, specificity and referrals per 1,000 across thresholds, and mark the 90% sensitivity point.
  2. Bootstrap the development set to put a confidence interval on the chosen threshold's sensitivity, and choose a safer threshold.
  3. Model a referral capacity limit (for example 120 per 1,000) and find the best sensitivity achievable within it.
  4. Design a monthly monitoring report that would have detected the camera change within three months.
  5. Optional: train the CNN from the imaging track on RetinaMNIST and compare its score distributions with the simulation.

Open the notebook in ColabView on GitHubWorked solution coming soon

Key takeaways

  • AI retinopathy screening is well established, but real-world performance varies between systems.
  • The operating threshold decides referrals, missed cases and specialist workload.
  • Ungradable images need a safe alternative route.
  • Equipment changes can silently raise missed cases; monitor and audit continuously.

Terms used: Dataset shift, Positive predictive value (PPV), Sensitivity (recall), Specificity

Related: CNNs for chest X-ray and retinal images, Diagnostic accuracy metrics, Type 2 diabetes

FAQ

Is AI used for diabetic retinopathy screening?

Yes. Several systems are authorised in different countries, including autonomous systems in primary care, and many screening programmes use or are evaluating them.

How accurate are AI retinopathy screening systems?

Pivotal and development studies report high sensitivity and specificity, but a head-to-head real-world study found sensitivities ranging from about 51% to 86% across seven systems. Local validation is essential.

What happens to ungradable images?

They need another route, such as repeat imaging with pupil dilation or referral for examination, because the AI cannot assess them.

Why might a programme miss more cases after changing cameras?

Image differences can shift the AI's scores. With the same threshold, fewer people cross it, so referrals fall and missed cases rise unless the system is revalidated.

Does AI replace retina specialists?

No. It can take on grading of screening images, but people with referable disease still need specialist examination and treatment.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.