Medical AI · Deep learning & imaging · Lesson 5 of 5

Transfer learning and imaging foundation models

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on 2026-10-18 · Last updated 2026-10-18

Advanced
Technical focus
About 30 minutes
Includes exercise

Before this lesson: CNNs for chest X-ray and retinal images

In short

Transfer learning reuses a model trained on a large dataset as the starting point for a new task, so far fewer labelled images are needed. You can fine-tune all of its layers on the new data or freeze the learned features and retrain only the final layer. Imaging foundation models take this further: pretrained on millions of unlabelled medical images, such as retinal photographs, they can be adapted to many tasks with little labelled data. A pretrained model still has to be validated at each new site, because used as it is, it can fail on a different scanner.

You will learn to

Explain transfer learning, fine-tuning and frozen-feature (linear probe) approaches
Compare training from scratch, fine-tuning and frozen features with only 100 local images
Describe how imaging foundation models are trained and used
Identify the validation needed before using a pretrained model at a new site

Why reuse a model

Labelled medical images are expensive: each needs expert annotation. A network that has already learned edges, textures and shapes from a large dataset can learn a new task from far fewer labels. For years the usual starting point was a network pretrained on ImageNet, a large collection of everyday photographs; research found that this helps mainly with small datasets and faster training, and that simpler models can sometimes match it (Raghu et al., 2019).

Approach	What is trained	When it suits
From scratch	All layers, from random starting weights	Large labelled datasets
Fine-tuning	All layers, starting from pretrained weights	Moderate data; target images differ from the pretraining data
Frozen features (linear probe)	Only the final layer	Very little data; quick baseline; target similar to pretraining data
Foundation model adaptation	A small head or adapters on a large pretrained medical model	Many tasks from one model; little labelled data

Transfer to a new scanner with 100 images

The experiment reuses the synthetic scans and SmallCNN from the CNN lesson. A model is pretrained on 4,000 labelled images from site A. Site B has a different scanner, with lower contrast and an intensity offset, and only 100 labelled images.

python

import copy

def new_scanner(n, seed):
    """Target site: a different scanner gives lower contrast and an intensity offset."""
    X, y = make_images(n, seed)
    return 0.6 * X + 0.5, y

X_source, y_source = make_images(4000, seed=10)          # plenty of labelled images from site A
X_small, y_small = new_scanner(100, seed=11)              # only 100 labelled images at site B
X_target_test, y_target_test = new_scanner(1000, seed=12)

torch.manual_seed(0)
pretrained = train(SmallCNN(), X_source, y_source)
print(f"Pretrained model, applied to site B as is:  {accuracy(pretrained, X_target_test, y_target_test):.3f}")

torch.manual_seed(0)
scratch = train(SmallCNN(), X_small, y_small, epochs=30)
print(f"Trained from scratch on 100 site-B images:  {accuracy(scratch, X_target_test, y_target_test):.3f}")

fine_tuned = train(copy.deepcopy(pretrained), X_small, y_small, epochs=30)
print(f"Pretrained, fine-tuned on 100 site-B images: {accuracy(fine_tuned, X_target_test, y_target_test):.3f}")

# Pretrained model, applied to site B as is:  0.504
# Trained from scratch on 100 site-B images:  0.724
# Pretrained, fine-tuned on 100 site-B images: 0.854

Run the first block of the CNN lesson first. Compare four ways of getting a model for site B.

Used as it is, the pretrained model is no better than chance at site B (0.504): the new scanner's intensities break it. This is dataset shift, and it is why every site needs its own validation.
Training from scratch on only 100 images reaches 0.724; fine-tuning the pretrained model on the same 100 images reaches 0.854.

About the numbers

The code uses fixed random seeds, PyTorch 2.8 and a laptop CPU. Other versions or hardware can give slightly different numbers; the pattern of results is what matters.
Images in this track are synthetic, so they run anywhere in seconds. The exercises point to MedMNIST, an openly licensed collection of small medical-image datasets, for real images.

Freezing the learned features

python

frozen = copy.deepcopy(pretrained)
for p in frozen.features.parameters():
    p.requires_grad = False                       # keep the learned filters; retrain only the head
frozen.head = nn.Sequential(nn.Flatten(), nn.Linear(16 * 7 * 7, 2))
torch.manual_seed(0)
opt = torch.optim.Adam(frozen.head.parameters(), lr=1e-3)
for _ in range(30):
    for idx in torch.randperm(len(X_small)).split(64):
        opt.zero_grad()
        nn.functional.cross_entropy(frozen(X_small[idx]), y_small[idx]).backward()
        opt.step()
print(f"Frozen features, new head on 100 images:    {accuracy(frozen, X_target_test, y_target_test):.3f}")
trainable = sum(p.numel() for p in frozen.parameters() if p.requires_grad)
print(f"Trainable parameters: {trainable} of {sum(p.numel() for p in frozen.parameters())}")

# Frozen features, new head on 100 images:    0.772
# Trainable parameters: 1570 of 2818

Continuing from the previous block: keep the convolutional filters fixed and retrain only the final layer.

Retraining only the head (0.772) beats training from scratch but falls short of full fine-tuning here, because the new scanner changes the low-level image statistics the early filters respond to.
Frozen features are cheaper and less prone to overfitting with very small datasets; with foundation models trained on similar images, they often work well.

Imaging foundation models

Foundation models for medical imaging are pretrained, often with self-supervised learning that needs no labels, on very large collections of medical images. RETFound, for example, was pretrained on 1.6 million retinal images and then adapted to detect eye diseases and to predict some systemic conditions, outperforming comparison models in many tasks (Zhou et al., 2023). Others pair images with reports, such as models trained on chest X-rays and radiology reports that can classify findings without task-specific labels (Tiu et al., 2022), or are designed for interactive segmentation across many image types (Ma et al., 2024).

Benefits: less labelled data for each new task; one model can serve many tasks.
Cautions: pretraining data may under-represent some populations, devices or diseases; licences can restrict clinical or commercial use; and adapted models still need external validation like any other.
Regulatory status depends on the final product and its intended use, not on the foundation model inside it.

Questions before adopting a pretrained model

What data was it pretrained on: which countries, devices and patient groups?
What licence covers the weights, and does it allow clinical use?
How does it perform on images from our own equipment and population, before and after adaptation?

Exercise

Compare transfer-learning strategies on synthetic and real images, and plan the validation of a pretrained model.

Data: Synthetic scans in the notebook; optional MedMNIST RetinaMNIST or PneumoniaMNIST; optional torchvision pretrained ResNet-18 (Generated in the notebook; MedMNIST under CC BY 4.0; check the licence of any pretrained weights)

Reproduce the four strategies and repeat with 25, 100 and 500 site-B images; plot accuracy against the number of labelled images.
Normalise each site's images (subtract the mean, divide by the standard deviation) and see whether the pretrained model works at site B without fine-tuning.
Optional: fine-tune an ImageNet-pretrained ResNet-18 on RetinaMNIST and compare with training the same network from scratch.
Write the validation plan you would require before using an imaging foundation model in a diabetic retinopathy screening programme.

Open the notebook in Colab View on GitHubWorked solution coming soon

Key takeaways

Transfer learning reuses pretrained models so far fewer local labels are needed.
Fine-tuning adapts all layers; frozen features retrain only the head.
Imaging foundation models are pretrained on millions of medical images for many tasks.
Pretrained models can fail at new sites as they are; adapt and validate locally.

Terms used: Dataset shift, Fine-tuning, Foundation model, Transfer learning

FAQ

What is transfer learning?

Starting a new task from a model already trained on a large dataset, so that it learns the new task with less labelled data and training time.

What is the difference between fine-tuning and a linear probe?

Fine-tuning updates all of the pretrained model's weights on the new data. A linear probe keeps the pretrained layers fixed and trains only a new final layer.

What is a medical imaging foundation model?

A large model pretrained, often without labels, on very many medical images, which can then be adapted to many diagnostic tasks with little labelled data. RETFound for retinal images is one example.

Does a foundation model remove the need for local validation?

No. Differences in devices, populations and image quality can still change performance, so adapted models need validation at each site.

Can ImageNet-pretrained models be used for medical images?

Yes, and they often help with small datasets, although medical-specific pretraining can do better for some tasks. Either way, validate on the target data.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.