Medical AI · Generative AI & LLMs · Lesson 6 of 7

Multimodal models: text, images and more

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on 2026-10-18 · Last updated 2026-10-18

Intermediate
Technical focus
About 30 minutes
Includes exercise

Before this lesson: How large language models work

In short

Multimodal models process more than one kind of data, most often images and text. Vision-language models learn a shared space in which an image and a matching description lie close together, which allows zero-shot classification and draft report generation. They inherit the shortcut problem of imaging AI: they can learn hospital, scanner or label artefacts instead of disease. Evaluations of AI-drafted radiology reports find clinically significant errors in both AI and human reports, so their role so far is assistive.

You will learn to

Explain how vision-language models align images and text in a shared embedding space
Perform zero-shot classification with similarity between image and text embeddings
Show how a shortcut feature inflates performance and fails at a new hospital
Appraise studies of AI-generated radiology reports

What 'multimodal' means

A multimodal model takes in, and sometimes produces, several types of data: text, images, audio, signals or structured data. In medicine, the most developed examples combine images with text: describing an X-ray, answering questions about a pathology slide, or drafting a radiology report. Researchers have proposed 'generalist medical AI' that combines many modalities, though such systems remain largely in research.

Contrastive models such as CLIP are trained on many image-caption pairs to pull matching image and text embeddings together and push mismatched ones apart. Generative vision-language models connect an image encoder to a language model so it can write about images.

Zero-shot classification in a shared space

Once images and text share a space, an image can be classified without task-specific training: embed short descriptions of each possible finding and pick the one most similar to the image. The toy below uses invented embeddings to show the mechanics.

python

import numpy as np

rng = np.random.default_rng(12)
dim = 32
classes = ["normal", "pneumonia", "pleural effusion"]
concepts = rng.normal(size=(3, dim))       # what each finding "looks like" in the shared space
hospital_tag = rng.normal(size=dim)        # a scanner/hospital artefact visible in images

def normalise(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

# Text embeddings for prompts like "a chest X-ray showing pneumonia"
text = normalise(concepts + rng.normal(0, 0.2, (3, dim)))

def make_images(n, p_tag_by_class):
    labels = rng.integers(0, 3, n)
    tagged = rng.random(n) < np.array(p_tag_by_class)[labels]
    imgs = concepts[labels] + rng.normal(0, 3.0, (n, dim)) + 3.0 * tagged[:, None] * hospital_tag
    return normalise(imgs), labels

def zero_shot_accuracy(imgs, labels):
    predictions = (imgs @ text.T).argmax(axis=1)    # most similar text prompt
    return (predictions == labels).mean()

imgs, labels = make_images(3000, [0.1, 0.1, 0.1])
print(f"Zero-shot accuracy: {zero_shot_accuracy(imgs, labels):.1%}")

# Zero-shot accuracy: 77.8%

Toy embeddings for three chest X-ray findings, with a scanner artefact visible in some images.

Shortcuts: learning the hospital instead of the disease

Imaging models have repeatedly been found to rely on features unrelated to disease: hospital-specific markers, portable-scanner appearance, text labels on the image. A pneumonia model studied by Zech and colleagues could identify which hospital an X-ray came from, and its performance varied between hospitals. In the toy below, pneumonia patients at the development hospital are mostly imaged on a scanner that leaves a recognisable artefact.

python

from sklearn.linear_model import LogisticRegression

# Development hospital: pneumonia patients mostly imaged on the tagged portable scanner
dev_imgs, dev_labels = make_images(3000, [0.1, 0.9, 0.1])
probe = LogisticRegression(max_iter=2000).fit(dev_imgs, dev_labels)
test_same, y_same = make_images(3000, [0.1, 0.9, 0.1])
test_new, y_new = make_images(3000, [0.5, 0.5, 0.5])   # new hospital: scanner unrelated to diagnosis
print(f"Fine-tuned classifier, same hospital: {probe.score(test_same, y_same):.1%}")
print(f"Fine-tuned classifier, new hospital:  {probe.score(test_new, y_new):.1%}")
print(f"Zero-shot, new hospital:              {zero_shot_accuracy(test_new, y_new):.1%}")

# Fine-tuned classifier, same hospital: 84.7%
# Fine-tuned classifier, new hospital:  67.6%
# Zero-shot, new hospital:              72.6%

Continuing from the previous block: train a classifier on embeddings at one hospital, test at a new one.

The trained classifier looked better than zero-shot at its own hospital (84.7% versus 77.8%) partly because it learned that the artefact means pneumonia. At a new hospital it fell to 67.6%, below the zero-shot approach.
Large pre-trained models do not remove this risk: they can carry their own shortcuts, and fine-tuning on local data can add new ones. Test at sites and on equipment the model has not seen.

AI-drafted radiology reports

Tanno and colleagues (Nature Medicine, 2025) asked 27 board-certified radiologists in the US and India to compare and correct chest X-ray reports written by a vision-language model and by radiologists. Clinically significant errors appeared in both: for inpatient and outpatient cases, 24.8% of cases had such errors in both reports, 22.8% only in the AI report and 14.0% only in the human report. The authors also tested an assistive setting in which clinicians revised AI drafts, pointing to collaboration rather than autonomous reporting as the realistic role.

Question to ask	Why
Who graded the reports, and were they blinded?	Knowing the source of a report biases judgement
Were errors weighted by clinical significance?	A missed pneumothorax is not the same as a style issue
Was the comparison against radiologists in realistic conditions?	Time pressure and access to prior images change human performance
Were the test images from new hospitals and equipment?	Shortcuts appear only out of distribution

Images carry identifiers too

Medical images can contain names or record numbers burned into the pixels or stored in DICOM headers. De-identify both before using images with external tools.

Exercise

Explore zero-shot classification and shortcut learning with toy embeddings, then with an open chest X-ray dataset if available.

Data: Toy embeddings generated in the notebook; optional extension uses an openly licensed chest X-ray dataset (Generated in the notebook; extension dataset under its own licence)

Reproduce the zero-shot and shortcut results, and vary how strongly the artefact is tied to pneumonia.
Remove the artefact direction from the embeddings (project it out) and retrain; how does new-hospital accuracy change?
Try 'prompt ensembling': average several text descriptions per class and compare zero-shot accuracy.
Optional: with an open chest X-ray dataset and an open vision-language model, compare zero-shot accuracy for two sites or acquisition types.

The runnable notebook and worked solution will be published soon. You can work through the steps with the code on this page in the meantime.

Key takeaways

Vision-language models align images and text, enabling zero-shot classification and report drafting.
Shortcuts such as scanner artefacts inflate results and fail at new sites.
AI and human radiology reports both contain clinically significant errors; AI's role so far is assistive.
Evaluate on unseen hospitals and equipment, and de-identify image pixels and metadata.

Terms used: Dataset shift, DICOM, Embedding, Shortcut learning, Vision-language model, Zero-shot learning

FAQ

What is a vision-language model?

A model that processes both images and text, either by aligning them in a shared space (contrastive models such as CLIP) or by connecting an image encoder to a language model that writes about images.

What is zero-shot classification?

Classifying inputs into categories the model was not specifically trained on, by comparing an image with text descriptions of each category.

Can AI write radiology reports?

Research systems can draft reports, and studies show errors in both AI and human reports. Current evidence supports drafting with radiologist review rather than autonomous reporting.

What is shortcut learning?

When a model learns a feature that happens to correlate with the label in training data, such as a hospital marker, instead of the actual clinical signal. It fails when that correlation changes.

Do general multimodal chatbots read medical images reliably?

Not reliably enough for clinical decisions. General-purpose models are not validated as medical devices for image interpretation, and identifiable images should not be uploaded to public tools.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.