Skip to content
Dr. Ram KumarDiabetes & metabolic health

Medical AI · Generative AI & LLMs · Lesson 5 of 7

Fine-tuning with LoRA on synthetic data

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on · Last updated

  • Advanced
  • Technical focus
  • About 35 minutes
  • Includes exercise

Before this lesson: How large language models work, Prompting for clinical tasks

In short

Fine-tuning adapts a pre-trained model to a specific task by training it further on examples. LoRA (low-rank adaptation) does this cheaply: it freezes the original weights and trains small add-on matrices, often less than 1% of the parameters. Fine-tuning is best for teaching a consistent format, style or narrow skill; for up-to-date facts, retrieval is usually better. Fine-tuning on patient data raises privacy risks, because models can memorise training text.

You will learn to

  • Explain when to use prompting, retrieval or fine-tuning
  • Describe how LoRA adapts a frozen weight matrix with a low-rank update
  • Implement LoRA on a toy problem in NumPy and see the effect of rank
  • Identify data, privacy and evaluation requirements for clinical fine-tuning

Prompting, retrieval or fine-tuning?

Start with prompting and an evaluation set. Add retrieval if the model needs specific documents. Consider fine-tuning when prompting plateaus and you have enough good examples. Van Veen and colleagues found that adapted models produced clinical summaries that clinicians often preferred to those written by medical experts, but adaptation included both prompting and parameter-efficient fine-tuning, compared on careful evaluations.

ApproachBest forNot good for
Prompting (with examples)Quick start; tasks the model can already do with instructionsLong, complex formats; consistent behaviour at scale
Retrieval (RAG)Answering from specific, current documents such as guidelinesChanging the model's style or skills
Fine-tuningA consistent format or style, a narrow repeated task, a smaller cheaper modelTeaching new or changing facts; small, unrepresentative datasets

How LoRA works

A language model is made of large weight matrices. Full fine-tuning updates all of them, which needs a lot of memory and produces a full copy of the model. LoRA keeps each original matrix W frozen and learns an update written as the product of two thin matrices, B × A, where the inner dimension r (the rank) is small. The adapted layer computes (W + B × A) applied to its input.

Because B starts at zero, the adapted model begins identical to the original. The adapters are small files that can be swapped in and out of the same base model. QLoRA combines this with a compressed (quantised) base model so that fine-tuning fits on a single GPU.

python
d = 4096   # width of one weight matrix in a mid-sized language model layer
for rank in [4, 8, 16, 64]:
    full = d * d
    lora = 2 * d * rank          # A is r x d, B is d x r
    print(f"rank {rank:>2}: LoRA trains {lora:>9,} parameters vs {full:,} ({lora / full:.2%})")

# rank  4: LoRA trains    32,768 parameters vs 16,777,216 (0.20%)
# rank  8: LoRA trains    65,536 parameters vs 16,777,216 (0.39%)
# rank 16: LoRA trains   131,072 parameters vs 16,777,216 (0.78%)
# rank 64: LoRA trains   524,288 parameters vs 16,777,216 (3.12%)
How many parameters LoRA trains for one 4,096 × 4,096 weight matrix.

LoRA in NumPy

The toy below has one frozen 64 × 64 'pre-trained' layer. The new task needs a change to that layer of rank 3. We train LoRA adapters of different ranks and measure how well each fits the new task.

python
import numpy as np

rng = np.random.default_rng(0)
d = 64
W0 = rng.normal(0, 1 / np.sqrt(d), (d, d))                 # frozen "pre-trained" weights
U, V = rng.normal(0, 0.3, (d, 3)), rng.normal(0, 0.3, (3, d))
true_change = U @ V                                          # the new task needs a rank-3 change
X = rng.normal(size=(2000, d))
Y = X @ (W0 + true_change).T

def train_lora(rank, steps=1000, lr=0.05):
    r = np.random.default_rng(1)
    A = r.normal(0, 0.01, (rank, d))   # trained
    B = np.zeros((d, rank))            # trained; starts at zero so the model starts as W0
    for _ in range(steps):
        err = X @ (W0 + B @ A).T - Y
        grad = err.T @ X / len(X)
        B, A = B - lr * grad @ A.T, A - lr * B.T @ grad
    return np.mean(err**2), A.size + B.size

print(f"No adaptation:  error {np.mean((X @ W0.T - Y)**2):.3f}")
for rank in [1, 2, 3, 8]:
    error, n_params = train_lora(rank)
    print(f"LoRA rank {rank}:    error {error:.3f}   trained parameters {n_params} of {W0.size}")

# No adaptation:  error 1.596
# LoRA rank 1:    error 0.886   trained parameters 128 of 4096
# LoRA rank 2:    error 0.351   trained parameters 256 of 4096
# LoRA rank 3:    error 0.000   trained parameters 384 of 4096
# LoRA rank 8:    error 0.000   trained parameters 1024 of 4096
Train LoRA adapters of rank 1, 2, 3 and 8 on a task that needs a rank-3 change.
  • W0 never changes. Only A and B are trained, 384 numbers instead of 4,096 for rank 3.
  • A rank smaller than the change the task needs cannot fit it fully; a larger rank adds capacity that is not needed. Real fine-tuning works because useful task adaptations are often well approximated by low-rank updates.
  • Libraries such as Hugging Face PEFT apply the same idea to the attention and feed-forward layers of real models.

Data for clinical fine-tuning

  • Quality beats quantity: a few hundred to a few thousand carefully reviewed examples of the exact input and desired output often matter more than a large, noisy set.
  • Synthetic data, such as notes written by clinicians for training or generated and then reviewed, avoids exposing patient records and can cover rare cases deliberately.
  • Hold out a test set, created separately, and evaluate the fine-tuned model against the base model with the best prompt, not against a weak baseline.
  • Check that the model has not lost general abilities or safety behaviours (catastrophic forgetting), and re-run safety tests.

Models can memorise training text

  • Language models can reproduce parts of their training data verbatim, as shown by training-data extraction attacks. Fine-tuning on identifiable patient records risks leaking them through the model.
  • Fine-tune on de-identified or synthetic data where possible, under data-governance approval, and treat a model fine-tuned on patient data as sensitive in itself.

Exercise

Implement and explore LoRA in NumPy, then plan a responsible clinical fine-tuning project.

Data: Synthetic matrices and examples generated in the notebook (Generated in the notebook)

  1. Reproduce the rank comparison and plot error against rank.
  2. Add noise to the targets and use a held-out set to see whether a large rank overfits.
  3. Apply a LoRA adapter, then 'merge' it into W0 (W0 + B @ A) and confirm the outputs are identical.
  4. Optional, with a GPU: fine-tune a small open model with a LoRA adapter (Hugging Face PEFT) on the provided synthetic note-to-summary pairs, and compare with the base model plus prompting on held-out examples.
  5. Write a one-page plan covering data source, de-identification, evaluation, safety testing and who approves deployment.

The runnable notebook and worked solution will be published soon. You can work through the steps with the code on this page in the meantime.

Key takeaways

  • Try prompting and retrieval before fine-tuning; fine-tune for format, style or narrow skills.
  • LoRA trains small low-rank adapters on a frozen model, often under 1% of parameters.
  • Evaluate against the best prompted baseline and re-test safety after fine-tuning.
  • Models can memorise training data: avoid fine-tuning on identifiable patient records.

Terms used: Fine-tuning, Foundation model, LoRA (low-rank adaptation), Synthetic data

Related: Prompting for clinical tasks, Privacy and de-identification, RAG track

FAQ

What is LoRA?

Low-rank adaptation: a fine-tuning method that freezes a model's weights and trains small pairs of matrices whose product is added to selected layers, drastically reducing what needs training and storing.

Should I fine-tune a model on our hospital's guidelines?

Usually not. Guidelines change and need to be quoted accurately, which retrieval does better. Fine-tuning suits consistent formats or tasks, not keeping facts up to date.

How much data do I need to fine-tune?

For a narrow task, often hundreds to a few thousand high-quality examples. The quality and representativeness of examples matter more than the number.

Is it safe to fine-tune on patient notes?

It carries privacy risk because models can memorise and reproduce training text. Use de-identified or synthetic data where possible, with governance approval and security controls.

What is QLoRA?

LoRA applied to a quantised (compressed) base model, which reduces memory needs enough to fine-tune large models on a single GPU.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.