Medical AI · Retrieval-augmented generation (RAG) · Lesson 1 of 7
Why RAG? Grounding answers in trusted sources
Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on · Last updated
- Beginner
- Clinical focus
- About 20 minutes
- Includes exercise
Before this lesson: How large language models work, Hallucination and grounding
In short
Retrieval-augmented generation (RAG) answers a question in two steps: first it retrieves the most relevant passages from a trusted document collection, such as your organisation's guidelines; then it gives those passages to a language model with instructions to answer only from them and cite them. This keeps answers current and checkable without retraining the model. RAG reduces, but does not remove, errors: retrieval can miss the right passage, and the model can still misread what it is given.
You will learn to
- Describe the indexing, retrieval, augmentation and generation steps of RAG
- Explain why RAG suits clinical questions about guidelines and local protocols
- Build a minimal retrieve-then-prompt pipeline in Python
- List the failure modes that RAG does not solve
How RAG works
Lewis and colleagues introduced the term in 2020. Today the pattern underlies most systems that answer questions from specific documents, from guideline assistants to tools that search a patient's record.
| Step | What happens | Example |
|---|---|---|
| 1. Index | Split trusted documents into passages (chunks) and store them so they can be searched | Each section of a clinic protocol becomes a searchable chunk |
| 2. Retrieve | Find the passages most relevant to the question | 'HbA1c after a treatment change' retrieves the monitoring section |
| 3. Augment | Put the retrieved passages into the prompt with instructions | 'Answer only from the sources below and cite them' |
| 4. Generate | The model writes an answer based on the passages, with citations | 'Check every 3 months after a change [protocol-2026]' |
Why it suits medicine
Studies support the idea. A retrieval-grounded clinical system, Almanac, gave answers that clinicians rated as more factual than those of ungrounded models (Zakka et al., 2024), and in the MIRAGE benchmark, adding retrieval improved the accuracy of several language models on medical questions by up to 18% (Xiong et al., 2024).
- Currency: guidelines change, and a model's built-in knowledge stops at its training cut-off. Updating the documents updates the answers, with no retraining.
- Local relevance: answers can follow your organisation's protocols, formularies and pathways rather than generic advice.
- Checkability: each statement can point to a source passage that a clinician can read.
- Control: the organisation decides which documents are trusted, and access rules can limit who sees what.
A minimal retrieve-then-prompt example
The example retrieves from a tiny synthetic document library using TF-IDF keyword similarity, then builds the prompt that would be sent to a model. Note that the library contains an outdated 2024 protocol and its 2026 update.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
documents = {
"protocol-2024": "Clinic protocol 2024: review HbA1c every 6 months when stable.",
"protocol-2026": "Clinic protocol 2026 update: review HbA1c every 3 months for the first year "
"after any treatment change, then every 6 months when stable.",
"foot-care": "Foot checks: examine both feet at every diabetes review and test sensation "
"with a monofilament once a year.",
"eye-screening": "Eye screening: arrange retinal photography at diagnosis and then at the "
"interval set by the local screening programme.",
}
question = "How often should HbA1c be reviewed after a treatment change?"
vectoriser = TfidfVectorizer(stop_words="english").fit(list(documents.values()) + [question])
doc_vectors = vectoriser.transform(documents.values())
scores = cosine_similarity(vectoriser.transform([question]), doc_vectors)[0]
for name, score in sorted(zip(documents, scores), key=lambda t: -t[1]):
print(f"{score:.2f} {name}")
# 0.34 protocol-2026
# 0.13 protocol-2024
# 0.00 foot-care
# 0.00 eye-screening- The 2026 update ranks first here, but only because its wording matches the question better. Nothing in plain similarity search prefers the current version, which is why real systems retire outdated documents or filter by version (covered in the embeddings lesson).
Building the grounded prompt
best = max(zip(documents, scores), key=lambda t: t[1])[0]
prompt = f"""Answer using only the source below. If the source does not answer, say so.
Cite the source id in square brackets.
Source [{best}]: {documents[best]}
Question: {question}"""
print(prompt)
# Answer using only the source below. If the source does not answer, say so.
# Cite the source id in square brackets.
#
# Source [protocol-2026]: Clinic protocol 2026 update: review HbA1c every 3 months for the first year after any treatment change, then every 6 months when stable.
#
# Question: How often should HbA1c be reviewed after a treatment change?- The instructions matter: 'only the source', 'say so if it does not answer' and 'cite the source id' give the model a safe way to decline and make its answer checkable.
- This prompt would be sent to an approved model using the patterns from the prompting lesson, and the answer validated before use.
About the teaching document
- The protocol used in this track was written for teaching. Its statements are broadly in line with common practice, but it is not clinical guidance and must not be used for patient care.
- In a real system, the documents would be your organisation's approved guidelines, with version control and an owner responsible for keeping them current.
What RAG does not fix
| Failure | What happens | Covered in |
|---|---|---|
| Retrieval miss | The right passage is not retrieved, so the model answers from the wrong one or from memory | Chunking; embeddings; retrieval evaluation |
| Outdated or conflicting sources | Old and new versions both retrieved | Document governance; metadata filters |
| Ignoring the source | The model adds or changes details despite instructions; long contexts make this worse, as models use information in the middle of long inputs less well (Liu et al., 2024) | Citations and grounding |
| False citations | A citation is attached to a statement the source does not support | Citations and grounding |
| Questions the documents cannot answer | The model guesses instead of declining | Prompting; end-to-end evaluation |
Exercise
Build a minimal RAG pipeline over a synthetic document library and observe where it succeeds and fails.
Data: Synthetic protocol documents written for the notebook (Created for this course; not clinical guidance)
- Reproduce the retrieval and prompt-building steps.
- Write five questions, including one the documents cannot answer, and record which document is retrieved for each.
- Retire the 2024 protocol by adding a 'superseded' flag and filtering it out; check the rankings again.
- Optional, with an approved model: send the prompts and compare answers with and without the retrieved source.
The runnable notebook and worked solution will be published soon. You can work through the steps with the code on this page in the meantime.
Key takeaways
- RAG retrieves trusted passages and asks the model to answer from them, with citations.
- It keeps answers current and local without retraining the model.
- Retrieval quality and document governance decide much of the result.
- RAG reduces errors but does not remove them; answers still need checking.
Terms used: Chunking, Grounding, Hallucination, RAG (retrieval-augmented generation)
Related: Chunking clinical guidelines, Hallucination and grounding, Fine-tuning with LoRA
FAQ
What is retrieval-augmented generation?
A method in which relevant passages are first retrieved from a trusted document collection and then given to a language model, which is instructed to answer from them and cite them.
Is RAG better than fine-tuning for clinical guidelines?
Usually, yes. Guidelines change and must be quoted accurately; RAG lets you update the documents without retraining and shows where each answer came from.
Does RAG stop hallucinations?
It reduces them by supplying trusted text, but the model can still misread or embellish passages, and retrieval can fail. Answers need validation and, where it matters, clinician review.
What documents should a clinical RAG system use?
Approved, version-controlled sources with a named owner, such as current guidelines, local protocols and formularies, with outdated versions retired.
Can RAG be used with patient records?
Yes, for example to answer questions about a patient's history, but this requires strict access control, data protection and audit, because the retrieved text is identifiable patient information.
Sources
- Lewis P, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS 2020. arXiv:2005.11401
- Zakka C, et al. Almanac — retrieval-augmented language models for clinical medicine. NEJM AI 2024;1(2). doi:10.1056/AIoa2300068
- Xiong G, et al. Benchmarking retrieval-augmented generation for medicine. Findings of ACL 2024. arXiv:2402.13178
- Liu NF, et al. Lost in the middle: how language models use long contexts. Transactions of the ACL 2024;12:157–173. doi:10.1162/tacl_a_00638
- Gao Y, et al. Retrieval-augmented generation for large language models: a survey. 2023. arXiv:2312.10997
Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.