Medical AI · Retrieval-augmented generation (RAG) · Lesson 6 of 7

Build a guideline Q&A assistant

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on 2026-10-18 · Last updated 2026-10-18

Intermediate
Technical focus
About 35 minutes
Includes exercise

Before this lesson: Citations and answer grounding, Embeddings and vector stores

In short

A guideline question-answering assistant combines the RAG steps: split the guideline into sections, embed them, retrieve the best match for a question, decline if the match is weak, and answer from the retrieved text with a citation. The version built here answers by quoting the source, which keeps every word traceable and runs without an AI model API. Testing it shows a typical failure: quoting too little can drop the most important step, such as how much carbohydrate to give for a low glucose.

You will learn to

Assemble chunking, abbreviation expansion, retrieval and answering into one assistant
Make the assistant decline questions its documents do not cover
Answer extractively with citations, and see how answer length affects safety
Explain how to swap in a language model for generation safely

The design

Step	This assistant	Why
Index	One chunk per protocol section, split into sentences	Sections keep rules whole; sentences can be quoted
Query	Expand known abbreviations	Fixes misses such as 'hypo'
Retrieve	Embedding similarity; best section only	Small document; one section usually answers
Decline	If the best score is below a threshold	Avoids answering from an irrelevant section
Answer	Quote sentences from the section, with a citation	Every word is traceable to the source

About the teaching document

The protocol used in this track was written for teaching. Its statements are broadly in line with common practice, but it is not clinical guidance and must not be used for patient care.
In a real system, the documents would be your organisation's approved guidelines, with version control and an owner responsible for keeping them current.

The assistant

python

import re
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Index: one chunk per section, each split into sentences that can be quoted and cited
chunks = []
for i, part in enumerate(re.split(r"\n(?=## )", PROTOCOL)[1:], start=1):
    heading, *body = part.strip().splitlines()
    sentences = re.split(r"(?<=\.)\s+", " ".join(body).strip())
    chunks.append({"id": f"s{i}", "heading": heading.lstrip("# "), "sentences": sentences})
chunk_emb = model.encode([c["heading"] + ". " + " ".join(c["sentences"]) for c in chunks],
                         normalize_embeddings=True)

ABBREVIATIONS = {"hypo": "hypo (hypoglycaemia, low blood glucose)", "biguanide": "biguanide (metformin)"}

def expand(q):
    for short, long in ABBREVIATIONS.items():
        q = re.sub(rf"\b{short}\b", long, q, flags=re.IGNORECASE)
    return q

def answer(question, min_score=0.35, max_sentences=2):
    q = expand(question)
    q_emb = model.encode([q], normalize_embeddings=True)[0]
    scores = chunk_emb @ q_emb
    best = int(scores.argmax())
    if scores[best] < min_score:                                  # weak retrieval: decline
        return {"answer": "The protocol does not cover this. Please ask your care team.",
                "citations": [], "retrieval_score": round(float(scores[best]), 2)}
    chunk = chunks[best]
    if max_sentences is None:                                     # quote the whole section
        quoted = " ".join(chunk["sentences"])
    else:
        sent_scores = model.encode(chunk["sentences"], normalize_embeddings=True) @ q_emb
        top = sorted(np.argsort(-sent_scores)[:max_sentences])    # keep document order
        quoted = " ".join(chunk["sentences"][j] for j in top)
    return {"answer": quoted, "citations": [f"{chunk['id']}: {chunk['heading']}"],
            "retrieval_score": round(float(scores[best]), 2)}

Run the protocol block from the chunking lesson first. Installs: pip install sentence-transformers

Trying it out

python

for q in ["When should metformin be stopped?",
          "What should I do about a hypo?",
          "What dose of semaglutide should I start?"]:
    r = answer(q)
    print(f"Q: {q}\nA: {r['answer']}\n   citations {r['citations']}  retrieval score {r['retrieval_score']}\n")

# Q: When should metformin be stopped?
# A: Metformin should not be started if eGFR is below 45. If eGFR falls below 30 during treatment, stop metformin.
#    citations ['s2: 2. Kidney function and metformin']  retrieval score 0.53
#
# Q: What should I do about a hypo?
# A: Repeat if glucose is still below 70 mg/dL. Review any medicine that may have caused the low glucose.
#    citations ['s4: 4. Low blood glucose']  retrieval score 0.63
#
# Q: What dose of semaglutide should I start?
# A: The protocol does not cover this. Please ask your care team.
#    citations []  retrieval score 0.32

Continuing from the previous block: two covered questions and one the protocol does not cover.

The metformin answer is correct and cited, and the semaglutide question is declined because the protocol does not cover it.
The 'hypo' answer found the right section but is dangerously incomplete: it quotes the two sentences most similar to the question and leaves out the key step, treating with 15 g of fast-acting carbohydrate.

Fixing the incomplete answer

python

r = answer("What should I do about a hypo?", max_sentences=None)
print(f"A: {r['answer']}\n   citations {r['citations']}")

# A: A glucose below 70 mg/dL is low. Treat with 15 g of fast-acting carbohydrate and recheck after 15 minutes. Repeat if glucose is still below 70 mg/dL. Review any medicine that may have caused the low glucose.
#    citations ['s4: 4. Low blood glucose']

Continuing from the previous blocks: quote the whole section instead of the top two sentences.

For short, structured sections, quoting the whole section is safer than picking sentences: the steps of a protocol depend on each other.
For long sections, a better approach is to quote enough surrounding context, or to let a model write a summary that is then checked against the section.

Adding a language model safely

Quoting is safe but can be awkward to read. A language model can rewrite the retrieved section as a direct answer, using the prompting and validation patterns from earlier lessons.

Give the model only the retrieved section, with instructions to answer from it, say when it does not cover the question, and cite it.
Keep the decline rule outside the model: if retrieval is weak, do not call the model at all.
Check the model's answer against the section with the citation checks from the previous lesson, including negation, before showing it.
Show the quoted source alongside the generated answer, so clinicians can see what it is based on.
Use only models and services approved for clinical use, and never send identifiable patient details in questions.

Exercise

Build, test and improve a guideline Q&A assistant over the synthetic protocol.

Data: Synthetic protocol in the notebook; small open embedding model all-MiniLM-L6-v2 (Course content; model under Apache 2.0)

Reproduce the assistant and the three example questions.
Write ten questions, including two out of scope and two using abbreviations, and record each answer.
Compare max_sentences=2 with whole-section quoting on all ten questions; which answers lose important information?
Add a second retrieved section when the top two scores are close, and cite both.
Optional, with an approved model: add a generation step that rewrites the quoted section as an answer, and check it with the citation checker.

Open the notebook in Colab View on GitHubWorked solution coming soon

Key takeaways

A guideline assistant chains chunking, expansion, retrieval, a decline rule and answering.
Decline when retrieval is weak, outside the language model.
Extractive answers are traceable but must quote enough context to be safe.
If a model writes the answer, check it against the source before showing it.

Terms used: Chunking, Embedding, Grounding, RAG (retrieval-augmented generation)

FAQ

Can I build a guideline assistant without an AI model API?

Yes. Retrieval with a small open embedding model and answers quoted from the source work offline, as in this lesson. A language model can be added later to make answers more readable.

How does the assistant know when to say it cannot answer?

It declines when the best retrieval score falls below a threshold, chosen by testing questions the documents can and cannot answer.

Why did the first answer about hypoglycaemia leave out the treatment?

Because it picked the two sentences most similar to the question, which were not the most important ones. Quoting the whole short section fixed it.

Should a guideline assistant be used by patients?

Only with great care, clear limits and regulatory review. Patient-facing tools that give advice about treatment may be medical devices, and they must handle emergencies and out-of-scope questions safely.

How do I keep the assistant up to date?

Re-index whenever the guideline changes, store version metadata, retire outdated versions, and re-run the evaluation set after every update.

Sources

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.