Skip to content

Medical AI · Clinical NLP · Lesson 1 of 4

Working with clinical notes

Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on · Last updated

  • Beginner
  • Technical focus
  • About 25 minutes
  • Includes exercise

Before this lesson: Healthcare data types: records, images, signals, notes and omics, Setting up Python and Colab for clinical data work

In short

Clinical notes hold much of the information that never reaches structured fields, but they are hard for computers: they mix sections, use local abbreviations, record what is absent as often as what is present ('denies chest pain'), and repeat text copied forward from earlier notes. Before any model, split notes into sections, expand abbreviations carefully, detect negation and uncertainty, and watch for duplicated text. Simple rules handle a lot of this, and their failures show where better tools are needed.

You will learn to

  • Describe what makes clinical text different from ordinary text
  • Split a note into sections and expand abbreviations in Python
  • Detect negated findings with a NegEx-style rule
  • Measure copied-forward text between notes

Why clinical text is hard

Duplication is common: one large study found that half of the text in inpatient notes was duplicated from earlier notes (Steinkamp et al., 2022).

FeatureExampleWhy it matters
SectionsHistory, Medications, Examination, PlanThe same word means different things in different sections
AbbreviationsT2DM, c/o, BD, SOBMany are local or ambiguous; 'MS' can mean several things
Negation'Denies chest pain', 'No ulcers'A finding mentioned is often a finding absent
Uncertainty and context'Probable neuropathy', 'mother had diabetes'Not every mention is a confirmed fact about this patient
Copy-forwardYesterday's examination pasted into today's noteOld findings look current; duplication inflates counts

Splitting a note into sections

python
import re

note = """HISTORY: 58M with T2DM x 10 yrs, HTN. c/o tingling in both feet x 3 mo. Denies chest pain. No SOB.
MEDICATIONS: Metformin 1 g BD. Amlodipine 5 mg OD.
EXAMINATION: Reduced monofilament sensation both feet. No ulcers. Pedal pulses present.
ASSESSMENT: Probable diabetic peripheral neuropathy. HbA1c 8.4%.
PLAN: Refer for DPN testing. Foot care advice. Review in 3 mo."""

def split_sections(text):
    """Split a note on headings written in capitals followed by a colon."""
    parts = re.split(r"^([A-Z][A-Z ]+):\s*", text, flags=re.MULTILINE)[1:]
    return {parts[i].title(): parts[i + 1].strip() for i in range(0, len(parts), 2)}

for name, body in split_sections(note).items():
    print(f"[{name}] {body}")

# [History] 58M with T2DM x 10 yrs, HTN. c/o tingling in both feet x 3 mo. Denies chest pain. No SOB.
# [Medications] Metformin 1 g BD. Amlodipine 5 mg OD.
# [Examination] Reduced monofilament sensation both feet. No ulcers. Pedal pulses present.
# [Assessment] Probable diabetic peripheral neuropathy. HbA1c 8.4%.
# [Plan] Refer for DPN testing. Foot care advice. Review in 3 mo.
A synthetic diabetes clinic note, split on its capitalised headings.
  • Real notes rarely have such tidy headings; section detection often needs a list of heading variants or a trained model.

Expanding abbreviations

python
ABBREVIATIONS = {
    "T2DM": "type 2 diabetes mellitus", "HTN": "hypertension", "c/o": "complains of",
    "SOB": "shortness of breath", "BD": "twice daily", "OD": "once daily",
    "mo": "months", "yrs": "years", "DPN": "diabetic peripheral neuropathy",
}
def expand(text):
    for short, long in ABBREVIATIONS.items():
        text = re.sub(rf"(?<![\w/]){re.escape(short)}(?![\w/])", long, text)
    return text

print(expand(split_sections(note)["History"]))

# 58M with type 2 diabetes mellitus x 10 years, hypertension. complains of tingling in both feet x 3 months. Denies chest pain. No shortness of breath.
Continuing from the previous block: expand a small, curated abbreviation list.
  • Abbreviation lists must be local and reviewed: 'OD' means 'once daily' in some countries and 'right eye' in ophthalmology. Expanding the wrong meaning silently changes the record.

Detecting negation

NegEx, published in 2001, marks a finding as negated when a negation cue such as 'no' or 'denies' appears shortly before it in the same sentence. Its successor ConText also handles uncertainty, history and who the finding refers to.

python
NEGATION_CUES = r"\b(no|denies|denied|without|negative for|absent)\b"
FINDINGS = ["chest pain", "shortness of breath", "ulcers", "tingling", "pedal pulses",
            "reduced monofilament sensation"]

def findings_with_status(text):
    """NegEx-style: a finding is negated if a negation cue appears earlier in the same sentence."""
    results = []
    for sentence in re.split(r"(?<=[.!?])\s+", text):
        s = sentence.lower()
        for f in FINDINGS:
            pos = s.find(f)
            if pos >= 0:
                negated = re.search(NEGATION_CUES, s[:pos]) is not None
                results.append((f, "absent" if negated else "present"))
    return results

for finding, status in findings_with_status(expand(note)):
    print(f"{status:<8} {finding}")

# present  tingling
# absent   chest pain
# absent   shortness of breath
# present  reduced monofilament sensation
# absent   ulcers
# present  pedal pulses
Continuing from the previous blocks: a NegEx-style rule over the expanded note.
  • The rule correctly separates present findings from absent ones in this note.
  • It would miss uncertainty ('probable neuropathy' is not confirmed), family history ('mother had foot ulcers') and negation that comes after the finding ('ulcers: none'). Tools such as medspaCy implement ConText for these cases.

Copy-forward text

python
import difflib

visit_1 = "Tingling in both feet for 3 months. No ulcers. Metformin 1 g twice daily. Plan: refer for neuropathy testing."
visit_2 = visit_1 + " Neuropathy test done: reduced sural nerve response."
visit_3 = visit_2                                           # copied forward unchanged

for name, a, b in [("visit 2 vs 1", visit_1, visit_2), ("visit 3 vs 2", visit_2, visit_3)]:
    print(f"{name}: {difflib.SequenceMatcher(None, a, b).ratio():.0%} identical")

# visit 2 vs 1: 81% identical
# visit 3 vs 2: 100% identical
How much of each synthetic note is identical to the previous one?
  • Visit 3 is entirely copied. Was the feet examination repeated, or just pasted? A model counting mentions would think the finding was documented three times.
  • For research, use the first or the most recent mention of a finding, or remove duplicated text, and record which choice you made.

Synthetic notes only

  • Every note, name and number in this track is invented. Never practise on real clinical notes outside an approved, secure environment, and never paste them into public AI tools.

Exercise

Build a small clinical-text preprocessing pipeline and test where rules break.

Data: Twenty synthetic clinic notes written for the notebook (Created for this course; no real patient data)

  1. Reproduce section splitting, abbreviation expansion and negation detection on all twenty notes.
  2. Add rules for uncertainty ('probable', 'possible', 'query') and family history ('mother', 'father', 'family history of').
  3. Write five notes where your rules fail and explain why.
  4. Measure copy-forward between consecutive notes for each synthetic patient and flag notes that are more than 90% identical.
  5. Optional: run medspaCy's ConText on the same notes and compare.

Open the notebook in ColabView on GitHubWorked solution coming soon

Key takeaways

  • Clinical notes mix sections, abbreviations, negation, uncertainty and copied text.
  • Abbreviation lists must be local and reviewed; wrong expansions change meaning.
  • Negation and context detection are essential before counting findings.
  • Copy-forward makes old findings look current; handle duplication explicitly.

Terms used: EHR (electronic health record), Negation detection

Related: Named-entity recognition and clinical coding, Healthcare data types, DPN Check nerve conduction testing

FAQ

Why is clinical text hard for NLP?

It uses local abbreviations, incomplete sentences, many negated or uncertain findings, and text copied between notes, all of which can mislead general-purpose language tools.

What is NegEx?

A rule-based algorithm, published in 2001, that detects whether a finding in clinical text is negated, using cue words such as 'no' and 'denies' near the finding.

Can a large language model read clinical notes instead?

LLMs handle much of this variation well, but they also make errors with negation and context, and notes must only be processed in approved environments. The same evaluation principles apply.

What is copy-forward?

Copying text from a previous note into a new one. It saves time but can make old findings look current and inflates how often things appear in the record.

Should I remove stop words from clinical text?

Be careful: words like 'no', 'not' and 'without' are often in stop-word lists, and removing them destroys negation.

Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.