Medical AI · Clinical NLP · Lesson 1 of 4
Working with clinical notes
Reviewed by Dr. Ram Kumar G, MBBS, MD (Internal Medicine), MPH (USA), on · Last updated
- Beginner
- Technical focus
- About 25 minutes
- Includes exercise
Before this lesson: Healthcare data types: records, images, signals, notes and omics, Setting up Python and Colab for clinical data work
In short
Clinical notes hold much of the information that never reaches structured fields, but they are hard for computers: they mix sections, use local abbreviations, record what is absent as often as what is present ('denies chest pain'), and repeat text copied forward from earlier notes. Before any model, split notes into sections, expand abbreviations carefully, detect negation and uncertainty, and watch for duplicated text. Simple rules handle a lot of this, and their failures show where better tools are needed.
You will learn to
- Describe what makes clinical text different from ordinary text
- Split a note into sections and expand abbreviations in Python
- Detect negated findings with a NegEx-style rule
- Measure copied-forward text between notes
Why clinical text is hard
Duplication is common: one large study found that half of the text in inpatient notes was duplicated from earlier notes (Steinkamp et al., 2022).
| Feature | Example | Why it matters |
|---|---|---|
| Sections | History, Medications, Examination, Plan | The same word means different things in different sections |
| Abbreviations | T2DM, c/o, BD, SOB | Many are local or ambiguous; 'MS' can mean several things |
| Negation | 'Denies chest pain', 'No ulcers' | A finding mentioned is often a finding absent |
| Uncertainty and context | 'Probable neuropathy', 'mother had diabetes' | Not every mention is a confirmed fact about this patient |
| Copy-forward | Yesterday's examination pasted into today's note | Old findings look current; duplication inflates counts |
Splitting a note into sections
import re
note = """HISTORY: 58M with T2DM x 10 yrs, HTN. c/o tingling in both feet x 3 mo. Denies chest pain. No SOB.
MEDICATIONS: Metformin 1 g BD. Amlodipine 5 mg OD.
EXAMINATION: Reduced monofilament sensation both feet. No ulcers. Pedal pulses present.
ASSESSMENT: Probable diabetic peripheral neuropathy. HbA1c 8.4%.
PLAN: Refer for DPN testing. Foot care advice. Review in 3 mo."""
def split_sections(text):
"""Split a note on headings written in capitals followed by a colon."""
parts = re.split(r"^([A-Z][A-Z ]+):\s*", text, flags=re.MULTILINE)[1:]
return {parts[i].title(): parts[i + 1].strip() for i in range(0, len(parts), 2)}
for name, body in split_sections(note).items():
print(f"[{name}] {body}")
# [History] 58M with T2DM x 10 yrs, HTN. c/o tingling in both feet x 3 mo. Denies chest pain. No SOB.
# [Medications] Metformin 1 g BD. Amlodipine 5 mg OD.
# [Examination] Reduced monofilament sensation both feet. No ulcers. Pedal pulses present.
# [Assessment] Probable diabetic peripheral neuropathy. HbA1c 8.4%.
# [Plan] Refer for DPN testing. Foot care advice. Review in 3 mo.- Real notes rarely have such tidy headings; section detection often needs a list of heading variants or a trained model.
Expanding abbreviations
ABBREVIATIONS = {
"T2DM": "type 2 diabetes mellitus", "HTN": "hypertension", "c/o": "complains of",
"SOB": "shortness of breath", "BD": "twice daily", "OD": "once daily",
"mo": "months", "yrs": "years", "DPN": "diabetic peripheral neuropathy",
}
def expand(text):
for short, long in ABBREVIATIONS.items():
text = re.sub(rf"(?<![\w/]){re.escape(short)}(?![\w/])", long, text)
return text
print(expand(split_sections(note)["History"]))
# 58M with type 2 diabetes mellitus x 10 years, hypertension. complains of tingling in both feet x 3 months. Denies chest pain. No shortness of breath.- Abbreviation lists must be local and reviewed: 'OD' means 'once daily' in some countries and 'right eye' in ophthalmology. Expanding the wrong meaning silently changes the record.
Detecting negation
NegEx, published in 2001, marks a finding as negated when a negation cue such as 'no' or 'denies' appears shortly before it in the same sentence. Its successor ConText also handles uncertainty, history and who the finding refers to.
NEGATION_CUES = r"\b(no|denies|denied|without|negative for|absent)\b"
FINDINGS = ["chest pain", "shortness of breath", "ulcers", "tingling", "pedal pulses",
"reduced monofilament sensation"]
def findings_with_status(text):
"""NegEx-style: a finding is negated if a negation cue appears earlier in the same sentence."""
results = []
for sentence in re.split(r"(?<=[.!?])\s+", text):
s = sentence.lower()
for f in FINDINGS:
pos = s.find(f)
if pos >= 0:
negated = re.search(NEGATION_CUES, s[:pos]) is not None
results.append((f, "absent" if negated else "present"))
return results
for finding, status in findings_with_status(expand(note)):
print(f"{status:<8} {finding}")
# present tingling
# absent chest pain
# absent shortness of breath
# present reduced monofilament sensation
# absent ulcers
# present pedal pulses- The rule correctly separates present findings from absent ones in this note.
- It would miss uncertainty ('probable neuropathy' is not confirmed), family history ('mother had foot ulcers') and negation that comes after the finding ('ulcers: none'). Tools such as medspaCy implement ConText for these cases.
Copy-forward text
import difflib
visit_1 = "Tingling in both feet for 3 months. No ulcers. Metformin 1 g twice daily. Plan: refer for neuropathy testing."
visit_2 = visit_1 + " Neuropathy test done: reduced sural nerve response."
visit_3 = visit_2 # copied forward unchanged
for name, a, b in [("visit 2 vs 1", visit_1, visit_2), ("visit 3 vs 2", visit_2, visit_3)]:
print(f"{name}: {difflib.SequenceMatcher(None, a, b).ratio():.0%} identical")
# visit 2 vs 1: 81% identical
# visit 3 vs 2: 100% identical- Visit 3 is entirely copied. Was the feet examination repeated, or just pasted? A model counting mentions would think the finding was documented three times.
- For research, use the first or the most recent mention of a finding, or remove duplicated text, and record which choice you made.
Synthetic notes only
- Every note, name and number in this track is invented. Never practise on real clinical notes outside an approved, secure environment, and never paste them into public AI tools.
Exercise
Build a small clinical-text preprocessing pipeline and test where rules break.
Data: Twenty synthetic clinic notes written for the notebook (Created for this course; no real patient data)
- Reproduce section splitting, abbreviation expansion and negation detection on all twenty notes.
- Add rules for uncertainty ('probable', 'possible', 'query') and family history ('mother', 'father', 'family history of').
- Write five notes where your rules fail and explain why.
- Measure copy-forward between consecutive notes for each synthetic patient and flag notes that are more than 90% identical.
- Optional: run medspaCy's ConText on the same notes and compare.
Open the notebook in ColabView on GitHubWorked solution coming soon
Key takeaways
- Clinical notes mix sections, abbreviations, negation, uncertainty and copied text.
- Abbreviation lists must be local and reviewed; wrong expansions change meaning.
- Negation and context detection are essential before counting findings.
- Copy-forward makes old findings look current; handle duplication explicitly.
Terms used: EHR (electronic health record), Negation detection
Related: Named-entity recognition and clinical coding, Healthcare data types, DPN Check nerve conduction testing
FAQ
Why is clinical text hard for NLP?
It uses local abbreviations, incomplete sentences, many negated or uncertain findings, and text copied between notes, all of which can mislead general-purpose language tools.
What is NegEx?
A rule-based algorithm, published in 2001, that detects whether a finding in clinical text is negated, using cue words such as 'no' and 'denies' near the finding.
Can a large language model read clinical notes instead?
LLMs handle much of this variation well, but they also make errors with negation and context, and notes must only be processed in approved environments. The same evaluation principles apply.
What is copy-forward?
Copying text from a previous note into a new one. It saves time but can make old findings look current and inflates how often things appear in the record.
Should I remove stop words from clinical text?
Be careful: words like 'no', 'not' and 'without' are often in stop-word lists, and removing them destroys negation.
Sources
- Chapman WW, et al. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform 2001;34(5):301–310. doi:10.1006/jbin.2001.1029
- Harkema H, et al. ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports. J Biomed Inform 2009;42(5):839–851. doi:10.1016/j.jbi.2009.05.002
- Steinkamp JM, et al. Prevalence and sources of duplicate information in the electronic medical record. JAMA Netw Open 2022;5(9):e2233348. doi:10.1001/jamanetworkopen.2022.33348
- Eyre H, et al. Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python. AMIA Annual Symposium 2021. arXiv:2106.07799
- Spasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inform 2020;8(3):e17984. doi:10.2196/17984
Educational content only. This material teaches methods; it is not clinical advice. Code, models and examples here are not validated medical devices and must not be used to make decisions about real patients without proper clinical validation, governance and any required regulatory approval. Examples use synthetic or openly licensed de-identified data only. Never paste identifiable patient information into public AI tools. Mentions of models, tools or vendors are for identification, not endorsement.