Concept
Clinical information extraction from notes
10 studies1 discoveryEvidence last moved Sep 27, 2026
Clinical information extraction turns free-text notes (pathology reports, discharge summaries, clinic letters) into structured variables such as cancer stage, seizure frequency or a symptom being present. The evidence here comes from retrospective computational studies at single health systems or public archives that compare prompted large language models, fine-tuned transformers, weak supervision and rule or code-based baselines against expert chart review.
Much of what happens to patients is only written in notes, not in billing codes, so extraction decides what research and surveillance can see. Students often assume a big chatbot simply 'reads the chart'; these studies show where it works, where it invents answers, and why a smaller fine-tuned model can still win.
Studies
10
Findings
5
13 supporting · 0 challenging · 2 qualifying citations
Open tensions
2
Latest change
Concept page published
Clinical information extraction from notes
Currently
What we know
- Codes undercount; the notes hold the missing cases.
- Prompting alone can get far on clear, well-defined variables.
- A small amount of labelled data plus fine-tuning remains a strong baseline.
- Good retrieval is part of the extraction system, not an afterthought.
- Errors are often confident reasoning or invention, not just misses.
Largest unresolved question
Whether prompted LLMs match fine-tuned models depends on the task and the metric: GPT-3.5 beat a fine-tuned BERT on pathology staging accuracy, but GPT-4 lost to BioClinicalBERT on exact-match clinical NER, and the NER authors argue exact matching undervalues clinically sensible LLM answers.
Common misconceptions
If an LLM gets 89% accuracy on one hospital's reports, it will do the same elsewhere.
Every study here is from one health system or public archive; the haemorrhage model dropped on newer notes and the social-determinants models dropped on another hospital's ICU notes.
When the note says nothing, the model will answer 'unknown'.
GPT-3.5 often invented a consistent fake case for blank reports, and the smallest Llama 2 hallucinated features, so outputs need checking.
Better extraction automatically means better patient care.
These are retrospective accuracy studies; none tested whether using the extracted data improved outcomes, and the EHR prediction study explicitly calls for prospective trials.
Related
Claim ledger
What the evidence shows
Drawn from 10 studies in this library. Mix labels say which citation roles are present; they are not a strength score. Supports means evidence for a finding; Challenges means evidence against a stated position; Qualifies marks scope.
Codes undercount; the notes hold the missing cases.
Reading notes finds many cases that structured billing or diagnosis codes miss: a note-based postpartum haemorrhage phenotype found that over 47% of confirmed cases had no haemorrhage code, weak-supervision extraction found 504 hip-implant revisions versus 78 in billing codes, and models flagged adverse social needs in 93.8% of affected patients versus 2.0% by Z-codes.
- Can an off-the-shelf language model spot postpartum bleeding in notes?
- Can AI read doctors' notes to track failing hip implants?
- Can language models find social needs hidden in doctors' notes?
Study Role Design N Population Outcome Can an off-the-shelf language model spot postpartum bleeding in notes? Supports Computational / modellingZero-shot prompting of Flan-T5-XXL to extract 24 postpartum-hemorrhage concepts from discharge notes, evaluated against expert chart review and compared with regular expressions and ICD codes. N=1175 · 1175 manually annotated discharge summaries formed the concept-extraction test set; 50 notes for prompt development, 300 notes for PPH phenotype validation, 100 later notes for temporal generalisation; full cohort 271,081 notes. Obstetric discharge summaries from Mass General Brigham hospitals, 1998-2015 Binary F1 for concept extraction; positive predictive value of the PPH phenotype; subtype classification F1 versus ICD codes Can AI read doctors' notes to track failing hip implants? Supports Computational / modellingSnorkel data-programming: 50 labeling functions generate probabilistic labels for a bidirectional LSTM relation extractor, compared with rule majority vote and an LSTM trained on 150 hand-labelled documents; outputs then fed to survival and count models N=6583 · 6583 hip replacement patients in the EHR cohort; implant-system analyses restricted to 2704 single-implant patients; model test sets of a few hundred annotated notes Patients with hip replacement surgery at Stanford Health Care, 1995-2014, and their clinical notes Precision, recall and F1 of extracting implant complications and pain; revision events and complication risk by implant system Can language models find social needs hidden in doctors' notes? Supports Computational / modellingSentence-level multilabel classification on annotated oncology notes (60/20/20 split) plus two out-of-domain test sets; synthetic-data augmentation, data-quantity ablations, ChatGPT comparison on synthetic sentences, and demographic-injection bias tests. N=800 · 800 annotated radiotherapy clinic notes from 770 patients for development; out-of-domain tests used 200 immunotherapy notes and 200 MIMIC-III notes; ChatGPT comparison used 480 validated synthetic sentences. Clinic notes of cancer patients at Boston hospitals, plus ICU notes from MIMIC-III Macro-F1 for detecting any and adverse mentions of six social determinants; discrepancy rates after injecting demographic descriptors; patient-level detection versus structured Z-codes Prompting alone can get far on clear, well-defined variables.
Carefully prompted general LLMs can extract well-defined facts with high accuracy without task-specific training: GPT-3.5 averaged 0.89 accuracy on lung-cancer pathology attributes (vs 0.51 keyword search), zero-shot Flan-T5 reached F1 ≥0.75 on 21 of 23 haemorrhage concepts, and a locally run Llama 2 70b reached 100% sensitivity for cirrhosis.
- Can a local, private LLM pull symptoms out of medical notes?— Sensitivity for a vaguer feature, confusion, was only 76%; rare features rest on few positives.
Study Role Design N Population Outcome Can ChatGPT turn messy pathology reports into clean data? Supports Computational / modellingPrompt engineering on a small training set of public pathology reports, then held-out evaluation of GPT-3.5-turbo against expert-curated labels, a keyword-search baseline and a fine-tuned BERT NER model N=774 · 774 valid TCGA pathology reports in the test set; 78 CDSA reports used for prompt development; a separate 191 osteosarcoma reports for an extension De-identified lung cancer pathology reports from public archives (CDSA, TCGA) Accuracy and coverage for pathologic T stage, N stage, overall stage and histology type Can an off-the-shelf language model spot postpartum bleeding in notes? Supports Computational / modellingZero-shot prompting of Flan-T5-XXL to extract 24 postpartum-hemorrhage concepts from discharge notes, evaluated against expert chart review and compared with regular expressions and ICD codes. N=1175 · 1175 manually annotated discharge summaries formed the concept-extraction test set; 50 notes for prompt development, 300 notes for PPH phenotype validation, 100 later notes for temporal generalisation; full cohort 271,081 notes. Obstetric discharge summaries from Mass General Brigham hospitals, 1998-2015 Binary F1 for concept extraction; positive predictive value of the PPH phenotype; subtype classification F1 versus ICD codes Can a local, private LLM pull symptoms out of medical notes? Supports Computational / modellingZero-shot and prompted information extraction by Llama 2 (7b, 13b, 70b) run locally, scored against a three-rater consensus ground truth N=500 · 500 admission histories from the MIMIC-IV database, each labelled for five features Emergency/ICU patient histories from MIMIC-IV (Beth Israel Deaconess Medical Center) Sensitivity, specificity, precision, NPV and accuracy for detecting liver cirrhosis, ascites, abdominal pain, shortness of breath and confusion A small amount of labelled data plus fine-tuning remains a strong baseline.
On strict entity-level extraction, fine-tuned smaller models still beat prompted GPT: fine-tuned BioClinicalBERT beat GPT-4 on clinical NER in every setting, largely because GPT got entity boundaries slightly wrong; in seizure-note extraction the hand-labelled notes were the most important ingredient and just 70 labelled notes gave most of the final performance.
- Can better prompts make GPT good at spotting medical terms?
- Can language models read seizure counts from doctors' notes?
- Does pretraining BERT on hospital notes help it link medical terms?
Study Role Design N Population Outcome Can better prompts make GPT good at spotting medical terms? Supports Computational / modellingIncremental prompt ablation for GPT-3.5 and GPT-4 at temperature 0 (zero-, 1- and 5-shot), compared with fine-tuned BioClinicalBERT and a CRF on held-out test sets. Two corpora: 163 synthetic discharge summaries (MTSamples) and 91 vaccine adverse-event reports (VAERS), each split into train, validation and test. Synthetic clinical notes and public VAERS safety reports Precision, recall and F1 for clinical entity extraction under exact and relaxed matching Can language models read seizure counts from doctors' notes? Supports Computational / modellingThree pretrained transformers fine-tuned through a multi-step pipeline (domain masked-language modelling, public QA data, then 700 annotated paragraphs), tested on 300 held-out paragraphs with ablations. N=1000 · 1000 triple-annotated note paragraphs (700 train, 300 test), drawn from nearly 79 000 epilepsy progress notes at one health system. Epilepsy clinic progress notes written by 8 specialists at the University of Pennsylvania Health System, 2015-2018 Accuracy for classifying seizure freedom; F1 text overlap for extracting seizure frequency and date of last seizure; comparison with human annotators Does pretraining BERT on hospital notes help it link medical terms? Supports Computational / modellingBioBERT further pretrained on EHR notes (EhrBERT), then all BERT variants fine-tuned as classifiers for entity normalisation and compared on three corpora over three random seeds with t tests. No single N: pretraining used about 1.5 million EHR notes; evaluation used the MADE corpus (1089 EHR notes), the NCBI disease corpus (793 abstracts) and the CDR corpus (500 abstracts each for training, development and testing). Clinical EHR notes from one US medical centre and PubMed abstracts from two biomedical corpora Precision, recall and F1 for mapping entity mentions to vocabulary terms Good retrieval is part of the extraction system, not an afterthought.
How text is fed to the model matters: retrieving entity-centred 150-word windows (CLEAR) beat chunk-embedding and full-note input for all six LLMs tested while using 71–81% fewer tokens, and removing the entity-recognition step dropped GPT-4 F1 from 0.97 to 0.86.
- Can smarter retrieval help LLMs read long clinical notes?— Single hospital; the margin over a ten-chunk embedding baseline was small for some models.
Errors are often confident reasoning or invention, not just misses.
LLM extractors fail in specific, recognisable ways: GPT-3.5 applied the wrong staging rule to correctly extracted facts and fabricated a plausible case for blank reports; the smallest Llama 2 hallucinated absent features; Flan-T5 confused closely related procedures and dropped on newer notes.
- Can ChatGPT turn messy pathology reports into clean data?
- Can a local, private LLM pull symptoms out of medical notes?
- Can an off-the-shelf language model spot postpartum bleeding in notes?
Study Role Design N Population Outcome Can ChatGPT turn messy pathology reports into clean data? Supports Computational / modellingPrompt engineering on a small training set of public pathology reports, then held-out evaluation of GPT-3.5-turbo against expert-curated labels, a keyword-search baseline and a fine-tuned BERT NER model N=774 · 774 valid TCGA pathology reports in the test set; 78 CDSA reports used for prompt development; a separate 191 osteosarcoma reports for an extension De-identified lung cancer pathology reports from public archives (CDSA, TCGA) Accuracy and coverage for pathologic T stage, N stage, overall stage and histology type Can a local, private LLM pull symptoms out of medical notes? Supports Computational / modellingZero-shot and prompted information extraction by Llama 2 (7b, 13b, 70b) run locally, scored against a three-rater consensus ground truth N=500 · 500 admission histories from the MIMIC-IV database, each labelled for five features Emergency/ICU patient histories from MIMIC-IV (Beth Israel Deaconess Medical Center) Sensitivity, specificity, precision, NPV and accuracy for detecting liver cirrhosis, ascites, abdominal pain, shortness of breath and confusion Can an off-the-shelf language model spot postpartum bleeding in notes? Supports Computational / modellingZero-shot prompting of Flan-T5-XXL to extract 24 postpartum-hemorrhage concepts from discharge notes, evaluated against expert chart review and compared with regular expressions and ICD codes. N=1175 · 1175 manually annotated discharge summaries formed the concept-extraction test set; 50 notes for prompt development, 300 notes for PPH phenotype validation, 100 later notes for temporal generalisation; full cohort 271,081 notes. Obstetric discharge summaries from Mass General Brigham hospitals, 1998-2015 Binary F1 for concept extraction; positive predictive value of the PPH phenotype; subtype classification F1 versus ICD codes
Debates
Tensions and limits
Some items are genuine disagreements on the same question. Others mark different assays, populations, or outcomes.
Whether prompted LLMs match fine-tuned models depends on the task and the metric: GPT-3.5 beat a fine-tuned BERT on pathology staging accuracy, but GPT-4 lost to BioClinicalBERT on exact-match clinical NER, and the NER authors argue exact matching undervalues clinically sensible LLM answers.
Evidence for
Evidence against
PaperFren reads this as a limit on how far one study travels — different assays, populations, or outcomes — not a forced fight between papers.
Chain-of-thought style prompting helped GPT-3.5 on pathology (asking for tumour size first raised accuracy) but explicit chain-of-thought did not help and sometimes hurt Llama 2 on admission histories.
Evidence against
Qualified studies asking the same question reach different answers. The disagreement is listed, not scored.
Timeline
How understanding moved
Study years are when the paper was published. Evidence edits are dated changes to this page's claims. Explanations are when PaperFren added a Discovery — not a claim that the science happened that day.
2026
- Prompted GPT models approach fine-tuned clinical models on loose matches but slip on exact boundaries and rules
Concept page published
Clinical information extraction from notes
Change log
What changed
Dated edits to this page's evidence: studies added or removed from a claim, claims added or withdrawn, and new explanations tagged here. Rewordings are not listed.
- Concept page published
Papers
10 studies in this library bear on Clinical information extraction from notes, ordered by citations.
- Can deep learning use a whole hospital record to predict outcomes?
Neural networks reading a patient's entire raw hospital record predicted death, readmission and long stays more accurately than standard clinical scores.
- Can language models find social needs hidden in doctors' notes?
Fine-tuned language models found patients' social problems such as housing or transport trouble in clinic notes far more often than billing codes did, and were less swayed by race or gender wording than ChatGPT.
- Can ChatGPT turn messy pathology reports into clean data?
With carefully engineered prompts, ChatGPT extracted cancer stage and tumour type from pathology reports with about 89% average accuracy, beating older NLP methods, but it misapplied staging rules and invented answers for blank reports.
- Can better prompts make GPT good at spotting medical terms?
Carefully built prompts made GPT-4 nearly as good as a fine-tuned clinical model at loosely matching medical terms, but it still fell clearly short on exact boundaries.
- Does pretraining BERT on hospital notes help it link medical terms?
BERT-style models beat older rule- and dictionary-based tools at mapping medical phrases to standard terms, and pretraining on closer-domain text helped, though the gains between BERT variants were small.
- Can a local, private LLM pull symptoms out of medical notes?
An open language model running inside the hospital could reliably tag symptoms in doctors' notes, with the largest version doing best and struggling most with vaguely described symptoms.
- Can an off-the-shelf language model spot postpartum bleeding in notes?
Without any task-specific training, an open-source language model extracted bleeding-related facts from hospital notes well enough to find many postpartum hemorrhage cases that billing codes missed.
- Can language models read seizure counts from doctors' notes?
Fine-tuned language models pulled seizure frequency and last-seizure dates from clinic notes about as well as human annotators, and only needed a small labelled set.
Show 2 more studiesShow fewer studies
- Can smarter retrieval help LLMs read long clinical notes?
Retrieving only the passages around clinically relevant terms let language models extract patient information slightly more accurately while using far less text and time than feeding whole notes or embedding-based chunks.
- Can AI read doctors' notes to track failing hip implants?
Training a text model on labels produced by simple hand-written rules, rather than costly hand annotation, let it find far more hip-implant problems in doctors' notes than billing codes reveal.
Compare studies
Select 2–10 studies. Design and N are labels, not a ranking.
Nothing selected yet.
Questions
What is still open
Whether prompted LLMs match fine-tuned models depends on the task and the metric: GPT-3.5 beat a fine-tuned BERT on pathology staging accuracy, but GPT-4 lost to BioClinicalBERT on exact-match clinical NER, and the NER authors argue exact matching undervalues clinically sensible LLM answers.
Chain-of-thought style prompting helped GPT-3.5 on pathology (asking for tumour size first raised accuracy) but explicit chain-of-thought did not help and sometimes hurt Llama 2 on admission histories.
Ask PaperFren about Clinical information extraction from notes
Study this conceptflashcards and short-answer questions
Why might a hospital prefer a fine-tuned BERT model over GPT-4 for extracting medication names from notes?
In a head-to-head on clinical NER, fine-tuned BioClinicalBERT beat GPT-4 in every setting, mainly because GPT-4 misplaced entity boundaries under exact matching. The seizure-note study also showed small amounts of labelled data (about 70 notes) gave most of the performance. Privacy matters too: real notes could not be sent to the GPT API, which is why the Llama 2 study ran models locally. The trade-off is that fine-tuning needs labelled examples.
What do note-based extraction studies suggest about relying on billing codes for research?
Several single-system studies found that codes miss many cases: nearly half of confirmed postpartum haemorrhages lacked a code, note mining found hundreds more hip revisions than billing codes, and Z-codes flagged only 2% of patients with adverse social needs versus 93.8% by the models. Codes therefore undercount, which can bias comparisons. These are retrospective and single-site, so the exact gap will vary.