Clinical NLP
Can an off-the-shelf language model spot postpartum bleeding in notes?
Open access · cc by · source: Europe PMC
Without any task-specific training, an open-source language model extracted bleeding-related facts from hospital notes well enough to find many postpartum hemorrhage cases that billing codes missed.
Study at a glance
- Design
- Computational / modelling — Zero-shot prompting of Flan-T5-XXL to extract 24 postpartum-hemorrhage concepts from discharge notes, evaluated against expert chart review and compared with regular expressions and ICD codes.
- N
- N=1175 · 1175 manually annotated discharge summaries formed the concept-extraction test set; 50 notes for prompt development, 300 notes for PPH phenotype validation, 100 later notes for temporal generalisation; full cohort 271,081 notes.
- Population
- Obstetric discharge summaries from Mass General Brigham hospitals, 1998-2015
- Outcome
- Binary F1 for concept extraction; positive predictive value of the PPH phenotype; subtype classification F1 versus ICD codes
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
The model reached a binary F1 of at least 0.75 on 21 of 23 yes/no concepts and beat regular expressions significantly on nine concepts, especially ones written in varied ways. The hemorrhage phenotype had a positive predictive value of 0.95, and over 47% of confirmed cases had no hemorrhage diagnosis code. For subtyping, the approach outperformed codes for three of four subtypes but not the rare coagulation subtype.
Methodology
The researchers asked the open-source Flan-T5-XXL model simple yes/no or extraction questions about 24 postpartum-hemorrhage concepts in obstetric discharge summaries, using only 50 notes to design the prompts. They tested extraction on 1175 expert-annotated notes and compared it with hand-written regular expressions. They then combined extracted blood loss and delivery type into a rule-based hemorrhage definition, and extracted concepts into rules for four cause-based subtypes, checking both against chart review and diagnosis codes.
Limitations
All data come from one health system and mostly from before 2015; on 100 newer notes, performance dropped for several concepts with few positive examples. The phenotype was validated only among notes the model flagged as positive, so its sensitivity (how many true cases it misses) is not directly measured. The model confused closely related procedures and was not compared with newer models such as GPT-4, and discharge notes miss hemorrhages occurring later after birth.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Codes undercount; the notes hold the missing cases.
Reading notes finds many cases that structured billing or diagnosis codes miss: a note-based postpartum haemorrhage phenotype found that over 47% of confirmed cases had no haemorrhage code, weak-supervision extraction found 504 hip-implant revisions versus 78 in billing codes, and models flagged adverse social needs in 93.8% of affected patients versus 2.0% by Z-codes.
Evidence for the claim as stated.
Prompting alone can get far on clear, well-defined variables.
Carefully prompted general LLMs can extract well-defined facts with high accuracy without task-specific training: GPT-3.5 averaged 0.89 accuracy on lung-cancer pathology attributes (vs 0.51 keyword search), zero-shot Flan-T5 reached F1 ≥0.75 on 21 of 23 haemorrhage concepts, and a locally run Llama 2 70b reached 100% sensitivity for cirrhosis.
Evidence for the claim as stated.
Errors are often confident reasoning or invention, not just misses.
LLM extractors fail in specific, recognisable ways: GPT-3.5 applied the wrong staging rule to correctly extracted facts and fabricated a plausible case for blank reports; the smallest Llama 2 hallucinated absent features; Flan-T5 confused closely related procedures and dropped on newer notes.
Evidence for the claim as stated.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Can deep learning use a whole hospital record to predict outcomes?
- Can language models find social needs hidden in doctors' notes?
- Can ChatGPT turn messy pathology reports into clean data?
- Can better prompts make GPT good at spotting medical terms?
- Does pretraining BERT on hospital notes help it link medical terms?