Can an off-the-shelf language model spot postpartum bleeding in notes?
Without any task-specific training, an open-source language model extracted bleeding-related facts from hospital notes well enough to find many postpartum hemorrhage cases that billing codes missed.
Source
Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models
Study at a glance
- Design
- Computational / modelling — Zero-shot prompting of Flan-T5-XXL to extract 24 postpartum-hemorrhage concepts from discharge notes, evaluated against expert chart review and compared with regular expressions and ICD codes.
- N
- N=1175 · 1175 manually annotated discharge summaries formed the concept-extraction test set; 50 notes for prompt development, 300 notes for PPH phenotype validation, 100 later notes for temporal generalisation; full cohort 271,081 notes.
- Population
- Obstetric discharge summaries from Mass General Brigham hospitals, 1998-2015
- Outcome
- Binary F1 for concept extraction; positive predictive value of the PPH phenotype; subtype classification F1 versus ICD codes
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The researchers asked the open-source Flan-T5-XXL model simple yes/no or extraction questions about 24 postpartum-hemorrhage concepts in obstetric discharge summaries, using only 50 notes to design the prompts. They tested extraction on 1175 expert-annotated notes and compared it with hand-written regular expressions. They then combined extracted blood loss and delivery type into a rule-based hemorrhage definition, and extracted concepts into rules for four cause-based subtypes, checking both against chart review and diagnosis codes.
What they found
The model reached a binary F1 of at least 0.75 on 21 of 23 yes/no concepts and beat regular expressions significantly on nine concepts, especially ones written in varied ways. The hemorrhage phenotype had a positive predictive value of 0.95, and over 47% of confirmed cases had no hemorrhage diagnosis code. For subtyping, the approach outperformed codes for three of four subtypes but not the rare coagulation subtype.
The limits
What it doesn't show
All data come from one health system and mostly from before 2015; on 100 newer notes, performance dropped for several concepts with few positive examples. The phenotype was validated only among notes the model flagged as positive, so its sensitivity (how many true cases it misses) is not directly measured. The model confused closely related procedures and was not compared with newer models such as GPT-4, and discharge notes miss hemorrhages occurring later after birth.
Key terms
- Zero-shot learning
- Getting a model to perform a new task from instructions alone, without training it on labelled examples of that task.
- Phenotyping
- Identifying which patients have a condition, or a subtype of it, from health record data.
- Binary F1 score
- The harmonic mean of precision and recall for the positive class, useful when positives are rare.
- Positive predictive value
- The share of cases flagged positive by a method that are truly positive.
- Regular expression
- A hand-written text pattern used to match specific words or formats in documents.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
How was Flan-T5 adapted to the hemorrhage task?
Common questions
Why extract concepts first rather than ask the model 'does this patient have PPH?'
The extract-then-phenotype design keeps each step checkable and lets the definition be updated, for example when guidelines change the blood-loss threshold.
Why use an open-source model instead of ChatGPT?
It can run locally behind a hospital firewall, reducing privacy, security and cost concerns.
Where did the model make mistakes?
It confused related concepts, such as reading membrane rupture as uterine rupture, and missed unusual abbreviations or misspellings.
More on Clinical NLP