Clinical NLP
Can language models read seizure counts from doctors' notes?
Open access · cc by · source: Europe PMC
Fine-tuned language models pulled seizure frequency and last-seizure dates from clinic notes about as well as human annotators, and only needed a small labelled set.
Study at a glance
- Design
- Computational / modelling — Three pretrained transformers fine-tuned through a multi-step pipeline (domain masked-language modelling, public QA data, then 700 annotated paragraphs), tested on 300 held-out paragraphs with ablations.
- N
- N=1000 · 1000 triple-annotated note paragraphs (700 train, 300 test), drawn from nearly 79 000 epilepsy progress notes at one health system.
- Population
- Epilepsy clinic progress notes written by 8 specialists at the University of Pennsylvania Health System, 2015-2018
- Outcome
- Accuracy for classifying seizure freedom; F1 text overlap for extracting seizure frequency and date of last seizure; comparison with human annotators
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
For the yes/no seizure-freedom question, Bio_ClinicalBERT reached a median accuracy of 0.837, slightly but significantly below humans. For extracting frequency and last-seizure text, RoBERTa reached median F1 scores of 0.845 and 0.834, statistically indistinguishable from humans. Ablations showed the hand-labelled notes were the most important ingredient, domain pretraining sometimes helped, and the intermediate public QA data did not. Training on just 70 labelled notes already gave most of the final performance.
Methodology
The team framed information extraction as question answering: given a paragraph from an epilepsy clinic note, answer whether the patient had recent seizures, how often seizures occur, and when the last one was. Fifteen annotators labelled 1000 paragraphs, each read by three people. They fine-tuned BERT, Bio_ClinicalBERT and RoBERTa with a pipeline of unlabelled clinical text, public question-answering data and 700 labelled paragraphs, then tested on 300 held-out paragraphs, running each setup with 5 random seeds and removing pipeline steps one at a time.
Limitations
All notes came from one health system and one department, so performance at other hospitals or specialties is untested. Human performance was measured against a ground truth built from those same humans' labels, which flatters the humans. GPU limits ruled out larger long-document models, the one-year definition of 'recent' was a pragmatic choice, and all seizure types were lumped together. The models also struggled with temporal reasoning, such as working out that a seizure in June implies no seizures since.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
A small amount of labelled data plus fine-tuning remains a strong baseline.
On strict entity-level extraction, fine-tuned smaller models still beat prompted GPT: fine-tuned BioClinicalBERT beat GPT-4 on clinical NER in every setting, largely because GPT got entity boundaries slightly wrong; in seizure-note extraction the hand-labelled notes were the most important ingredient and just 70 labelled notes gave most of the final performance.
Evidence for the claim as stated.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Can deep learning use a whole hospital record to predict outcomes?
- Can language models find social needs hidden in doctors' notes?
- Can ChatGPT turn messy pathology reports into clean data?
- Can better prompts make GPT good at spotting medical terms?
- Does pretraining BERT on hospital notes help it link medical terms?