Can language models read seizure counts from doctors' notes?
Fine-tuned language models pulled seizure frequency and last-seizure dates from clinic notes about as well as human annotators, and only needed a small labelled set.
Source
Extracting seizure frequency from epilepsy clinic notes: a machine reading approach to natural language processing
Study at a glance
- Design
- Computational / modelling — Three pretrained transformers fine-tuned through a multi-step pipeline (domain masked-language modelling, public QA data, then 700 annotated paragraphs), tested on 300 held-out paragraphs with ablations.
- N
- N=1000 · 1000 triple-annotated note paragraphs (700 train, 300 test), drawn from nearly 79 000 epilepsy progress notes at one health system.
- Population
- Epilepsy clinic progress notes written by 8 specialists at the University of Pennsylvania Health System, 2015-2018
- Outcome
- Accuracy for classifying seizure freedom; F1 text overlap for extracting seizure frequency and date of last seizure; comparison with human annotators
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The team framed information extraction as question answering: given a paragraph from an epilepsy clinic note, answer whether the patient had recent seizures, how often seizures occur, and when the last one was. Fifteen annotators labelled 1000 paragraphs, each read by three people. They fine-tuned BERT, Bio_ClinicalBERT and RoBERTa with a pipeline of unlabelled clinical text, public question-answering data and 700 labelled paragraphs, then tested on 300 held-out paragraphs, running each setup with 5 random seeds and removing pipeline steps one at a time.
What they found
For the yes/no seizure-freedom question, Bio_ClinicalBERT reached a median accuracy of 0.837, slightly but significantly below humans. For extracting frequency and last-seizure text, RoBERTa reached median F1 scores of 0.845 and 0.834, statistically indistinguishable from humans. Ablations showed the hand-labelled notes were the most important ingredient, domain pretraining sometimes helped, and the intermediate public QA data did not. Training on just 70 labelled notes already gave most of the final performance.
The limits
What it doesn't show
All notes came from one health system and one department, so performance at other hospitals or specialties is untested. Human performance was measured against a ground truth built from those same humans' labels, which flatters the humans. GPU limits ruled out larger long-document models, the one-year definition of 'recent' was a pragmatic choice, and all seizure types were lumped together. The models also struggled with temporal reasoning, such as working out that a seizure in June implies no seizures since.
Key terms
- Question answering (extractive)
- A task where the model highlights the span of text in a passage that answers a question.
- Fine-tuning
- Continuing to train a pretrained model on a smaller, task-specific data set.
- Masked language modelling
- Unsupervised training where the model predicts hidden words, used here to adapt the model to clinical writing.
- F1 score (text overlap)
- The harmonic mean of precision and recall over the words shared by the predicted and correct answer spans.
- Ablation study
- Removing one component at a time to measure how much it contributes.
- Cohen's kappa
- Agreement between raters corrected for agreement expected by chance.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
How was seizure information extraction framed in this study?
Common questions
Why frame extraction as question answering rather than keyword rules?
Seizure frequency is written in many different ways; a QA model learns to find the answer span from examples instead of relying on hand-built rules and dictionaries.
Was the classification question harder than extraction?
Yes. Deciding seizure freedom needs reasoning about dates relative to when the note was written, whereas extraction often just copies a phrase; the models matched humans on extraction but not classification.
Does this show the model would work at my hospital?
Not directly. It was trained and tested on one centre's neurology notes; the authors expect a small amount of local fine-tuning would be needed.
More on Clinical NLP