Do bigger language models understand clinical notes better?
Scaling a clinical language model from hundreds of millions to 8.9 billion parameters, trained on tens of billions of words of real patient notes, improved performance on a range of medical language tasks, most for the hardest ones.
Source
A large language model for electronic health records
Study at a glance
- Design
- Computational / modelling — Pretrained GatorTron transformer models (345M, 3.9B, 8.9B parameters) on >90 billion words of clinical and biomedical text, then fine-tuned and compared with BioBERT, ClinicalBERT and BioMegatron on six public benchmarks
- N
- No single N: pretraining corpus from 290,482,002 clinical notes plus PubMed, Wikipedia and MIMIC-III; evaluation on six public benchmark datasets of differing sizes
- Population
- De-identified clinical notes from UF Health (2011-2021) plus public text corpora; public clinical NLP benchmarks
- Outcome
- F1 for concept and relation extraction, Pearson correlation for semantic similarity, accuracy for inference, exact match/F1 for question answering
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The team built GatorTron, a BERT-style transformer trained from scratch with masked-word and sentence-order prediction on over 90 billion words, mostly de-identified University of Florida Health clinical notes plus PubMed, Wikipedia and MIMIC-III. They trained three sizes (345 million, 3.9 billion and 8.9 billion parameters) and a base model on a quarter of the data. Each was fine-tuned on five tasks: clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference and medical question answering, and compared with BioBERT, ClinicalBERT and BioMegatron.
What they found
All GatorTron models beat the existing biomedical and clinical transformers on all five tasks. The largest model gave the best results on most tasks, for example 0.9020 accuracy on medical inference, 9.6% above BioBERT, and performance rose steadily with size for four of five tasks; similarity peaked at the medium model (0.8903 vs 0.8896 for large). Training the base model on the full corpus beat training on a quarter of it on almost every task. Gains were largest on complex tasks like inference and question answering and moderate on simpler extraction tasks.
The limits
What it doesn't show
Model size, amount of data and training corpus all differ between GatorTron and the comparison models, so the study cannot cleanly say which factor drives the gains. The clinical notes come from one health system, and the fine-tuning benchmarks are fairly small public datasets, so real-world transfer to other hospitals is not shown. Absolute scores on question answering about medications remained low (exact match about 0.32), and the authors note difficulty with long passages. The paper reports no variance across runs or significance tests, and the model needed very large compute (992 GPUs for about six days).
Key terms
- Masked language modelling
- A self-supervised pretraining task in which some words are hidden and the model learns to predict them from context.
- Fine-tuning
- Further training a pretrained model on a small labelled dataset for a specific task.
- Scaling
- Increasing parameters or training data to see whether performance keeps improving.
- Natural language inference
- Deciding whether a hypothesis sentence follows from, contradicts, or is neutral to a premise sentence.
- De-identification
- Removing protected health information such as names from clinical text before use.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was the largest GatorTron model's parameter count?
Common questions
Does this show bigger is always better?
Not quite; size helped on four of five tasks, but for semantic similarity the medium model was marginally best, and simple extraction tasks improved only modestly.
Why train on clinical notes rather than PubMed?
Clinical notes use different language (abbreviations, shorthand); GatorTron beat a similar-sized PubMed/Wikipedia model, suggesting in-domain data helps, though data size also differed.
Is GatorTron a chatbot like GPT?
No, it is an encoder (BERT-style) model used for understanding and extraction tasks after fine-tuning, not for free text generation.
More on Language models