Language models
Do bigger language models understand clinical notes better?
Open access · cc by · source: Europe PMC
Scaling a clinical language model from hundreds of millions to 8.9 billion parameters, trained on tens of billions of words of real patient notes, improved performance on a range of medical language tasks, most for the hardest ones.
Study at a glance
- Design
- Computational / modelling — Pretrained GatorTron transformer models (345M, 3.9B, 8.9B parameters) on >90 billion words of clinical and biomedical text, then fine-tuned and compared with BioBERT, ClinicalBERT and BioMegatron on six public benchmarks
- N
- No single N: pretraining corpus from 290,482,002 clinical notes plus PubMed, Wikipedia and MIMIC-III; evaluation on six public benchmark datasets of differing sizes
- Population
- De-identified clinical notes from UF Health (2011-2021) plus public text corpora; public clinical NLP benchmarks
- Outcome
- F1 for concept and relation extraction, Pearson correlation for semantic similarity, accuracy for inference, exact match/F1 for question answering
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
All GatorTron models beat the existing biomedical and clinical transformers on all five tasks. The largest model gave the best results on most tasks, for example 0.9020 accuracy on medical inference, 9.6% above BioBERT, and performance rose steadily with size for four of five tasks; similarity peaked at the medium model (0.8903 vs 0.8896 for large). Training the base model on the full corpus beat training on a quarter of it on almost every task. Gains were largest on complex tasks like inference and question answering and moderate on simpler extraction tasks.
Methodology
The team built GatorTron, a BERT-style transformer trained from scratch with masked-word and sentence-order prediction on over 90 billion words, mostly de-identified University of Florida Health clinical notes plus PubMed, Wikipedia and MIMIC-III. They trained three sizes (345 million, 3.9 billion and 8.9 billion parameters) and a base model on a quarter of the data. Each was fine-tuned on five tasks: clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference and medical question answering, and compared with BioBERT, ClinicalBERT and BioMegatron.
Limitations
Model size, amount of data and training corpus all differ between GatorTron and the comparison models, so the study cannot cleanly say which factor drives the gains. The clinical notes come from one health system, and the fine-tuning benchmarks are fairly small public datasets, so real-world transfer to other hospitals is not shown. Absolute scores on question answering about medications remained low (exact match about 0.32), and the authors note difficulty with long passages. The paper reports no variance across runs or significance tests, and the model needed very large compute (992 GPUs for about six days).
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Scale helps, but not always and not linearly.
More domain text and bigger models generally help, but with diminishing returns: BioBERT improved mostly up to 4.5 billion words; GatorTron's performance rose with size on four of five tasks and full-corpus training beat quarter-corpus; yet doubling EHR notes from 500k to 1M helped EhrBERT significantly on only one corpus.
Evidence for the claim as stated.
Expect big gains on reasoning-like tasks, modest ones on tagging.
Gains are largest on complex tasks and small on simpler extraction: GatorTron improved most on inference and question answering, while BioBERT's NER gain over prior best systems was 0.62 F1 points.
Evidence for the claim as stated.
Whether clinical-note pretraining adds much beyond biomedical-literature pretraining is unclear: EhrBERT beat BioBERT on clinical notes but differences were small and often not significant, while GatorTron beat all existing clinical transformers — though it also differs in size and data.
Evidence for the claim as stated.
Open questions
Tensions this paper is part of
From concept pages' “where studies disagree.” Disagreement means the same question; scope means different assays, populations, or outcomes.
Whether clinical-note pretraining adds much beyond biomedical-literature pretraining is unclear: EhrBERT beat BioBERT on clinical notes but differences were small and often not significant, while GatorTron beat all existing clinical transformers — though it also differs in size and data.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Does pre-training BERT on biomedical papers help it read biology?
- Can a language model learn useful features from protein sequences?
- Can fine-tuned language models draft replies to patient messages?
- Can a fine-tuned open LLM assign hospital billing codes from notes?
- Can a general chatbot-style model be taught to spot biomedical terms?