Clinical NLP
Does pretraining BERT on hospital notes help it link medical terms?
Open access · cc by · source: Europe PMC
BERT-style models beat older rule- and dictionary-based tools at mapping medical phrases to standard terms, and pretraining on closer-domain text helped, though the gains between BERT variants were small.
Study at a glance
- Design
- Computational / modelling — BioBERT further pretrained on EHR notes (EhrBERT), then all BERT variants fine-tuned as classifiers for entity normalisation and compared on three corpora over three random seeds with t tests.
- N
- No single N: pretraining used about 1.5 million EHR notes; evaluation used the MADE corpus (1089 EHR notes), the NCBI disease corpus (793 abstracts) and the CDR corpus (500 abstracts each for training, development and testing).
- Population
- Clinical EHR notes from one US medical centre and PubMed abstracts from two biomedical corpora
- Outcome
- Precision, recall and F1 for mapping entity mentions to vocabulary terms
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
All BERT-based models beat MetaMap and DNorm; EhrBERT improved F1 over those baselines by 2.36%, 1.98% and 3.9% on the three corpora. On the clinical MADE corpus EhrBERT beat BioBERT, which beat plain BERT, and biomedical or clinical models were significantly better than general BERT on every corpus. Differences between BioBERT and EhrBERT were small and often not significant, and doubling pretraining notes from 500k to 1M only helped significantly on one corpus.
Methodology
The authors took BioBERT and continued pretraining it on hospital electronic health record notes, creating EhrBERT in a smaller (500,000 notes) and a larger (1 million notes) version. They framed entity normalisation — mapping a phrase like 'fevers' to a vocabulary term like Fever — as a classification task, fine-tuned BERT, BioBERT and both EhrBERTs, and compared them with the MetaMap and DNorm tools on one clinical and two biomedical corpora. Each BERT model was run three times with different seeds and compared with t tests.
Limitations
With only three runs per model, the t tests have little power, so 'not significant' differences between close models are weak evidence of no difference. The EHR notes come from a single medical centre and EhrBERT is continued from BioBERT rather than trained from scratch, so the data-size result may not generalise. The task is treated as single-label classification, so mentions that map to several terms cannot be handled, and the model ignores vocabulary structure such as synonyms. The authors also note a bias toward frequent terms.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
A small amount of labelled data plus fine-tuning remains a strong baseline.
On strict entity-level extraction, fine-tuned smaller models still beat prompted GPT: fine-tuned BioClinicalBERT beat GPT-4 on clinical NER in every setting, largely because GPT got entity boundaries slightly wrong; in seizure-note extraction the hand-labelled notes were the most important ingredient and just 70 labelled notes gave most of the final performance.
Evidence for the claim as stated.
Matching pretraining text to the domain reliably helps.
Pretraining on biomedical or clinical text beats general-domain pretraining of the same architecture: BioBERT beat plain BERT on every NER dataset and improved relation extraction F1 by 2.80 points on average, and biomedical/clinical BERTs were significantly better than general BERT on all three entity-normalisation corpora.
Evidence for the claim as stated.
Scale helps, but not always and not linearly.
More domain text and bigger models generally help, but with diminishing returns: BioBERT improved mostly up to 4.5 billion words; GatorTron's performance rose with size on four of five tasks and full-corpus training beat quarter-corpus; yet doubling EHR notes from 500k to 1M helped EhrBERT significantly on only one corpus.
Evidence for the claim as stated.
Whether clinical-note pretraining adds much beyond biomedical-literature pretraining is unclear: EhrBERT beat BioBERT on clinical notes but differences were small and often not significant, while GatorTron beat all existing clinical transformers — though it also differs in size and data.
Evidence for the claim as stated.
Open questions
Tensions this paper is part of
From concept pages' “where studies disagree.” Disagreement means the same question; scope means different assays, populations, or outcomes.
Whether clinical-note pretraining adds much beyond biomedical-literature pretraining is unclear: EhrBERT beat BioBERT on clinical notes but differences were small and often not significant, while GatorTron beat all existing clinical transformers — though it also differs in size and data.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Can deep learning use a whole hospital record to predict outcomes?
- Can language models find social needs hidden in doctors' notes?
- Can ChatGPT turn messy pathology reports into clean data?
- Can better prompts make GPT good at spotting medical terms?
- Can a local, private LLM pull symptoms out of medical notes?