Does pretraining BERT on hospital notes help it link medical terms?
BERT-style models beat older rule- and dictionary-based tools at mapping medical phrases to standard terms, and pretraining on closer-domain text helped, though the gains between BERT variants were small.
Source
Fine-Tuning Bidirectional Encoder Representations From Transformers (BERT)-Based Models on Large-Scale Electronic Health Record Notes: An Empirical Study
Study at a glance
- Design
- Computational / modelling — BioBERT further pretrained on EHR notes (EhrBERT), then all BERT variants fine-tuned as classifiers for entity normalisation and compared on three corpora over three random seeds with t tests.
- N
- No single N: pretraining used about 1.5 million EHR notes; evaluation used the MADE corpus (1089 EHR notes), the NCBI disease corpus (793 abstracts) and the CDR corpus (500 abstracts each for training, development and testing).
- Population
- Clinical EHR notes from one US medical centre and PubMed abstracts from two biomedical corpora
- Outcome
- Precision, recall and F1 for mapping entity mentions to vocabulary terms
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors took BioBERT and continued pretraining it on hospital electronic health record notes, creating EhrBERT in a smaller (500,000 notes) and a larger (1 million notes) version. They framed entity normalisation — mapping a phrase like 'fevers' to a vocabulary term like Fever — as a classification task, fine-tuned BERT, BioBERT and both EhrBERTs, and compared them with the MetaMap and DNorm tools on one clinical and two biomedical corpora. Each BERT model was run three times with different seeds and compared with t tests.
What they found
All BERT-based models beat MetaMap and DNorm; EhrBERT improved F1 over those baselines by 2.36%, 1.98% and 3.9% on the three corpora. On the clinical MADE corpus EhrBERT beat BioBERT, which beat plain BERT, and biomedical or clinical models were significantly better than general BERT on every corpus. Differences between BioBERT and EhrBERT were small and often not significant, and doubling pretraining notes from 500k to 1M only helped significantly on one corpus.
The limits
What it doesn't show
With only three runs per model, the t tests have little power, so 'not significant' differences between close models are weak evidence of no difference. The EHR notes come from a single medical centre and EhrBERT is continued from BioBERT rather than trained from scratch, so the data-size result may not generalise. The task is treated as single-label classification, so mentions that map to several terms cannot be handled, and the model ignores vocabulary structure such as synonyms. The authors also note a bias toward frequent terms.
Key terms
- Entity normalisation
- Mapping a text mention (e.g. 'dyspnea on exertion') to a specific concept ID in a controlled vocabulary.
- Pretraining
- Training a model on large unlabelled text with self-supervised objectives before fine-tuning it on a labelled task.
- Fine-tuning
- Continuing to train a pretrained model on a smaller labelled dataset for a specific downstream task.
- Domain shift
- A mismatch between the kind of text a model was trained on and the kind it is applied to, which can reduce performance.
- F1 score
- The harmonic mean of precision and recall, used to summarise classification accuracy.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was EhrBERT's starting point before pretraining on EHR notes?
Common questions
Why not just train a classifier from scratch on the labelled data?
Labelled clinical data are small; pretrained models bring general language knowledge learned from huge unlabelled text, which usually improves accuracy with limited labels.
Did more pretraining data always help?
The 1M-note model scored slightly higher everywhere, but the difference was only statistically significant on one of the three corpora, so the authors argue the smaller model is usually enough.
Why were scores lower on the clinical corpus?
The MADE corpus maps to a vocabulary with tens of times more classes than the disease corpora, making classification much harder.
More on Clinical NLP