Language models
Does pre-training BERT on biomedical papers help it read biology?
Open access · cc by · source: Europe PMC
Letting a general language model keep learning from millions of biomedical abstracts and articles made it clearly better at biomedical text tasks than the same model trained only on Wikipedia and books.
Study at a glance
- Design
- Computational / modelling — BERT-base initialised from general-domain weights, further pre-trained on PubMed abstracts and PMC full texts, then fine-tuned and compared with BERT and prior best models on 15 biomedical datasets.
- N
- No single N: evaluation spans 15 benchmark datasets (9 NER, 3 relation extraction, 3 BioASQ question-answering batches), each with its own size.
- Population
- Biomedical text corpora and standard biomedical NLP benchmarks (e.g. NCBI Disease, BC2GM, CHEMPROT, GAD, BioASQ factoid).
- Outcome
- Entity-level precision/recall/F1 for NER, F1 for relation extraction, strict/lenient accuracy and mean reciprocal rank for QA.
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
BioBERT beat plain BERT on every NER dataset and beat the previous best models on six of nine, improving micro-averaged NER F1 by 0.62 points. It raised relation-extraction F1 by 2.80 points on average and question-answering mean reciprocal rank by 12.24 points over the prior best systems. Performance rose with more biomedical pre-training text (mostly improving up to 4.5 billion words) and with more pre-training steps.
Methodology
The authors took BERT, a transformer language model pre-trained on Wikipedia and books, and continued its masked-language-model pre-training on PubMed abstracts and PubMed Central full-text articles, creating BioBERT. They kept BERT's architecture and vocabulary unchanged. They then fine-tuned BERT and several BioBERT variants on three kinds of biomedical tasks (named entity recognition, relation extraction and question answering) and compared them with task-specific state-of-the-art models, also varying corpus size and pre-training steps.
Limitations
Gains over the prior state of the art are modest for NER (under one F1 point) and BioBERT did not win on every dataset; for LINNAEUS and Species-800 the train/test splits may differ from earlier work, so those comparisons are imperfect. Only the BERT-base size was used, and the original general-domain vocabulary was kept, so the paper does not test whether a larger model or a biomedical vocabulary would do better. About 30% of BioASQ factoid questions were dropped as unanswerable, so QA results cover only extractive questions. No statistical significance tests or run-to-run variance are reported for most comparisons.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Whether custom neural architectures still matter once pretrained transformers exist: CLSTM beat GRAM-CNN, but general BERT matched or beat it on genes and cross-corpus transfer, and BioBERT beat plain BERT on every NER dataset.
Same question, contrary or null result.
Matching pretraining text to the domain reliably helps.
Pretraining on biomedical or clinical text beats general-domain pretraining of the same architecture: BioBERT beat plain BERT on every NER dataset and improved relation extraction F1 by 2.80 points on average, and biomedical/clinical BERTs were significantly better than general BERT on all three entity-normalisation corpora.
Evidence for the claim as stated.
Scale helps, but not always and not linearly.
More domain text and bigger models generally help, but with diminishing returns: BioBERT improved mostly up to 4.5 billion words; GatorTron's performance rose with size on four of five tasks and full-corpus training beat quarter-corpus; yet doubling EHR notes from 500k to 1M helped EhrBERT significantly on only one corpus.
Evidence for the claim as stated.
Expect big gains on reasoning-like tasks, modest ones on tagging.
Gains are largest on complex tasks and small on simpler extraction: GatorTron improved most on inference and question answering, while BioBERT's NER gain over prior best systems was 0.62 F1 points.
Evidence for the claim as stated.
Open questions
Tensions this paper is part of
From concept pages' “where studies disagree.” Disagreement means the same question; scope means different assays, populations, or outcomes.
Whether custom neural architectures still matter once pretrained transformers exist: CLSTM beat GRAM-CNN, but general BERT matched or beat it on genes and cross-corpus transfer, and BioBERT beat plain BERT on every NER dataset.
Discoveries this paper informs or conflicts with
- Prompted GPT models approach fine-tuned clinical models on loose matches but slip on exact boundaries and rules
This paper informs this development.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Do bigger language models understand clinical notes better?
- Can a language model learn useful features from protein sequences?
- Can fine-tuned language models draft replies to patient messages?
- Can a fine-tuned open LLM assign hospital billing codes from notes?
- Can a general chatbot-style model be taught to spot biomedical terms?