Does pre-training BERT on biomedical papers help it read biology?
Letting a general language model keep learning from millions of biomedical abstracts and articles made it clearly better at biomedical text tasks than the same model trained only on Wikipedia and books.
Source
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Study at a glance
- Design
- Computational / modelling — BERT-base initialised from general-domain weights, further pre-trained on PubMed abstracts and PMC full texts, then fine-tuned and compared with BERT and prior best models on 15 biomedical datasets.
- N
- No single N: evaluation spans 15 benchmark datasets (9 NER, 3 relation extraction, 3 BioASQ question-answering batches), each with its own size.
- Population
- Biomedical text corpora and standard biomedical NLP benchmarks (e.g. NCBI Disease, BC2GM, CHEMPROT, GAD, BioASQ factoid).
- Outcome
- Entity-level precision/recall/F1 for NER, F1 for relation extraction, strict/lenient accuracy and mean reciprocal rank for QA.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors took BERT, a transformer language model pre-trained on Wikipedia and books, and continued its masked-language-model pre-training on PubMed abstracts and PubMed Central full-text articles, creating BioBERT. They kept BERT's architecture and vocabulary unchanged. They then fine-tuned BERT and several BioBERT variants on three kinds of biomedical tasks (named entity recognition, relation extraction and question answering) and compared them with task-specific state-of-the-art models, also varying corpus size and pre-training steps.
What they found
BioBERT beat plain BERT on every NER dataset and beat the previous best models on six of nine, improving micro-averaged NER F1 by 0.62 points. It raised relation-extraction F1 by 2.80 points on average and question-answering mean reciprocal rank by 12.24 points over the prior best systems. Performance rose with more biomedical pre-training text (mostly improving up to 4.5 billion words) and with more pre-training steps.
The limits
What it doesn't show
Gains over the prior state of the art are modest for NER (under one F1 point) and BioBERT did not win on every dataset; for LINNAEUS and Species-800 the train/test splits may differ from earlier work, so those comparisons are imperfect. Only the BERT-base size was used, and the original general-domain vocabulary was kept, so the paper does not test whether a larger model or a biomedical vocabulary would do better. About 30% of BioASQ factoid questions were dropped as unanswerable, so QA results cover only extractive questions. No statistical significance tests or run-to-run variance are reported for most comparisons.
Key terms
- Pre-training
- Training a model on large amounts of unlabelled text (here by predicting masked words) before adapting it to a specific task.
- Fine-tuning
- Briefly training a pre-trained model on a small labelled dataset for one task, usually adding only a simple output layer.
- Named entity recognition (NER)
- Finding and labelling mentions of things such as genes, diseases or chemicals in text.
- Masked language model
- A training objective where random words are hidden and the model must predict them from the words on both sides.
- Domain adaptation
- Adjusting a model trained on one kind of data (general English) so it works well on another (biomedical text).
- Mean reciprocal rank (MRR)
- A QA metric that averages 1 divided by the rank at which the correct answer appears in the model's list.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What distinguishes BioBERT from the original BERT-base?
Common questions
Why not just use the original BERT for biomedical text?
Biomedical writing is full of gene names, chemical notation and technical terms that are rare in Wikipedia, so the word statistics differ; plain BERT did reasonably well but fell below specialised models on NER, while BioBERT closed and passed that gap.
Did they build a new architecture?
No. BioBERT has exactly the same structure and vocabulary as BERT-base; the only change is the extra pre-training data, which is why its gains can be attributed to domain-specific text.
Does more biomedical text keep helping?
Within the range tested, yes: performance on three NER datasets mostly improved as the PubMed corpus grew toward 4.5 billion words and as pre-training steps increased.
More on Language models