Can a general chatbot-style model be taught to spot biomedical terms?
A general-purpose open language model, fine-tuned on instructions made from existing labelled data, found diseases, chemicals and genes in biomedical text about as well as a specialist biomedical model and far better than prompted GPT models.
Source
Advancing entity recognition in biomedicine via instruction tuning of large language models
Study at a glance
- Design
- Computational / modelling — Instruction tuning of LLaMA-1/LLaMA-2 7B on three NER training sets turned into instruction prompts; evaluated on held-out test sets and three unseen datasets against baselines.
- N
- N=22484 · Instruction-following demonstrations used for fine-tuning, built from the training splits of NCBI disease, BC5CDR-Chemical and BC2GM; test sets and three further generalisation datasets are separate.
- Population
- Sentences from biomedical abstracts and articles annotated for disease, chemical and gene mentions
- Outcome
- Strict (exact-match) and partial-match precision, recall and F1 for named entity recognition
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors converted three annotated biomedical corpora (diseases, chemicals, genes) into instruction examples asking a model to tag entity mentions in a sentence, and used them to fine-tune LLaMA 7B, giving BioNER-LLaMA. They compared it with three separately fine-tuned PubMedBERT models, zero- and five-shot GPT-3.5 and GPT-4, and a medical LLM (PMC-LLaMA), then tested all models on three datasets not used in training. Extra experiments shrank the training data, trained on one dataset at a time, and added annotation-guideline detail to the prompt.
What they found
One BioNER-LLaMA model matched the three task-specific PubMedBERT models, doing slightly better on two datasets and worse on chemicals, though the differences were not statistically significant; both beat PMC-LLaMA, and GPT models trailed well behind. Training on all three datasets together usually helped: on NCBI disease a single-dataset model scored an F1 of 0.762 versus 0.876 for the combined model. Cutting the training data to 10% lowered performance by roughly 11-14%, and adding richer instructions made no noticeable difference.
The limits
What it doesn't show
The gains over PubMedBERT were small and not statistically significant, so the paper shows parity, not superiority, and the LLaMA model needs far more compute than a 110M-parameter BERT. GPT models were tested with basic prompts and random few-shot examples, which the authors admit likely understates what they can do. Many remaining errors come from annotation-guideline conventions (entity boundaries, abbreviations such as element symbols), so strict F1 partly measures agreement with dataset rules rather than true understanding. Only NER was tested, so claims about other biomedical tasks are speculative.
Key terms
- Named entity recognition (NER)
- Finding and labelling mentions of things like diseases, chemicals or genes in text.
- Instruction tuning
- Supervised fine-tuning of a language model on examples written as an instruction, an input and the desired output, so it learns to follow task instructions.
- Strict vs partial F1
- Strict F1 only counts a predicted entity correct if its boundaries match exactly; partial F1 also gives credit for overlapping spans.
- Zero-shot / few-shot prompting
- Asking a model to do a task with no worked examples, or with a handful of examples in the prompt, without updating its weights.
- PubMedBERT
- A BERT model pretrained on PubMed text, a strong specialist baseline for biomedical language tasks.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was the main result for BioNER-LLaMA versus fine-tuned PubMedBERT?
Common questions
Why did GPT-4 do so badly if it is a bigger model?
It was only prompted, not trained on the datasets, and NER needs exact boundary placement that generative models struggle with; the simple prompts and random examples probably made it worse.
What is the advantage of one instruction-tuned model over three BERT models?
A single model can switch entity type just by changing the instruction, and the same approach could extend to other extraction tasks without building separate models.
Why do strict and partial scores differ so much on some datasets?
Datasets have fine-grained rules about how much text to include; the model often tagged the longest phrase, which overlaps the gold answer but fails an exact match.
More on Language models