Skip to content
PaperFren

Language models

Can a general chatbot-style model be taught to spot biomedical terms?

Keloth VK, Hu Y, Xie Q, et al. · Bioinformatics (Oxford, England) · 2024

Open access · cc by · source: Europe PMC

A general-purpose open language model, fine-tuned on instructions made from existing labelled data, found diseases, chemicals and genes in biomedical text about as well as a specialist biomedical model and far better than prompted GPT models.

Save to libraryView originalAsk PaperFrenStudy view

Study at a glance

Design
Computational / modelling — Instruction tuning of LLaMA-1/LLaMA-2 7B on three NER training sets turned into instruction prompts; evaluated on held-out test sets and three unseen datasets against baselines.
N
N=22484 · Instruction-following demonstrations used for fine-tuning, built from the training splits of NCBI disease, BC5CDR-Chemical and BC2GM; test sets and three further generalisation datasets are separate.
Population
Sentences from biomedical abstracts and articles annotated for disease, chemical and gene mentions
Outcome
Strict (exact-match) and partial-match precision, recall and F1 for named entity recognition

Structured fields used in claim comparison tables when every cited study has a complete layer.

Key findings

One BioNER-LLaMA model matched the three task-specific PubMedBERT models, doing slightly better on two datasets and worse on chemicals, though the differences were not statistically significant; both beat PMC-LLaMA, and GPT models trailed well behind. Training on all three datasets together usually helped: on NCBI disease a single-dataset model scored an F1 of 0.762 versus 0.876 for the combined model. Cutting the training data to 10% lowered performance by roughly 11-14%, and adding richer instructions made no noticeable difference.

Methodology

The authors converted three annotated biomedical corpora (diseases, chemicals, genes) into instruction examples asking a model to tag entity mentions in a sentence, and used them to fine-tune LLaMA 7B, giving BioNER-LLaMA. They compared it with three separately fine-tuned PubMedBERT models, zero- and five-shot GPT-3.5 and GPT-4, and a medical LLM (PMC-LLaMA), then tested all models on three datasets not used in training. Extra experiments shrank the training data, trained on one dataset at a time, and added annotation-guideline detail to the prompt.

Limitations

The gains over PubMedBERT were small and not statistically significant, so the paper shows parity, not superiority, and the LLaMA model needs far more compute than a 110M-parameter BERT. GPT models were tested with basic prompts and random few-shot examples, which the authors admit likely understates what they can do. Many remaining errors come from annotation-guideline conventions (entity boundaries, abbreviations such as element symbols), so strict F1 partly measures agreement with dataset rules rather than true understanding. Only NER was tested, so claims about other biomedical tasks are speculative.

How this study connects

Role on claims

Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.

  • SupportsBiomedical named entity recognitionconcept

    Training across datasets is a reliable lever with small data.

    Sharing information across datasets or entity types helps, especially when data are scarce: multi-task training gave 16 significant gains in 27 same-entity experiments and helped most as training data shrank; CollaboNet cut entity-type confusions from 30.8% to 18.6% of errors; one LLaMA trained on three datasets beat single-dataset training (0.876 vs 0.762 F1 on NCBI disease).

    Evidence for the claim as stated.

  • SupportsDomain-specific language model pretrainingconcept

    Tuned LLMs reach parity, at much higher compute cost.

    Fine-tuning open LLMs on domain tasks can match or beat smaller specialised models: an instruction-tuned LLaMA matched three task-specific PubMedBERT models on biomedical NER (differences not significant), and DRG-LLaMA beat ClinicalBERT on diagnosis-related-group prediction (52.0% top-1 accuracy for 7B).

    Evidence for the claim as stated.

Related papers in this topic

Same topic cluster — not a recommendation engine.