Skip to content
PaperFren

Biomedical text mining

Does training on many biomedical datasets at once improve NER?

Crichton G, Pyysalo S, Chiu B, et al. · BMC bioinformatics · 2017

Open access · cc by · source: Europe PMC

Letting one neural network learn many biomedical name-tagging datasets together gave modest average gains, larger gains on some datasets, and the biggest benefit when training data was scarce.

How this study connects

Role on claims

Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.

  • Training across datasets is a reliable lever with small data.

    Sharing information across datasets or entity types helps, especially when data are scarce: multi-task training gave 16 significant gains in 27 same-entity experiments and helped most as training data shrank; CollaboNet cut entity-type confusions from 30.8% to 18.6% of errors; one LLaMA trained on three datasets beat single-dataset training (0.876 vs 0.762 F1 on NCBI disease).

    Evidence for the claim as stated.

Study at a glance

Design
Computational / modelling — Controlled comparison of a single-task CNN tagger against two multi-task CNN variants (shared layers with private outputs; POS-tagger-fed dependent model) across 15 NER corpora, plus pairwise, grouped and reduced-training-data experiments.
N
No single N: 15 biomedical NER corpora plus one POS-tagging corpus; pairwise experiment trained 15 x 14 models.
Population
Public biomedical text corpora annotated for anatomy, chemicals, diseases, genes/proteins and species.
Outcome
Exact mention-level macro F1 score on each corpus's test set.

Structured fields used in claim comparison tables when every cited study has a complete layer.

Inspect source passages

Does training on many biomedical datasets at once improve NER?

These passages occur verbatim in the stored paper text. They support the points listed below; they do not verify every statement in the explanation.

15 NER datasets used

We apply these to 15 datasets containing multiple named entities including Anatomy, Chemical, Disease, Gene/Protein and Species.

5 datasets improved significantly with the shared-output model

There were 5 datasets where it performed significantly better and 1 dataset where it was significantly worse.
Read passages in the original paper

Key findings

Training on all datasets together raised F-scores significantly on five datasets and lowered one for the shared-output model, while the part-of-speech model improved six and hurt none. Grouping datasets that annotate the same entity type produced 16 significant gains out of 27 experiments. The advantage of multi-task learning grew as training data shrank; for two datasets, half the training data plus a helper dataset beat the full data alone. Which dataset helped which was sometimes surprising, e.g. a gene/protein corpus best helped a chemical corpus.

Methodology

The authors built a convolutional neural network that tags each word in a sentence as part of a named entity (such as a gene or disease name) or not. They compared it with two multi-task versions: one sharing all layers except a separate output per dataset, and one feeding in features from a part-of-speech tagger. They trained on 15 public biomedical corpora, tested every pair of datasets, grouped datasets by entity type, and cut the training data to 50%, 25% and 10% to see how multi-task learning behaves with less data.

Limitations

Average improvements were small (under one F-score point overall), and the authors admit it is hard to predict in advance which dataset combinations will help. The models deliberately skip task-specific tricks such as gazetteers, so they are not state-of-the-art systems, and several corpora have no established NER benchmark to compare against. The models cannot handle overlapping annotations of the same term, and the static-embedding CNN setup predates transformer language models.

Related papers in this topic

Same topic cluster — not a recommendation engine.