Can one model find diseases, genes and how they relate in text?
Treating relation extraction as a sequence-labelling problem with conditional random fields matched or beat older methods, and relation-specific features drove most of the gain.
Source
Extraction of semantic biomedical relations from text using conditional random fields
Study at a glance
- Design
- Computational / modelling — Supervised sequence labelling with CRFs, evaluated by cross-validation on two annotated corpora against several baselines, plus a feature-by-feature ablation.
- N
- Two corpora: 3570 sentences from PubMed abstracts (disease-treatment relations) and 5720 GeneRIF sentences (gene-disease relations); the trained model was then run on 110881 GeneRIFs.
- Population
- Biomedical text: MEDLINE abstracts and Entrez Gene GeneRIF phrases
- Outcome
- Precision, recall and F-measure for named-entity recognition and relation extraction; accuracy for relation classification
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors cast 'semantic relation extraction' (finding entities and labelling the type of relationship between them) as tagging each word in a sentence. They built a cascaded CRF, which first tags entities and then feeds those tags into a second CRF that labels relations, and a one-step CRF that does both at once when the key entity is already known. They tested on 3570 PubMed sentences about diseases and treatments and 5720 GeneRIF sentences about genes and diseases, comparing against an SVM, a multilayer neural network, generative graphical models and simple rule-based systems.
What they found
On disease-treatment data the cascaded CRF reached 72% F-measure for entity recognition (best graphical model 71%) and 96.9% relation accuracy when entities were given, similar to the neural network's 96.6%; with entities unknown, accuracy fell to 79.5%. On gene-disease data the cascaded CRF matched a tuned CRF+SVM pipeline while training about ten times faster, whereas the one-step CRF was slightly worse because it found disease entities less well. In the ablation, adding a dictionary window feature lifted F-measure from 39.65 to 67.38.
The limits
What it doesn't show
Performance was judged by cross-validation on the authors' own annotated corpora, with a strict exact-boundary criterion that makes numbers hard to compare with other studies. Rare relation types (such as 'unrelated' and regulatory modification) had few examples and were extracted poorly. The relation types record association, not causation, and ignore the conditions under which a relation holds; the large gene-disease network built from GeneRIFs is noisy and was not independently validated. The work predates modern neural language models, so the baselines are dated.
Key terms
- Conditional random field (CRF)
- A discriminative probabilistic model that labels a whole sequence (such as the words of a sentence) while taking neighbouring labels into account.
- Named-entity recognition (NER)
- Finding and classifying the names of things, such as diseases or treatments, in text.
- Semantic relation extraction
- Detecting that two entities are related and labelling the type of relation, for example 'cures' or 'genetic variation'.
- Cascaded model
- A pipeline where one model's output is used as an input feature for a second model.
- F-measure
- The harmonic mean of precision and recall.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
How did the authors frame relation extraction for their CRF models?
Common questions
Why use a CRF instead of an SVM for this task?
A CRF labels a whole sequence at once and can take very many overlapping features without costly feature selection, whereas the SVM pipeline needed heavy tuning and the neural network could not handle the high-dimensional entity features at all.
Why was accuracy so much lower when entities were not given?
Relation errors cascade from entity errors: if the system misses a treatment name, it cannot then label that treatment's relation. Treatments were especially hard to spot.
What is the one-step CRF good for?
When the key entity is already known (as in GeneRIFs, which are tied to a gene), it tags diseases and relations together and trains faster, though here it was a little less accurate.
More on Biomedical text mining