Skip to content
PaperFren

Clinical NLP

Can AI read doctors' notes to track failing hip implants?

Callahan A, Fries JA, Ré C, et al. · NPJ digital medicine · 2019

Open access · cc by · source: Europe PMC

Training a text model on labels produced by simple hand-written rules, rather than costly hand annotation, let it find far more hip-implant problems in doctors' notes than billing codes reveal.

Save to libraryView originalAsk PaperFrenStudy view

Study at a glance

Design
Computational / modelling — Snorkel data-programming: 50 labeling functions generate probabilistic labels for a bidirectional LSTM relation extractor, compared with rule majority vote and an LSTM trained on 150 hand-labelled documents; outputs then fed to survival and count models
N
N=6583 · 6583 hip replacement patients in the EHR cohort; implant-system analyses restricted to 2704 single-implant patients; model test sets of a few hundred annotated notes
Population
Patients with hip replacement surgery at Stanford Health Care, 1995-2014, and their clinical notes
Outcome
Precision, recall and F1 of extracting implant complications and pain; revision events and complication risk by implant system

Structured fields used in claim comparison tables when every cited study has a complete layer.

Key findings

Weak supervision raised F1 by 9.2 and 24.6 points over the rule baseline for the two relation tasks, mainly by improving recall, while the small hand-labelled LSTM gained recall but lost a lot of precision. Performance kept improving with more weakly labelled documents. Reading the notes uncovered 504 extra revision surgeries beyond the 78 found in billing codes for single-implant patients, revealing differences between implant systems that codes alone missed, and patients who later had revisions had more post-surgery pain mentions.

Methodology

The team gathered the health records of 6583 hip replacement patients and wrote 50 labeling functions, short rules encoding clinical know-how, to label sentences in clinical notes as mentioning implant complications or hip pain. A generative model combined these noisy votes into probabilistic labels for thousands of documents, which trained a bidirectional LSTM. They compared this weakly supervised model with the rules' majority vote and with an LSTM trained on 150 hand-labelled documents, checked implant extraction against a curated joint registry, and then used the extracted events to compare implant systems.

Limitations

All data come from one health system and its surgeons' choice of implants, so the implant comparisons may be confounded by surgeon, case complexity and patient selection; the authors call these associations exploratory, not causal. Weak supervision trades some precision for recall, and the gold-standard labels themselves were imperfect, with only moderate-to-substantial annotator agreement. Operative reports were missing for some patients, so some implants were missed.

How this study connects

Role on claims

Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.

  • SupportsClinical information extraction from notesconcept

    Codes undercount; the notes hold the missing cases.

    Reading notes finds many cases that structured billing or diagnosis codes miss: a note-based postpartum haemorrhage phenotype found that over 47% of confirmed cases had no haemorrhage code, weak-supervision extraction found 504 hip-implant revisions versus 78 in billing codes, and models flagged adverse social needs in 93.8% of affected patients versus 2.0% by Z-codes.

    Evidence for the claim as stated.

Related papers in this topic

Same topic cluster — not a recommendation engine.