Skip to content
PaperFren

Clinical NLP

Can language models find social needs hidden in doctors' notes?

Guevara M, Chen S, Thomas S, et al. · NPJ digital medicine · 2024

Open access · cc by · source: Europe PMC

Fine-tuned language models found patients' social problems such as housing or transport trouble in clinic notes far more often than billing codes did, and were less swayed by race or gender wording than ChatGPT.

Save to libraryView originalAsk PaperFrenStudy view

Study at a glance

Design
Computational / modelling — Sentence-level multilabel classification on annotated oncology notes (60/20/20 split) plus two out-of-domain test sets; synthetic-data augmentation, data-quantity ablations, ChatGPT comparison on synthetic sentences, and demographic-injection bias tests.
N
N=800 · 800 annotated radiotherapy clinic notes from 770 patients for development; out-of-domain tests used 200 immunotherapy notes and 200 MIMIC-III notes; ChatGPT comparison used 480 validated synthetic sentences.
Population
Clinic notes of cancer patients at Boston hospitals, plus ICU notes from MIMIC-III
Outcome
Macro-F1 for detecting any and adverse mentions of six social determinants; discrepancy rates after injecting demographic descriptors; patient-level detection versus structured Z-codes

Structured fields used in claim comparison tables when every cited study has a complete layer.

Key findings

The best models reached a macro-F1 of about 0.71 for any mention and 0.70 for adverse mentions; larger Flan-T5 models beat BERT, and performance dropped on the ICU dataset from another hospital. Synthetic data mainly helped the rarest categories and let models keep performance with roughly half the real labelled data. Fine-tuned models changed their answer after demographic words were inserted less often than ChatGPT (14.3% versus 21.5% of sentence pairs for any mention). At patient level the models found adverse social needs in 93.8% of affected patients, whereas structured Z-codes flagged only 2.0%.

Methodology

Researchers annotated sentences in cancer patients' clinic notes for six social determinants of health: employment, housing, transportation, parental status, relationship status and social support. They fine-tuned BERT and several sizes of Flan-T5 to flag any mention and adverse mentions, with and without extra synthetic sentences generated by GPT-3.5. They tested on held-out notes from the same hospital, a different patient group and a different hospital's ICU database, compared their models with GPT-3.5 and GPT-4 prompted with few or no examples, and checked whether adding race or gender words changed predictions.

Limitations

All notes come from Boston hospitals serving a mostly white population, so the models may not generalise elsewhere. ChatGPT could only be tested on synthetic sentences it had itself helped generate, not real notes, so the head-to-head comparison may not reflect real clinical performance. The bias test used a small set of sentence pairs, so differences between specific demographic groups were mostly not significant. Labels depend on a small annotation team, and social support in particular was hard to annotate consistently.

How this study connects

Role on claims

Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.

  • SupportsClinical information extraction from notesconcept

    Codes undercount; the notes hold the missing cases.

    Reading notes finds many cases that structured billing or diagnosis codes miss: a note-based postpartum haemorrhage phenotype found that over 47% of confirmed cases had no haemorrhage code, weak-supervision extraction found 504 hip-implant revisions versus 78 in billing codes, and models flagged adverse social needs in 93.8% of affected patients versus 2.0% by Z-codes.

    Evidence for the claim as stated.

  • SupportsAlgorithmic fairness and biasconcept

    Demographic words alone can sway clinical NLP outputs.

    Language models can change outputs when only demographic words change: fine-tuned models changed their social-determinants answers in 14.3% of demographically altered sentence pairs versus 21.5% for ChatGPT.

    Evidence for the claim as stated.

Related papers in this topic

Same topic cluster — not a recommendation engine.