Can language models find social needs hidden in doctors' notes?
Fine-tuned language models found patients' social problems such as housing or transport trouble in clinic notes far more often than billing codes did, and were less swayed by race or gender wording than ChatGPT.
Source
Large language models to identify social determinants of health in electronic health records
Study at a glance
- Design
- Computational / modelling — Sentence-level multilabel classification on annotated oncology notes (60/20/20 split) plus two out-of-domain test sets; synthetic-data augmentation, data-quantity ablations, ChatGPT comparison on synthetic sentences, and demographic-injection bias tests.
- N
- N=800 · 800 annotated radiotherapy clinic notes from 770 patients for development; out-of-domain tests used 200 immunotherapy notes and 200 MIMIC-III notes; ChatGPT comparison used 480 validated synthetic sentences.
- Population
- Clinic notes of cancer patients at Boston hospitals, plus ICU notes from MIMIC-III
- Outcome
- Macro-F1 for detecting any and adverse mentions of six social determinants; discrepancy rates after injecting demographic descriptors; patient-level detection versus structured Z-codes
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
Researchers annotated sentences in cancer patients' clinic notes for six social determinants of health: employment, housing, transportation, parental status, relationship status and social support. They fine-tuned BERT and several sizes of Flan-T5 to flag any mention and adverse mentions, with and without extra synthetic sentences generated by GPT-3.5. They tested on held-out notes from the same hospital, a different patient group and a different hospital's ICU database, compared their models with GPT-3.5 and GPT-4 prompted with few or no examples, and checked whether adding race or gender words changed predictions.
What they found
The best models reached a macro-F1 of about 0.71 for any mention and 0.70 for adverse mentions; larger Flan-T5 models beat BERT, and performance dropped on the ICU dataset from another hospital. Synthetic data mainly helped the rarest categories and let models keep performance with roughly half the real labelled data. Fine-tuned models changed their answer after demographic words were inserted less often than ChatGPT (14.3% versus 21.5% of sentence pairs for any mention). At patient level the models found adverse social needs in 93.8% of affected patients, whereas structured Z-codes flagged only 2.0%.
The limits
What it doesn't show
All notes come from Boston hospitals serving a mostly white population, so the models may not generalise elsewhere. ChatGPT could only be tested on synthetic sentences it had itself helped generate, not real notes, so the head-to-head comparison may not reflect real clinical performance. The bias test used a small set of sentence pairs, so differences between specific demographic groups were mostly not significant. Labels depend on a small annotation team, and social support in particular was hard to annotate consistently.
Key terms
- Social determinants of health (SDoH)
- Non-medical conditions such as housing, employment, transport and social support that affect health outcomes.
- Fine-tuning
- Further training a pretrained language model on labelled examples for a specific task.
- Synthetic data augmentation
- Adding artificially generated training examples, here sentences written by GPT-3.5, to boost scarce classes.
- Macro-F1
- The F1 score averaged equally across classes, so rare classes count as much as common ones.
- Few-shot / zero-shot prompting
- Asking a large model to do a task from a handful of examples, or none, in the prompt without updating its weights.
- LoRA
- Low-rank adaptation, a parameter-efficient method that fine-tunes a small number of added weights instead of the whole model.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was the main use of GPT-3.5 in model training?
Common questions
Why not just use the structured codes doctors already enter?
Social needs are rarely coded. In this study Z-codes flagged adverse social needs for only a tiny fraction of the patients whose notes described them.
Did synthetic data replace real annotated data?
No. Synthetic sentences alone did not produce good models, but adding them let models keep performance when about half the real labelled data were removed and helped rare classes.
Why were the fine-tuned models compared with ChatGPT only on synthetic sentences?
Protected patient notes could not be sent to the ChatGPT API, so the comparison used manually checked synthetic sentences, which limits how far it applies to real notes.
More on Clinical NLP