Clinical NLP
Can a local, private LLM pull symptoms out of medical notes?
Open access · cc by · source: Europe PMC
An open language model running inside the hospital could reliably tag symptoms in doctors' notes, with the largest version doing best and struggling most with vaguely described symptoms.
Study at a glance
- Design
- Computational / modelling — Zero-shot and prompted information extraction by Llama 2 (7b, 13b, 70b) run locally, scored against a three-rater consensus ground truth
- N
- N=500 · 500 admission histories from the MIMIC-IV database, each labelled for five features
- Population
- Emergency/ICU patient histories from MIMIC-IV (Beth Israel Deaconess Medical Center)
- Outcome
- Sensitivity, specificity, precision, NPV and accuracy for detecting liver cirrhosis, ascites, abdominal pain, shortness of breath and confusion
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
The 70-billion-parameter model reached 100% sensitivity for cirrhosis and 95% for ascites, but only 76% for confusion, with high specificity for every feature. Bigger models did better, with the biggest jump between the 7b and 13b versions, and the smallest model hallucinated features that were not in the text. Prompt engineering helped the small model most; explicitly asking for chain-of-thought reasoning did not help and sometimes hurt, while the largest model did well even with simple prompts.
Methodology
The authors built a pipeline that runs the open Llama 2 model on local hardware, so patient text never leaves the institution. They asked it to read 500 patient histories and say whether each mentioned liver cirrhosis, ascites, abdominal pain, shortness of breath or confusion, forcing a quoted excerpt plus a yes/no answer in valid JSON. Three model sizes and several prompting strategies (one-shot examples, definitions, chain-of-thought questioning, system versus user prompt placement) were compared against labels agreed by three blinded human raters.
Limitations
The data are only the first 500 notes from one US database, and the texts are in English, so performance in other hospitals or languages is untested. Some features were rare (ascites appeared in only 20 reports), so estimates for them rest on few positive cases. The study compares Llama 2 sizes and prompts but does not benchmark against a cloud model like GPT-4 or a fine-tuned model on the same data, and extraction accuracy is not the same as showing any benefit to patient care.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Prompting alone can get far on clear, well-defined variables.
Carefully prompted general LLMs can extract well-defined facts with high accuracy without task-specific training: GPT-3.5 averaged 0.89 accuracy on lung-cancer pathology attributes (vs 0.51 keyword search), zero-shot Flan-T5 reached F1 ≥0.75 on 21 of 23 haemorrhage concepts, and a locally run Llama 2 70b reached 100% sensitivity for cirrhosis.
Evidence for the claim as stated.
Prompting alone can get far on clear, well-defined variables.
Carefully prompted general LLMs can extract well-defined facts with high accuracy without task-specific training: GPT-3.5 averaged 0.89 accuracy on lung-cancer pathology attributes (vs 0.51 keyword search), zero-shot Flan-T5 reached F1 ≥0.75 on 21 of 23 haemorrhage concepts, and a locally run Llama 2 70b reached 100% sensitivity for cirrhosis.
Scope note — Sensitivity for a vaguer feature, confusion, was only 76%; rare features rest on few positives.
Limits the claim's scope: a different population, assay, or outcome.
Errors are often confident reasoning or invention, not just misses.
LLM extractors fail in specific, recognisable ways: GPT-3.5 applied the wrong staging rule to correctly extracted facts and fabricated a plausible case for blank reports; the smallest Llama 2 hallucinated absent features; Flan-T5 confused closely related procedures and dropped on newer notes.
Evidence for the claim as stated.
Chain-of-thought style prompting helped GPT-3.5 on pathology (asking for tumour size first raised accuracy) but explicit chain-of-thought did not help and sometimes hurt Llama 2 on admission histories.
Same question, contrary or null result.
Open questions
Tensions this paper is part of
From concept pages' “where studies disagree.” Disagreement means the same question; scope means different assays, populations, or outcomes.
Chain-of-thought style prompting helped GPT-3.5 on pathology (asking for tumour size first raised accuracy) but explicit chain-of-thought did not help and sometimes hurt Llama 2 on admission histories.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Can deep learning use a whole hospital record to predict outcomes?
- Can language models find social needs hidden in doctors' notes?
- Can ChatGPT turn messy pathology reports into clean data?
- Can better prompts make GPT good at spotting medical terms?
- Does pretraining BERT on hospital notes help it link medical terms?