Can a local, private LLM pull symptoms out of medical notes?
An open language model running inside the hospital could reliably tag symptoms in doctors' notes, with the largest version doing best and struggling most with vaguely described symptoms.
Source
Privacy-preserving large language models for structured medical information retrieval
Study at a glance
- Design
- Computational / modelling — Zero-shot and prompted information extraction by Llama 2 (7b, 13b, 70b) run locally, scored against a three-rater consensus ground truth
- N
- N=500 · 500 admission histories from the MIMIC-IV database, each labelled for five features
- Population
- Emergency/ICU patient histories from MIMIC-IV (Beth Israel Deaconess Medical Center)
- Outcome
- Sensitivity, specificity, precision, NPV and accuracy for detecting liver cirrhosis, ascites, abdominal pain, shortness of breath and confusion
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors built a pipeline that runs the open Llama 2 model on local hardware, so patient text never leaves the institution. They asked it to read 500 patient histories and say whether each mentioned liver cirrhosis, ascites, abdominal pain, shortness of breath or confusion, forcing a quoted excerpt plus a yes/no answer in valid JSON. Three model sizes and several prompting strategies (one-shot examples, definitions, chain-of-thought questioning, system versus user prompt placement) were compared against labels agreed by three blinded human raters.
What they found
The 70-billion-parameter model reached 100% sensitivity for cirrhosis and 95% for ascites, but only 76% for confusion, with high specificity for every feature. Bigger models did better, with the biggest jump between the 7b and 13b versions, and the smallest model hallucinated features that were not in the text. Prompt engineering helped the small model most; explicitly asking for chain-of-thought reasoning did not help and sometimes hurt, while the largest model did well even with simple prompts.
The limits
What it doesn't show
The data are only the first 500 notes from one US database, and the texts are in English, so performance in other hospitals or languages is untested. Some features were rare (ascites appeared in only 20 reports), so estimates for them rest on few positive cases. The study compares Llama 2 sizes and prompts but does not benchmark against a cloud model like GPT-4 or a fine-tuned model on the same data, and extraction accuracy is not the same as showing any benefit to patient care.
Key terms
- Zero-shot prompting
- Asking a pretrained model to do a task it was never specifically trained on, using only instructions in the prompt.
- One-shot prompting
- Including a single worked example in the prompt to show the model the desired input-output format.
- Quantization
- Storing a model's parameters at lower numerical precision so it uses less GPU memory and can run on modest hardware.
- Sensitivity and specificity
- Sensitivity is the share of true positives the model catches; specificity is the share of true negatives it correctly leaves unflagged.
- Hallucination
- When a language model states something as present or true that is not supported by its input.
- Grammar-based sampling
- Constraining the model's text generation with formal rules so the output always follows a required structure such as valid JSON.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was the main motivation for using Llama 2 instead of a cloud model?
Common questions
Why not just use GPT-4, which the authors say is often better?
GPT-4 runs on remote servers, so using it means sending patient data off-site, which raises legal problems especially in the EU. A local model keeps data inside the hospital.
Why was confusion the hardest feature to detect?
Confusion is rarely stated in a single standard phrase and doctors document it inconsistently, so the model had to infer it from context; even the human raters needed to agree on a definition.
Does a bigger model always need less prompt tweaking?
In this study, yes: prompt changes shifted the 7b model's accuracy by several points but had small and inconsistent effects on the 70b model, which already did well with simple prompts.
More on Clinical NLP