Can fine-tuned language models draft replies to patient messages?
A language model fine-tuned only on doctors' real portal replies wrote drafts no better than the doctors, but adding richer, empathetic example replies to its training made it about as good as ChatGPT.
Source
Leveraging large language models for generating responses to patient messages-a subjective analysis
Study at a glance
- Design
- Computational / modelling — Two LoRA-fine-tuned LLaMA-65B models (local data only vs local plus GPT-improved open-source data) compared with ChatGPT-3.5, ChatGPT-4 and real provider replies, rated blind by physicians on Likert scales plus BERTScore
- N
- Training: nearly half a million local message-response pairs; evaluation: 70 responses to 10 rephrased patient messages, each rated by 4 primary care physicians
- Population
- Adult primary care patient-portal messages and provider replies from one US academic medical centre
- Outcome
- Physician ratings (1-5) of empathy, responsiveness, accuracy and usefulness; BERTScore similarity to provider replies
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
Using nearly half a million de-identified patient messages and provider replies from Vanderbilt's patient portal, the team fine-tuned the open LLaMA-65B model with low-rank adaptation. CLAIR-Short was trained on local data only; CLAIR-Long also added 5000 online patient questions whose answers had been rewritten by GPT-3.5 to be fuller and more empathetic. Four blinded primary care physicians rated responses to 10 typical patient messages from both models, ChatGPT-3.5, ChatGPT-4 and the real provider on empathy, responsiveness, accuracy and usefulness.
What they found
ChatGPT responses got the highest ratings, with CLAIR-Long close behind: CLAIR-Long did not differ significantly from either ChatGPT on empathy, accuracy or usefulness, though it scored lower on responsiveness. CLAIR-Long was rated significantly better than CLAIR-Short on empathy, accuracy and usefulness, while CLAIR-Short was statistically indistinguishable from the real providers' brief replies. Of the 20 top-rated responses only one came from the actual doctor, and reviewers found no hallucinations, though they flagged some outdated advice.
The limits
What it doesn't show
The evaluation is tiny: 10 hand-picked, rephrased single-issue messages and four male physicians from one centre, with only moderate inter-rater agreement. Patients never rated the responses, so it says nothing about patient satisfaction or safety in real use. Longer, more detailed answers may simply be preferred as templates, and models trained on historical replies can repeat outdated guidance, as seen in an antibiotic and a COVID treatment example.
Key terms
- Fine-tuning
- Further training a pretrained language model on task-specific examples so its outputs match that task and style.
- Low-rank adaptation (LoRA)
- A cheap fine-tuning method that trains small added weight matrices instead of updating every parameter of a large model.
- Data augmentation
- Adding extra or transformed training examples, here online patient questions with GPT-rewritten fuller answers.
- BERTScore
- An automatic metric that measures how similar generated text is to a reference text using contextual word embeddings; it measures similarity, not correctness.
- Intraclass correlation (ICC)
- A statistic for how consistently different raters score the same items.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
Which model was trained only on local portal messages and replies?
Common questions
Why did training on real doctors' replies not produce better drafts?
Real portal replies are short and often lack empathy or explanation, so a model that imitates them inherits those traits; quality of training data mattered more than its size.
Why bother fine-tuning if ChatGPT was rated best?
An open model can be run locally on protected health information and can match local doctors' tone, which matters for hospitals that cannot send patient data to external services.
Does this show AI replies are safe to send to patients?
No. Only physicians rated a handful of drafts; the authors frame the models as producing starting drafts that a clinician must review.
More on Language models