Clinical NLP
Can better prompts make GPT good at spotting medical terms?
Open access · cc by · source: Europe PMC
Carefully built prompts made GPT-4 nearly as good as a fine-tuned clinical model at loosely matching medical terms, but it still fell clearly short on exact boundaries.
Study at a glance
- Design
- Computational / modelling — Incremental prompt ablation for GPT-3.5 and GPT-4 at temperature 0 (zero-, 1- and 5-shot), compared with fine-tuned BioClinicalBERT and a CRF on held-out test sets.
- N
- Two corpora: 163 synthetic discharge summaries (MTSamples) and 91 vaccine adverse-event reports (VAERS), each split into train, validation and test.
- Population
- Synthetic clinical notes and public VAERS safety reports
- Outcome
- Precision, recall and F1 for clinical entity extraction under exact and relaxed matching
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
Guidelines plus error-based rules raised F1 in every setting, more for GPT-3.5 (average gain 0.09) than GPT-4 (0.06), and more examples generally helped. The best setup, GPT-4 with all prompt parts and 5 examples, reached relaxed-match F1 of 0.861 on MTSamples and 0.736 on VAERS. BioClinicalBERT still won everywhere, with exact-match F1 of 0.785 on MTSamples, and GPT's exact-match scores were much lower than its relaxed ones because it often got entity boundaries slightly wrong.
Methodology
The authors asked GPT-3.5 and GPT-4 to mark medical problems, treatments and tests in synthetic discharge summaries, and nervous-system adverse events in vaccine safety reports. They added prompt parts one at a time: a basic task description, entity definitions and rules from the annotation guidelines, extra rules written after studying the models' mistakes on training data, and 1 or 5 labelled examples. Results on held-out test sets were compared with a fine-tuned BioClinicalBERT model and a conditional random field.
Limitations
The datasets are small (163 and 91 documents) and one is fully synthetic, so results may not hold on real hospital notes, which could not be sent to the API for privacy reasons. Only two GPT model snapshots were tested, few-shot examples were picked at random, and error rules were tuned on the same training data used for examples. The authors also argue that exact-match scoring may undervalue clinically sensible LLM answers, so the metric itself shapes the conclusion.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
A small amount of labelled data plus fine-tuning remains a strong baseline.
On strict entity-level extraction, fine-tuned smaller models still beat prompted GPT: fine-tuned BioClinicalBERT beat GPT-4 on clinical NER in every setting, largely because GPT got entity boundaries slightly wrong; in seizure-note extraction the hand-labelled notes were the most important ingredient and just 70 labelled notes gave most of the final performance.
Evidence for the claim as stated.
Whether prompted LLMs match fine-tuned models depends on the task and the metric: GPT-3.5 beat a fine-tuned BERT on pathology staging accuracy, but GPT-4 lost to BioClinicalBERT on exact-match clinical NER, and the NER authors argue exact matching undervalues clinically sensible LLM answers.
Evidence for the claim as stated.
Whether prompted LLMs match fine-tuned models depends on the task and the metric: GPT-3.5 beat a fine-tuned BERT on pathology staging accuracy, but GPT-4 lost to BioClinicalBERT on exact-match clinical NER, and the NER authors argue exact matching undervalues clinically sensible LLM answers.
Same question, contrary or null result.
Strict F1 partly measures agreement with labelling rules.
Boundary and annotation-convention errors are a large share of what remains: relaxing exact boundary matching raised NERBio's scores by roughly 3–7 points, most CLSTM errors involved boundaries, and GPT's exact-match NER scores were much lower than relaxed ones.
Evidence for the claim as stated.
Open questions
Tensions this paper is part of
From concept pages' “where studies disagree.” Disagreement means the same question; scope means different assays, populations, or outcomes.
Whether prompted LLMs match fine-tuned models depends on the task and the metric: GPT-3.5 beat a fine-tuned BERT on pathology staging accuracy, but GPT-4 lost to BioClinicalBERT on exact-match clinical NER, and the NER authors argue exact matching undervalues clinically sensible LLM answers.
Whether prompted LLMs match fine-tuned models depends on the task and the metric: GPT-3.5 beat a fine-tuned BERT on pathology staging accuracy, but GPT-4 lost to BioClinicalBERT on exact-match clinical NER, and the NER authors argue exact matching undervalues clinically sensible LLM answers.
Discoveries this paper informs or conflicts with
- Prompted GPT models approach fine-tuned clinical models on loose matches but slip on exact boundaries and rules
This paper informs this development.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Can deep learning use a whole hospital record to predict outcomes?
- Can language models find social needs hidden in doctors' notes?
- Can ChatGPT turn messy pathology reports into clean data?
- Does pretraining BERT on hospital notes help it link medical terms?
- Can a local, private LLM pull symptoms out of medical notes?