Can better prompts make GPT good at spotting medical terms?
Carefully built prompts made GPT-4 nearly as good as a fine-tuned clinical model at loosely matching medical terms, but it still fell clearly short on exact boundaries.
Source
Improving large language models for clinical named entity recognition via prompt engineering
Study at a glance
- Design
- Computational / modelling — Incremental prompt ablation for GPT-3.5 and GPT-4 at temperature 0 (zero-, 1- and 5-shot), compared with fine-tuned BioClinicalBERT and a CRF on held-out test sets.
- N
- Two corpora: 163 synthetic discharge summaries (MTSamples) and 91 vaccine adverse-event reports (VAERS), each split into train, validation and test.
- Population
- Synthetic clinical notes and public VAERS safety reports
- Outcome
- Precision, recall and F1 for clinical entity extraction under exact and relaxed matching
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors asked GPT-3.5 and GPT-4 to mark medical problems, treatments and tests in synthetic discharge summaries, and nervous-system adverse events in vaccine safety reports. They added prompt parts one at a time: a basic task description, entity definitions and rules from the annotation guidelines, extra rules written after studying the models' mistakes on training data, and 1 or 5 labelled examples. Results on held-out test sets were compared with a fine-tuned BioClinicalBERT model and a conditional random field.
What they found
Guidelines plus error-based rules raised F1 in every setting, more for GPT-3.5 (average gain 0.09) than GPT-4 (0.06), and more examples generally helped. The best setup, GPT-4 with all prompt parts and 5 examples, reached relaxed-match F1 of 0.861 on MTSamples and 0.736 on VAERS. BioClinicalBERT still won everywhere, with exact-match F1 of 0.785 on MTSamples, and GPT's exact-match scores were much lower than its relaxed ones because it often got entity boundaries slightly wrong.
The limits
What it doesn't show
The datasets are small (163 and 91 documents) and one is fully synthetic, so results may not hold on real hospital notes, which could not be sent to the API for privacy reasons. Only two GPT model snapshots were tested, few-shot examples were picked at random, and error rules were tuned on the same training data used for examples. The authors also argue that exact-match scoring may undervalue clinically sensible LLM answers, so the metric itself shapes the conclusion.
Key terms
- Named entity recognition
- Finding spans of text that name things of a given type, such as a medical problem or a test, and labelling their type.
- Prompt engineering
- Designing the instructions and examples given to a language model to steer its output without retraining it.
- Few-shot learning
- Giving a model a handful of worked examples in the prompt so it can perform a new task.
- Exact versus relaxed match
- Exact match needs identical span boundaries and type; relaxed match accepts any overlap with the correct type.
- Fine-tuning
- Further training a pretrained model on labelled task data, as done here with BioClinicalBERT.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
Which model achieved the highest F1 on both datasets?
Common questions
Did GPT-4 beat the fine-tuned model?
No. BioClinicalBERT was best on both datasets, but GPT-4 came fairly close under relaxed matching while using only a few examples.
Why is exact-match performance so much lower for GPT?
GPT often included or dropped words like articles and modifiers around an entity, so its spans overlapped the right answer but did not match exactly.
Which prompt parts helped most?
Annotation-guideline definitions and error-based rules gave steady gains, and 5 examples usually beat 1 or none.
More on Clinical NLP