Large language models · Clinical NLP · Prompting
Prompted GPT models approach fine-tuned clinical models on loose matches but slip on exact boundaries and rules
Save this development or follow its topic to track what changes.
Short answer
Prompt engineering makes general LLMs competitive for clinical extraction, but fine-tuned domain models still win on exact boundaries, and LLMs can apply rules wrongly or fabricate answers.
What happened
On two small clinical NER corpora, adding annotation guidelines, error-based rules and examples raised GPT-4 to a relaxed-match F1 of 0.861 on synthetic discharge summaries, yet BioClinicalBERT still won everywhere, with exact-match F1 of 0.785. On 774 pathology reports, an engineered ChatGPT prompt averaged 0.89 accuracy across stage and histology, above keyword search (0.51) and a BERT model (0.76), but misapplied staging rules and invented a consistent case for blank reports. Earlier, BioBERT showed that continued pretraining on biomedical text gave consistent gains over general BERT.
Why it matters
It is tempting to treat a general chatbot as a drop-in replacement for trained clinical NLP models. The comparison shows where prompting closes the gap and where it does not, and why exact-match metrics matter when the output feeds a database.
Evidence
- Study type
- Benchmark comparisons of prompted LLMs against fine-tuned models
- Sample
- 163 synthetic discharge summaries and 91 VAERS reports; 774 TCGA pathology test reports
- Journal
- Journal of the American Medical Informatics Association · peer reviewed
- Replication
- Similar patterns across two groups and tasks, but on specific model snapshots that have since changed
- Limitations
- Small and partly synthetic corpora, one cancer type and staging edition, prompts tuned on few reports, and limited reproducibility of commercial models.
What this connects to
Sources
The 3 studies this explanation is built from, by the role each plays. Every source links to PaperFren’s explanation of it and to the original paper.
Primary study
- Can better prompts make GPT good at spotting medical terms?
Carefully built prompts made GPT-4 nearly as good as a fine-tuned clinical model at loosely matching medical terms, but it still fell clearly short on exact boundaries.
What it does not showLimitations
The datasets are small (163 and 91 documents) and one is fully synthetic, so results may not hold on real hospital notes, which could not be sent to the API for privacy reasons. Only two GPT model snapshots were tested, few-shot examples were picked at random, and error rules were tuned on the same training data used for examples. The authors also argue that exact-match scoring may undervalue clinically sensible LLM answers, so the metric itself shapes the conclusion.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
Supporting evidence
- Can ChatGPT turn messy pathology reports into clean data?
With carefully engineered prompts, ChatGPT extracted cancer stage and tumour type from pathology reports with about 89% average accuracy, beating older NLP methods, but it misapplied staging rules and invented answers for blank reports.
What it does not showLimitations
The prompt was tuned on only 78 reports, and results come from one cancer type, one staging edition and public archive reports, so accuracy on other hospitals' notes is untested. Commercial models change over time; the two runs agreed on attributes at a rate of only 0.913, so results may not be exactly reproducible. The baselines were trained on the same small annotated set, which may understate what a well-resourced conventional system could do. Overall-stage accuracy of 0.76 is too low for unsupervised clinical use, and the hallucination on empty inputs shows outputs need checking.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
Landmark research
- Does pre-training BERT on biomedical papers help it read biology?
Letting a general language model keep learning from millions of biomedical abstracts and articles made it clearly better at biomedical text tasks than the same model trained only on Wikipedia and books.
What it does not showLimitations
Gains over the prior state of the art are modest for NER (under one F1 point) and BioBERT did not win on every dataset; for LINNAEUS and Species-800 the train/test splits may differ from earlier work, so those comparisons are imperfect. Only the BERT-base size was used, and the original general-domain vocabulary was kept, so the paper does not test whether a larger model or a biomedical vocabulary would do better. About 30% of BioASQ factoid questions were dropped as unanswerable, so QA results cover only extractive questions. No statistical significance tests or run-to-run variance are reported for most comparisons.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
Before
Either domain-pretrained models like BioBERT are needed for clinical text, or a large general model with a good prompt makes them obsolete.
Now
Neither extreme holds in these studies: prompting got close on loose matching and beat older baselines on one structured task, but fine-tuned models kept the edge on exact extraction. Datasets were small or synthetic, commercial models change over time (run-to-run agreement 0.913), and real hospital notes were not tested for privacy reasons.