Can ChatGPT turn messy pathology reports into clean data?
With carefully engineered prompts, ChatGPT extracted cancer stage and tumour type from pathology reports with about 89% average accuracy, beating older NLP methods, but it misapplied staging rules and invented answers for blank reports.
Source
A critical assessment of using ChatGPT for extracting structured data from clinical notes
Study at a glance
- Design
- Computational / modelling — Prompt engineering on a small training set of public pathology reports, then held-out evaluation of GPT-3.5-turbo against expert-curated labels, a keyword-search baseline and a fine-tuned BERT NER model
- N
- N=774 · 774 valid TCGA pathology reports in the test set; 78 CDSA reports used for prompt development; a separate 191 osteosarcoma reports for an extension
- Population
- De-identified lung cancer pathology reports from public archives (CDSA, TCGA)
- Outcome
- Accuracy and coverage for pathologic T stage, N stage, overall stage and histology type
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors iteratively refined a prompt for GPT-3.5-turbo on 78 lung cancer pathology reports, asking for tumour size, pT and pN stage, overall stage and histology in a fixed multiple-choice JSON format with evidence and confidence. They then applied the final prompt to 774 separate TCGA reports and compared its answers with expert-curated labels. They also compared it with a keyword-search method and a fine-tuned Bio_ClinicalBERT entity recogniser, checked reproducibility across two runs a month apart, and analysed errors case by case.
What they found
On the test set, accuracy was 0.87 for pT, 0.91 for pN, 0.76 for overall stage and 0.99 for histology, averaging 0.89, versus 0.51 for keyword search and 0.76 for the BERT model. Each prompt change (structured output, evidence requirement, asking for tumour size, giving a worked example) raised training accuracy, from 0.854 up to 0.936. Many errors came from applying the wrong staging rule to correctly extracted facts, such as counting positive lymph nodes instead of using their location. For blank reports the model often produced a consistent fabricated case rather than answering unknown.
The limits
What it doesn't show
The prompt was tuned on only 78 reports, and results come from one cancer type, one staging edition and public archive reports, so accuracy on other hospitals' notes is untested. Commercial models change over time; the two runs agreed on attributes at a rate of only 0.913, so results may not be exactly reproducible. The baselines were trained on the same small annotated set, which may understate what a well-resourced conventional system could do. Overall-stage accuracy of 0.76 is too low for unsupervised clinical use, and the hallucination on empty inputs shows outputs need checking.
Key terms
- Prompt engineering
- Designing and refining the instructions given to a language model to get accurate, consistently formatted answers.
- Chain-of-thought prompting
- Asking the model for intermediate steps (here, tumour size) before its final answer to improve reasoning.
- Few-shot prompting
- Including worked examples in the prompt so the model learns the task in context without retraining.
- Hallucination
- A language model generating plausible but fabricated content not supported by the input.
- Named entity recognition (NER)
- A supervised NLP method that tags spans of text as belonging to categories, requiring annotated training data.
Flashcards
0 of 9 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
Which attribute did ChatGPT extract least accurately?
Common questions
Why did ChatGPT get histology nearly right but overall stage much less often?
Histology is usually stated directly, whereas overall stage requires combining T and N categories with staging rules; errors in either step, or a wrong rule, propagate.
Why didn't they just fine-tune a model?
Supervised methods need weeks of expert annotation; the appeal of prompting is that it needs few labelled examples, and here it also outperformed the fine-tuned baseline.
Is GPT-4 better?
A later GPT-4 Turbo run improved performance by over 5%, but at higher cost.
More on Clinical NLP