Clinical NLP
Can ChatGPT turn messy pathology reports into clean data?
Open access · cc by · source: Europe PMC
With carefully engineered prompts, ChatGPT extracted cancer stage and tumour type from pathology reports with about 89% average accuracy, beating older NLP methods, but it misapplied staging rules and invented answers for blank reports.
Study at a glance
- Design
- Computational / modelling — Prompt engineering on a small training set of public pathology reports, then held-out evaluation of GPT-3.5-turbo against expert-curated labels, a keyword-search baseline and a fine-tuned BERT NER model
- N
- N=774 · 774 valid TCGA pathology reports in the test set; 78 CDSA reports used for prompt development; a separate 191 osteosarcoma reports for an extension
- Population
- De-identified lung cancer pathology reports from public archives (CDSA, TCGA)
- Outcome
- Accuracy and coverage for pathologic T stage, N stage, overall stage and histology type
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
On the test set, accuracy was 0.87 for pT, 0.91 for pN, 0.76 for overall stage and 0.99 for histology, averaging 0.89, versus 0.51 for keyword search and 0.76 for the BERT model. Each prompt change (structured output, evidence requirement, asking for tumour size, giving a worked example) raised training accuracy, from 0.854 up to 0.936. Many errors came from applying the wrong staging rule to correctly extracted facts, such as counting positive lymph nodes instead of using their location. For blank reports the model often produced a consistent fabricated case rather than answering unknown.
Methodology
The authors iteratively refined a prompt for GPT-3.5-turbo on 78 lung cancer pathology reports, asking for tumour size, pT and pN stage, overall stage and histology in a fixed multiple-choice JSON format with evidence and confidence. They then applied the final prompt to 774 separate TCGA reports and compared its answers with expert-curated labels. They also compared it with a keyword-search method and a fine-tuned Bio_ClinicalBERT entity recogniser, checked reproducibility across two runs a month apart, and analysed errors case by case.
Limitations
The prompt was tuned on only 78 reports, and results come from one cancer type, one staging edition and public archive reports, so accuracy on other hospitals' notes is untested. Commercial models change over time; the two runs agreed on attributes at a rate of only 0.913, so results may not be exactly reproducible. The baselines were trained on the same small annotated set, which may understate what a well-resourced conventional system could do. Overall-stage accuracy of 0.76 is too low for unsupervised clinical use, and the hallucination on empty inputs shows outputs need checking.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Prompting alone can get far on clear, well-defined variables.
Carefully prompted general LLMs can extract well-defined facts with high accuracy without task-specific training: GPT-3.5 averaged 0.89 accuracy on lung-cancer pathology attributes (vs 0.51 keyword search), zero-shot Flan-T5 reached F1 ≥0.75 on 21 of 23 haemorrhage concepts, and a locally run Llama 2 70b reached 100% sensitivity for cirrhosis.
Evidence for the claim as stated.
Errors are often confident reasoning or invention, not just misses.
LLM extractors fail in specific, recognisable ways: GPT-3.5 applied the wrong staging rule to correctly extracted facts and fabricated a plausible case for blank reports; the smallest Llama 2 hallucinated absent features; Flan-T5 confused closely related procedures and dropped on newer notes.
Evidence for the claim as stated.
Whether prompted LLMs match fine-tuned models depends on the task and the metric: GPT-3.5 beat a fine-tuned BERT on pathology staging accuracy, but GPT-4 lost to BioClinicalBERT on exact-match clinical NER, and the NER authors argue exact matching undervalues clinically sensible LLM answers.
Evidence for the claim as stated.
Chain-of-thought style prompting helped GPT-3.5 on pathology (asking for tumour size first raised accuracy) but explicit chain-of-thought did not help and sometimes hurt Llama 2 on admission histories.
Evidence for the claim as stated.
Expert review stays necessary.
Hallucinations and outdated advice appear in AI output: a hallucinated drug name in decision-support suggestions and outdated antibiotic and COVID advice in model replies were flagged.
Evidence for the claim as stated.
Open questions
Tensions this paper is part of
From concept pages' “where studies disagree.” Disagreement means the same question; scope means different assays, populations, or outcomes.
Whether prompted LLMs match fine-tuned models depends on the task and the metric: GPT-3.5 beat a fine-tuned BERT on pathology staging accuracy, but GPT-4 lost to BioClinicalBERT on exact-match clinical NER, and the NER authors argue exact matching undervalues clinically sensible LLM answers.
Chain-of-thought style prompting helped GPT-3.5 on pathology (asking for tumour size first raised accuracy) but explicit chain-of-thought did not help and sometimes hurt Llama 2 on admission histories.
Discoveries this paper informs or conflicts with
- Prompted GPT models approach fine-tuned clinical models on loose matches but slip on exact boundaries and rules
This paper informs this development.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Can deep learning use a whole hospital record to predict outcomes?
- Can language models find social needs hidden in doctors' notes?
- Can better prompts make GPT good at spotting medical terms?
- Does pretraining BERT on hospital notes help it link medical terms?
- Can a local, private LLM pull symptoms out of medical notes?