Can an LLM pipeline speed up systematic reviews without losing quality?
Breaking systematic-review work into structured steps that a large language model performs and experts can check found far more relevant studies and extracted data more accurately than simply prompting GPT-4.
Source
Accelerating clinical evidence synthesis with large language models
Study at a glance
- Design
- Computational / modelling — LLM pipeline (TrialMind) evaluated against GPT-4 prompting, manual and embedding-ranking baselines on a new benchmark built from published reviews, plus annotator ratings and a two-person AI-assisted vs manual user study.
- N
- No single N: benchmark of 100 systematic reviews covering 2220 studies with 1334 characteristic and 1049 result annotations; 8 annotators rated evidence; user study had 2 participants.
- Population
- Published oncology systematic reviews and the clinical studies they include (PubMed Central).
- Outcome
- Search recall, screening Recall@20/50, data-extraction accuracy, hallucination precision/recall, annotator preference, task time.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors turned 100 published cancer-therapy systematic reviews into a benchmark (TrialReviewBench) with three tasks: searching PubMed for the right studies, ranking candidates for inclusion, and extracting study details and results. They built TrialMind, a GPT-4-based pipeline following the standard PRISMA review steps with query generation and refinement, criteria-based screening, and reasoning-plus-code result extraction. They compared it with GPT-4 prompted directly, a manual keyword baseline and embedding-based rankers, had 8 annotators judge synthesised forest plots, and ran a small user study comparing experts with and without the tool.
What they found
TrialMind's searches recovered about 78% of the target studies on average, versus about 7% for plain GPT-4 queries and 19% for the manual baseline. In screening, it placed on average 43% of target studies in the top 50 out of 2,000 similar candidates. Extraction accuracy was high for study design but much lower for numerical results (for immunotherapy 0.95 vs 0.42), and for result extraction it beat GPT-4 in every topic. In the user study, AI-assisted screening improved recall by 71.4% and took 44.2% less time, and assisted extraction was 23.5% more accurate with 63.4% less time.
The limits
What it doesn't show
The user study had only two participants, so its time savings are illustrative rather than reliable estimates. The benchmark is limited to oncology therapy reviews from PubMed Central, and the authors say generalisation to diagnostics, prevention or other fields is unproven. The system still makes errors, especially with numbers, so human checking remains essential, and it does not cover quality assessment or writing the review. It also depends on a costly proprietary model.
Key terms
- Systematic review
- A structured, reproducible search and synthesis of all studies answering a defined research question.
- PRISMA
- A reporting standard laying out the identification, screening and inclusion stages of a systematic review.
- PICO
- Population, Intervention, Comparison, Outcome - the elements used to define a clinical research question.
- Recall@k
- The fraction of truly relevant items that appear among the top k ranked results.
- Hallucination
- Output generated by a language model that is not supported by the input documents.
- Human-in-the-loop
- A design where people can inspect and correct each intermediate AI output before it moves on.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was TrialMind's average search recall on the benchmark?
Common questions
Why did plain GPT-4 search so poorly?
Asking it to write a Boolean query directly produced overly narrow queries; TrialMind generated, expanded and refined terms, retrieving many more candidates.
Which extraction job was hardest?
Numerical results, which require finding the right subgroup and sometimes calculating values; accuracy was far lower than for study-design fields.
Does this replace human reviewers?
No; the authors stress that errors can occur at any stage and the tool is built so experts verify each step.
More on Language models