Can smarter retrieval help LLMs read long clinical notes?
Retrieving only the passages around clinically relevant terms let language models extract patient information slightly more accurately while using far less text and time than feeding whole notes or embedding-based chunks.
Source
Clinical entity augmented retrieval for clinical information extraction
Study at a glance
- Design
- Computational / modelling — Pipeline of zero-shot NER (Flan-T5), embedding + GPT-4 entity filtering, UMLS/GPT-4 synonym augmentation and 150-word context windows, fed to six LLMs for 5-shot variable extraction; compared with BGE chunk-embedding RAG (top-3/5/10) and full-note chunking, plus weak-labelling and BERT-distillation baselines.
- N
- No single N: Stanford MOUD cohort (767 training and 505 testing patients) with 420 physician-annotated test notes for 13 variables; CheXpert subset with 1000 annotated test radiology reports for 5 findings; NER sensitivity checked on 1382 NCBI and 450 MOUD entities.
- Population
- Electronic health record notes from Stanford patients treated for opioid use disorder, and Stanford chest X-ray radiology reports.
- Outcome
- F1 (and sensitivity, specificity, PPV, NPV) for extracting presence/absence/uncertainty of clinical variables; inference time, model calls and input tokens per note; AUC of distilled BERT classifiers.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors built CLEAR, a retrieval pipeline that first finds all clinical terms in a note with a zero-shot language model, keeps those related to the question (for example 'depression'), adds synonyms from a medical ontology and GPT-4, and then passes only the 150 words around each match to a language model. They compared it with a standard embedding-based retrieval method and with feeding the full note, using six LLMs to extract 13 variables from opioid-use-disorder clinic notes and 5 findings from chest X-ray reports, judged against physician labels. They also used CLEAR's outputs to train a small BERT classifier.
What they found
Averaged across models, CLEAR reached an F1 of 0.90 on the opioid-cohort variables and 0.96 on the radiology findings, and beat chunk-embedding and full-note approaches for all models (e.g. GPT-4: 0.97 vs 0.88 vs 0.90). It needed 1.68 model calls per note versus 4.94 for chunk embedding, used 71% fewer input tokens than chunk embedding and 81% fewer than full notes, and ran faster. Removing the initial NER step dropped F1 from 0.97 to 0.86, and BERT models fine-tuned on CLEAR labels performed within the range of the large models.
The limits
What it doesn't show
The accuracy advantage over alternatives is small for several models (e.g. Flan-T5: 0.91 vs 0.88), and with ten retrieved chunks the embedding method nearly caught up. All data come from one hospital, the opioid test set was split by date so it contains more COVID-era notes, and only information-extraction tasks were tested, not summarisation or reasoning. Chunk sizes and prompts were not fully tuned, so a better-configured embedding baseline might close the gap. The pipeline also depends on GPT-4 for entity filtering and synonym generation.
Key terms
- Retrieval-augmented generation (RAG)
- Giving a language model relevant passages fetched from a larger document collection, rather than the whole collection, before it answers.
- Named entity recognition (NER)
- Automatically finding mentions of things such as diseases or drugs in text.
- Context window
- The maximum amount of text a language model can take in at once.
- Zero-shot / few-shot prompting
- Asking a model to do a task with no worked examples, or with only a handful, instead of training it on labelled data.
- Model distillation
- Using a large model's outputs as labels to train a smaller, cheaper model to do the same job.
- F1 score
- The harmonic mean of precision and recall, balancing false positives and false negatives.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What does CLEAR use to decide which parts of a note to retrieve?
Common questions
Why not just give the LLM the whole note?
Notes can exceed the model's context window, and even when they fit, performance tends to drop as irrelevant text is added; in this study full-note prompting was both slower and less accurate than CLEAR.
Why did embedding-based retrieval do worse if it pulled similar text?
Retrieved text overlapped similarly in successes and failures, but the embedding model tended to rank the most useful chunk lower, so relevant passages were often not in the top few.
Is the NER step really necessary?
Yes for many variables: relying only on ontology and GPT-4 synonyms dropped average F1 from 0.97 to 0.86, because real notes use phrasings that synonym lists miss.
More on Clinical NLP