Can a fine-tuned open LLM assign hospital billing codes from notes?
A LLaMA model fine-tuned on discharge summaries predicted hospital diagnosis-related groups more accurately than earlier leading text classifiers, and got better with larger models and longer inputs.
Source
DRG-LLaMA : tuning LLaMA model to predict diagnosis-related group for hospitalized patients
Study at a glance
- Design
- Computational / modelling — LoRA fine-tuning of LLaMA 7B/13B for DRG classification from MIMIC-IV discharge summaries, 90/10 stratified split, compared with CAML and ClinicalBERT
- N
- N=236192 · 236,192 hospital stays in the MIMIC-IV training data the authors describe (vs. 17,815 in earlier MIMIC-III work); 738 distinct DRG labels
- Population
- Hospital stays with Medicare severity DRG codes in the MIMIC-IV database (Beth Israel Deaconess Medical Center)
- Outcome
- Top-1 and top-5 accuracy, macro-F1 and AUC for predicting the DRG code
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors took the 'brief hospital course' section of discharge summaries from the public MIMIC-IV database and fine-tuned LLaMA models using LoRA, a cheap method that trains small add-on matrices instead of all weights. They framed the job either as picking one of 738 DRG codes directly, or as predicting a base DRG and a complication-severity label separately and then combining them. They compared against two strong baselines, CAML (a convolutional network with attention) and ClinicalBERT.
What they found
The 7B model reached 52.0% top-1 and 84.8% top-5 accuracy and beat both baselines on every metric, with the biggest gain in macro-F1 (about 40% relative improvement over ClinicalBERT). The 13B model with 1024 input tokens did best, at 54.6% top-1 accuracy. The two-label approach gave nearly the same final accuracy (51.5%). Rare DRGs were predicted much worse than common ones, and surgical DRGs were easiest.
The limits
What it doesn't show
Discharge summaries only exist after the patient leaves, so this does not show the model can predict DRGs early in a stay, which is the operationally useful case. Everything comes from one hospital's data, so accuracy at other hospitals is untested. Compute limits meant models no larger than 13 billion parameters and no broad hyperparameter search, and the error analysis looked at only 10 hand-picked mistakes.
Key terms
- Diagnosis-related group (DRG)
- A billing category assigned to each hospital stay, based on the main diagnosis or procedure plus complications, that determines payment.
- LoRA
- Low-rank adaptation: fine-tuning a large model by freezing its weights and training small low-rank matrices added to each layer.
- Macro-F1
- The F1 score computed separately for every class and then averaged, so rare classes count as much as common ones.
- Top-k accuracy
- The share of cases where the correct label is among the model's k highest-ranked guesses.
- Single-label vs two-label classification
- Predicting the full code at once, versus predicting two component labels and combining them with rules.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was the top-1 accuracy of the 7B DRG-LLaMA model with 512 input tokens?
Common questions
Why is AUC high for every model even when accuracy is only around 50%?
With hundreds of rare classes, most predictions are easy true negatives, which inflates AUC; macro-F1 is a harsher and more informative measure here.
Does a bigger model help?
Yes within the range tested: moving from 7B to 13B parameters and from 512 to 1024 input tokens improved all metrics.
Why did the model fail on some codes?
Codes with few training examples were predicted poorly, and some errors came from missing information in the note or from the model missing key clinical concepts.
More on Language models