Skip to content
PaperFren

Language models

Can a fine-tuned open LLM assign hospital billing codes from notes?

Wang H, Gao C, Dantona C, et al. · NPJ digital medicine · 2024

Open access · cc by · source: Europe PMC

A LLaMA model fine-tuned on discharge summaries predicted hospital diagnosis-related groups more accurately than earlier leading text classifiers, and got better with larger models and longer inputs.

Save to libraryView originalAsk PaperFrenStudy view

Study at a glance

Design
Computational / modelling — LoRA fine-tuning of LLaMA 7B/13B for DRG classification from MIMIC-IV discharge summaries, 90/10 stratified split, compared with CAML and ClinicalBERT
N
N=236192 · 236,192 hospital stays in the MIMIC-IV training data the authors describe (vs. 17,815 in earlier MIMIC-III work); 738 distinct DRG labels
Population
Hospital stays with Medicare severity DRG codes in the MIMIC-IV database (Beth Israel Deaconess Medical Center)
Outcome
Top-1 and top-5 accuracy, macro-F1 and AUC for predicting the DRG code

Structured fields used in claim comparison tables when every cited study has a complete layer.

Key findings

The 7B model reached 52.0% top-1 and 84.8% top-5 accuracy and beat both baselines on every metric, with the biggest gain in macro-F1 (about 40% relative improvement over ClinicalBERT). The 13B model with 1024 input tokens did best, at 54.6% top-1 accuracy. The two-label approach gave nearly the same final accuracy (51.5%). Rare DRGs were predicted much worse than common ones, and surgical DRGs were easiest.

Methodology

The authors took the 'brief hospital course' section of discharge summaries from the public MIMIC-IV database and fine-tuned LLaMA models using LoRA, a cheap method that trains small add-on matrices instead of all weights. They framed the job either as picking one of 738 DRG codes directly, or as predicting a base DRG and a complication-severity label separately and then combining them. They compared against two strong baselines, CAML (a convolutional network with attention) and ClinicalBERT.

Limitations

Discharge summaries only exist after the patient leaves, so this does not show the model can predict DRGs early in a stay, which is the operationally useful case. Everything comes from one hospital's data, so accuracy at other hospitals is untested. Compute limits meant models no larger than 13 billion parameters and no broad hyperparameter search, and the error analysis looked at only 10 hand-picked mistakes.

How this study connects

Role on claims

Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.

  • SupportsDomain-specific language model pretrainingconcept

    Tuned LLMs reach parity, at much higher compute cost.

    Fine-tuning open LLMs on domain tasks can match or beat smaller specialised models: an instruction-tuned LLaMA matched three task-specific PubMedBERT models on biomedical NER (differences not significant), and DRG-LLaMA beat ClinicalBERT on diagnosis-related-group prediction (52.0% top-1 accuracy for 7B).

    Evidence for the claim as stated.

Related papers in this topic

Same topic cluster — not a recommendation engine.