Can machine learning build simple, readable hospital risk scores?
An automated method produced a nine-item point score for predicting hospital death that was almost as accurate as a black-box random forest while being simple enough to add up by hand.
Source
AutoScore: A Machine Learning-Based Automatic Clinical Score Generator and Its Application to Mortality Prediction Using Electronic Health Records
Study at a glance
- Design
- Computational / modelling — Retrospective model development on MIMIC-III ICU admissions with a 70/10/20 train/validation/test split, comparing AutoScore point scores with five baseline models.
- N
- N=44918 · 44,918 ICU admission episodes from one hospital (BIDMC) in MIMIC-III; 3958 died in hospital.
- Population
- Adult ICU admissions aged 18-90 at Beth Israel Deaconess Medical Center, 2001-2012
- Outcome
- Inpatient mortality; AUC, sensitivity, specificity and calibration on the held-out test set
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The AutoScore framework ranks candidate predictors with a random forest, keeps the top few, cuts continuous variables into quantile bands, fits logistic regression and rounds the coefficients into integer points. The number of variables is chosen from a 'parsimony plot' of validation performance against model size. The authors applied it to first-day vital signs and lab tests from ICU admissions and compared it with full logistic regression, stepwise regression, LASSO and two random forests on an unseen test set.
What they found
A nine-variable point score reached an AUC of 0.780 and a 12-variable score 0.789, similar to full logistic regression (0.778, 24 variables) and LASSO (0.772, 17 variables). The full random forest was most accurate (AUC 0.809) but uninterpretable, and a nine-variable random forest (0.785) matched AutoScore closely. AutoScore's score was well calibrated, whereas the random forests calibrated worst.
The limits
What it doesn't show
All data came from one hospital's retrospective records, with no external or prospective validation, so the specific score is not ready for clinical use, as the authors say. Useful predictors such as interventions (e.g. intubation) were unavailable. The accuracy differences between models are small and overlap in their confidence intervals, and interpretability was argued from model simplicity rather than measured with clinicians.
Key terms
- Point-based risk score
- A model where each predictor category adds a whole number of points, and the total indicates risk.
- AUC (area under the ROC curve)
- The probability that the model ranks a random positive case above a random negative one; 0.5 is chance, 1 is perfect.
- Parsimony
- Using as few variables as possible while keeping good predictive accuracy.
- Calibration
- How closely predicted risks match the observed rates of the outcome.
- LASSO
- Regression with a penalty that shrinks some coefficients to exactly zero, performing variable selection.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What is the final output of AutoScore?
Common questions
If the full random forest had the highest AUC, why not use it?
It uses 100 trees and cannot easily be explained or used at the bedside, and it was the worst calibrated; AutoScore gave up a little accuracy for a transparent, well-calibrated score.
How was the number of variables chosen?
By plotting validation-set performance against the number of variables and stopping where adding more no longer clearly helped, which gave nine.
Why turn continuous values into categories?
Banding lets the score capture U-shaped risks, such as both low and high temperature being dangerous, and makes each band a simple point value.
More on Interpretability