Interpretability
Can machine learning build simple, readable hospital risk scores?
Open access · cc by · source: Europe PMC
An automated method produced a nine-item point score for predicting hospital death that was almost as accurate as a black-box random forest while being simple enough to add up by hand.
Study at a glance
- Design
- Computational / modelling — Retrospective model development on MIMIC-III ICU admissions with a 70/10/20 train/validation/test split, comparing AutoScore point scores with five baseline models.
- N
- N=44918 · 44,918 ICU admission episodes from one hospital (BIDMC) in MIMIC-III; 3958 died in hospital.
- Population
- Adult ICU admissions aged 18-90 at Beth Israel Deaconess Medical Center, 2001-2012
- Outcome
- Inpatient mortality; AUC, sensitivity, specificity and calibration on the held-out test set
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
A nine-variable point score reached an AUC of 0.780 and a 12-variable score 0.789, similar to full logistic regression (0.778, 24 variables) and LASSO (0.772, 17 variables). The full random forest was most accurate (AUC 0.809) but uninterpretable, and a nine-variable random forest (0.785) matched AutoScore closely. AutoScore's score was well calibrated, whereas the random forests calibrated worst.
Methodology
The AutoScore framework ranks candidate predictors with a random forest, keeps the top few, cuts continuous variables into quantile bands, fits logistic regression and rounds the coefficients into integer points. The number of variables is chosen from a 'parsimony plot' of validation performance against model size. The authors applied it to first-day vital signs and lab tests from ICU admissions and compared it with full logistic regression, stepwise regression, LASSO and two random forests on an unseen test set.
Limitations
All data came from one hospital's retrospective records, with no external or prospective validation, so the specific score is not ready for clinical use, as the authors say. Useful predictors such as interventions (e.g. intubation) were unavailable. The accuracy differences between models are small and overlap in their confidence intervals, and interpretability was argued from model simplicity rather than measured with clinicians.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Interpretability can cost little accuracy on tabular clinical data.
Simple point scores can come close to complex models: a nine-variable AutoScore reached AUC 0.780 vs 0.778 for 24-variable logistic regression, and only the full random forest was clearly more accurate (0.809) while being uninterpretable and worse calibrated.
Evidence for the claim as stated.
'Interpretable' means different things: point scores are interpretable by simplicity (not measured with clinicians), knowledge-guided networks by structure, and a robot navigation paper uses 'explainable' to mean reward terms have physical meaning, not that the policy was analysed.
Evidence for the claim as stated.
Open questions
Tensions this paper is part of
From concept pages' “where studies disagree.” Disagreement means the same question; scope means different assays, populations, or outcomes.
'Interpretable' means different things: point scores are interpretable by simplicity (not measured with clinicians), knowledge-guided networks by structure, and a robot navigation paper uses 'explainable' to mean reward terms have physical meaning, not that the policy was analysed.
Related papers in this topic
Same topic cluster — not a recommendation engine.