Does a surgery-death prediction model work at other hospitals?
A lightweight gradient-boosted model using only routine lab values and basic surgical details predicted 30-day post-surgery death well and kept that accuracy when moved to other hospitals.
Source
Multi-center validation of machine learning model for preoperative prediction of postoperative mortality
Study at a glance
- Design
- Computational / modelling — Retrospective EHR modelling: four classifiers trained per hospital on a small set of objective preoperative variables, then externally validated on the other three hospitals.
- N
- N=454404 · Adult non-cardiac surgery patients pooled across four hospitals (SNUH largest); each hospital split 6:2:2 into train/validation/test.
- Population
- Adults undergoing non-cardiac surgery at four hospitals in Seoul, South Korea
- Outcome
- In-hospital death within 30 days of surgery (AUROC, AUPRC)
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The researchers pulled electronic records for adults having non-cardiac surgery at four Korean hospitals and used only objective inputs that every hospital records automatically: age, sex, BMI, 12 preoperative lab tests, surgery type, anaesthesia type and emergency status. They trained logistic regression, random forest, XGBoost and a five-layer neural network at each hospital, compared them with the clinician-assigned ASA physical status score, and tested each hospital's model on the other hospitals' data.
What they found
XGBoost was usually best; at the largest hospital it reached an AUROC of 0.942 and an AUPRC of 0.175, and all machine-learning models beat the ASA score. Models from the hospitals with the most data transferred best, for example an AUROC of 0.941 when the largest hospital's model was tested on another site. Resampling to fix class imbalance mostly made performance worse. The most important predictor differed by hospital: albumin at two sites, age at one and prothrombin time at another.
The limits
What it doesn't show
Deaths were very rare (0.2 to 0.4% per hospital), so precision-recall scores stayed low and the model would flag many patients who survive. All four hospitals are in one country, so transfer to different health systems is untested. The model slightly overestimated risk at some sites, and the authors did not try transfer techniques such as federated learning. Being retrospective, it does not show that using the model improves patient outcomes.
Key terms
- External validation
- Testing a model on data from a different source than it was trained on, to check it generalizes.
- AUROC
- Probability the model ranks a random death above a random survivor; 0.5 is chance, 1 is perfect.
- AUPRC
- Area under the precision-recall curve; low when positives are rare, making it a harsher metric for imbalanced data.
- XGBoost
- An ensemble of gradient-boosted decision trees, often strong on tabular data.
- Class imbalance
- When one outcome is far rarer than the other, which can make models and metrics misleading.
Flashcards
0 of 9 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was the prediction target?
Common questions
Why is AUROC high but AUPRC low?
Because death is so rare, even a good ranking produces many false alarms relative to true deaths, which precision-recall metrics penalize.
Did deep learning beat simpler models?
No; XGBoost generally performed best, and the five-layer neural network was not consistently better than logistic regression.
What made the model transferable?
Using only objective, automatically recorded variables and training on a large dataset; bigger training hospitals produced models that transferred best.
More on Generalisation and overfitting