Generalisation and overfitting
Does a surgery-death prediction model work at other hospitals?
Open access · cc by · source: Europe PMC
A lightweight gradient-boosted model using only routine lab values and basic surgical details predicted 30-day post-surgery death well and kept that accuracy when moved to other hospitals.
Study at a glance
- Design
- Computational / modelling — Retrospective EHR modelling: four classifiers trained per hospital on a small set of objective preoperative variables, then externally validated on the other three hospitals.
- N
- N=454404 · Adult non-cardiac surgery patients pooled across four hospitals (SNUH largest); each hospital split 6:2:2 into train/validation/test.
- Population
- Adults undergoing non-cardiac surgery at four hospitals in Seoul, South Korea
- Outcome
- In-hospital death within 30 days of surgery (AUROC, AUPRC)
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
XGBoost was usually best; at the largest hospital it reached an AUROC of 0.942 and an AUPRC of 0.175, and all machine-learning models beat the ASA score. Models from the hospitals with the most data transferred best, for example an AUROC of 0.941 when the largest hospital's model was tested on another site. Resampling to fix class imbalance mostly made performance worse. The most important predictor differed by hospital: albumin at two sites, age at one and prothrombin time at another.
Methodology
The researchers pulled electronic records for adults having non-cardiac surgery at four Korean hospitals and used only objective inputs that every hospital records automatically: age, sex, BMI, 12 preoperative lab tests, surgery type, anaesthesia type and emergency status. They trained logistic regression, random forest, XGBoost and a five-layer neural network at each hospital, compared them with the clinician-assigned ASA physical status score, and tested each hospital's model on the other hospitals' data.
Limitations
Deaths were very rare (0.2 to 0.4% per hospital), so precision-recall scores stayed low and the model would flag many patients who survive. All four hospitals are in one country, so transfer to different health systems is untested. The model slightly overestimated risk at some sites, and the authors did not try transfer techniques such as federated learning. Being retrospective, it does not show that using the model improves patient outcomes.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
External validation across sites reveals what internal testing cannot, and training-set size helps transfer.
In a retrospective study of about 454,000 surgical patients at four Korean hospitals, models trained at the hospitals with the most data transferred best to other sites (AUROC 0.941 when the largest hospital's model was tested elsewhere), and the most important predictor differed between hospitals.
Evidence for the claim as stated.
Related papers in this topic
Same topic cluster — not a recommendation engine.