In Journal of the American College of Surgeons ; h5-index 61.0
BACKGROUND : The ACS NSQIP risk calculator (RC) uses regression to make predictions for 14, 30-day surgical outcomes. While this approach provides accurate (discrimination and calibration) risk estimates, they might be improved by machine learning (ML). To investigate this possibility, accuracy for regression-based risk estimates were compared to estimates from an extreme gradient boosting (XGB) ML algorithm.
METHODS : A cohort of 5,020,713 million NSQIP patient records was randomly divided into 80% for model construction and 20% for validation. Risk predictions using regression and XGB-ML were made for 13 RC binary 30-day surgical complications and 1 continuous outcome (length of stay, LOS). For the binary outcomes, discrimination was evaluated using AUROC (area under the receiver operating characteristic curve) and AUPRC (area under the precision recall curve), and calibration was evaluated using Hosmer-Lemeshow (H-L) statistics. Mean squared error (MSE) and a calibration curve analog were evaluated for the continuous LOS outcome.
RESULTS : For every binary outcome, discrimination (AUROC and AUPRC) was slightly greater for XGB-ML than for regression (mean [across the outcomes] AUROC was 0.8299 versus 0.8251, and mean AUPRC was 0.1558 versus 0.1476, for XGB-ML and regression, respectively). For each outcome miscalibration was greater (larger H-L values) with regression; there was statistically significant miscalibration for all regression-based estimates but only for 4 of 13 when XGB-ML was used. For LOS, MSE was lower for XGB-ML.
CONCLUSIONS : XGB-ML provided more accurate risk estimates than regression in terms of discrimination and calibration. Differences in calibration between regression and XGB-ML were of substantial magnitude and support transitioning the RC to XGB-ML.
Liu Yaoming, Ko Clifford Y, Hall Bruce L, Cohen Mark E
2023-Jan-12
