In Journal of the American Medical Informatics Association : JAMIA
OBJECTIVE : To develop a machine learning framework to forecast emergency department (ED) crowding and to evaluate model performance under spatial and temporal data drift.
MATERIALS AND METHODS : We obtained four datasets, identified by the location: 1-large academic hospital and 2-rural hospital, and time period: pre-COVID (Jan 1, 2019-Feb 1, 2020) and COVID-era (May 15, 2020-Feb 1, 2021). Our primary target was a binary outcome that is equal to 1 if the number of patients with acute respiratory illness that were ED boarding for more than four hours was above a prescribed historical percentile. We trained a random forest and used the area under the curve (AUC) to evaluate out-of-sample performance for two experiments: 1) we evaluated the impact of sudden temporal drift by training models using pre-COVID data and testing them during the COVID-era, 2) we evaluated the impact of spatial drift by testing models trained at Location 1 on data from Location 2, and vice versa.
RESULTS : The baseline AUC values for ED boarding ranged from 0.54 (pre-COVID at Location 2) to 0.81 (COVID-era at Location 1). Models trained with pre-COVID data performed similarly to COVID-era models (0.82 vs. 0.78 at Location 1). Models that were transferred from Location 2 to Location 1 performed worse than models trained at Location 1 (0.51 vs. 0.78).
DISCUSSION AND CONCLUSION : Our results demonstrate that ED boarding is a predictable metric for ED crowding, models were not significantly impacted by temporal data drift, and any attempts at implementation must consider spatial data drift.
Smith Ari J, Patterson Brian W, Pulia Michael S, Mayer John, Schwei Rebecca J, Nagarajan Radha, Liao Frank, Shah Manish N, Boutilier Justin J
2022-Oct-29
COVID-19, Emergency medicine, data drift, emergency department boarding, machine learning
