Machine Learning–Based Early Warning Systems for Acute Care Utilization During Systemic Therapy for Cancer
Ontario Institute for Cancer Research, Toronto, Ontario, Canada
Vector Institute, Toronto, Ontario, Canada
Search for other papers by Robert C. Grant in
Current site
Google Scholar
PubMed
Ontario Institute for Cancer Research, Toronto, Ontario, Canada
Search for other papers by Jiang Chen He in
Current site
Google Scholar
PubMed
Search for other papers by Ferhana Khan in
Current site
Google Scholar
PubMed
Search for other papers by Ning Liu in
Current site
Google Scholar
PubMed
Search for other papers by Sho Podolsky in
Current site
Google Scholar
PubMed
Search for other papers by Yosuf Kaliwal in
Current site
Google Scholar
PubMed
Search for other papers by Melanie Powis in
Current site
Google Scholar
PubMed
Search for other papers by Faiyaz Notta in
Current site
Google Scholar
PubMed
Odette Cancer Centre, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada
Search for other papers by Kelvin K.W. Chan in
Current site
Google Scholar
PubMed
Massachusetts Institute of Technology, Cambridge, Massachusetts
Search for other papers by Marzyeh Ghassemi in
Current site
Google Scholar
PubMed
Ontario Institute for Cancer Research, Toronto, Ontario, Canada
Search for other papers by Steven Gallinger in
Current site
Google Scholar
PubMed
ICES, Toronto, Ontario, Canada
Search for other papers by Monika K. Krzyzanowska in
Current site
Google Scholar
PubMed
Background: Emergency department visits and hospitalizations frequently occur during systemic therapy for cancer. We developed and evaluated a longitudinal warning system for acute care use. Methods: Using a retrospective population-based cohort of patients who started intravenous systemic therapy for nonhematologic cancers between July 1, 2014, and June 30, 2020, we randomly separated patients into cohorts for model training, hyperparameter tuning and model selection, and system testing. Predictive features included static features, such as demographics, cancer type, and treatment regimens, and dynamic features, such as patient-reported symptoms and laboratory values. The longitudinal warning system predicted the probability of acute care utilization within 30 days after each treatment session. Machine learning systems were developed in the training and tuning cohorts and evaluated in the testing cohort. Sensitivity analyses considered feature importance, other acute care endpoints, and performance within subgroups. Results: The cohort included 105,129 patients who received 1,216,385 treatment sessions. Acute care followed 182,444 (15.0%) treatments within 30 days. The ensemble model achieved an area under the receiver operating characteristic curve of 0.742 (95% CI, 0.739–0.745) and was well calibrated in the test cohort. Important predictive features included prior acute care use, treatment regimen, and laboratory tests. If the system was set to alarm approximately once every 15 treatments, 25.5% of acute care events would be preceded by an alarm, and 47.4% of patients would experience acute care after an alarm. The system underestimated risk for some treatment regimens and potentially underserved populations such as females and non-English speakers. Conclusions: Machine learning warning systems can detect patients at risk for acute care utilization, which can aid in preventive intervention and facilitate tailored treatment. Future research should address potential biases and prospectively evaluate impact after system deployment.
Submitted December 6, 2022; final revision received June 12, 2023; accepted for publication June 12, 2023.
Author contributions: Conceptualization: Grant, Ghassemi, Krzyzanowska. Data curation: Grant, He, Khan, Liu, Podolsky, Kaliwal, Powis. Formal analysis: Grant, He. Funding acquisition: Powis, Gallinger. Methodology: Grant, Ghassemi. Project administration: Grant, Chan, Krzyzanowska. Supervision: Grant, Notta, Chan, Ghassemi, Gallinger, Krzyzanowska. Writing—original draft: Grant. Writing—review and editing: All authors.
Data availability statement: Although legal data-sharing agreements between ICES and data providers prohibit ICES from making the dataset publicly available, ICES may grant access to those who meet prespecified criteria for confidential access through applications at www.ices.on.ca/DAS (email: das@ices.on.ca). The code used to generate these analyses is available at https://github.com/ml4oncology/ml4oncology/tree/main/PROACCT.
Disclosures: R.C. Grant has disclosed serving as a consultant for Incyte and Tempus; and receiving grant/research support from Pfizer. M. Powis has disclosed serving as a consultant for Ipsen, Eisai, Eli Lilly and Company, and Bayer; and receiving grant/research support from Eisai, Exelixis, and Eli Lilly and Company. M.K. Krzyzanowska has disclosed serving as a consultant for Ipsen, Eisai, Lilly, and Bayer; and receiving grant/research support from Eisai, Exelixis, and Lilly. The remaining authors have disclosed that they have not received any financial considerations from any person or organization to support the preparation, analysis, results, or discussion of this article.
Funding: Research reported in this publication was supported by the Ontario Institute for Cancer Research (R.C. Grant, S. Gallinger), the Department of Medicine at the University of Toronto (R.C. Grant), the Princess Margaret Cancer Foundation (R.C. Grant, S. Gallinger), and the Hold ‘Em for Life Oncology Fellowship through the Eliot Phillipson Clinical-Scientist Training Program (R.C. Grant).
Disclaimer: The analyses, conclusions, opinions, and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred.
Supplementary Materials
- Supplemental Materials (PDF 528 KB)
© 2019-2024 National Comprehensive Cancer Network