Background
Collecting, monitoring, and responding to patient-generated health data (PGHD) are associated with improved patient satisfaction and quality of life in oncology.1–7 They are also potentially associated with improved patient survival.8,9 Many electronic health record (EHR) and patient engagement vendors have begun to enable remote monitoring and electronic PGHD collection. Healthcare organizations can now collect vast amounts of PGHD electronically with the potential to inform predictive models. This includes direct data entry by patients and their caregivers. An additional potential benefit of this evolution includes improvements in data quality to support care team coordination and clinical research.10–13 For example, activity data from devices have the potential to inform performance status preoperatively, or for clinical trials eligibility screening and monitoring. The science around using PGHD has also matured, with prior work done to identify best practices for statistical analyses in controlled research studies.14
Questions remain, however, regarding operationalizing PGHD management in the clinic, such as (1) the optimal amount and type of PGHD to collect, (2) standardizing and visualizing actionable PGHD,15 (3) integrating PGHD within EHRs to support clinician workflow, (4) triaging appropriate alarm notifications to the care team, (5) mitigating patient “survey fatigue,” and (6) identifying which PGHD instruments confer realized benefits to patients being monitored. We report here on the current state of adoption and implementation of PGHD collection and monitoring across leading cancer centers in the United States. We discuss current gaps and recommendations for cancer centers, researchers, and EHR vendors.
Methods
NCCN formed a Patient-Reported Outcomes (PRO) Workgroup in 2019 to assess the current state of PGHD collection at NCCN Member Institutions and provide recommendations for integrating PGHD into EHRs. The workgroup comprises representatives from 11 of the 31 NCCN Member Institutions, who have expertise in EHR optimization. The PRO Workgroup operates under the auspices of the NCCN EHR Oncology Advisory Group, which serves as a forum for group members to share challenges and innovative practices regarding the optimization of EHR systems.
The PRO Workgroup initiated its charge by developing a conceptual model for PROs (Figure 1). For the purposes of this report, we refer to the broader concept of PGHD,13,16–18 a subset of which are PROs. PGHD consists of 5 structured features (collection method, content, setting, purpose, integration touchpoints) under the broader agency of governance. Governance of PGHD emerged as a necessary component of a mature health system’s approach to operationalizing the collection and monitoring of PGHD. We identified the following 4 key outputs from collecting and monitoring PGHD: improved patient outcomes, efficient hospital operations, increased EHR data quality and integrity, and enhanced patient safety. The PRO Workgroup used the PGHD conceptual model to guide the development of the surveys used in this report.
Patient-generated health data model.
Abbreviation: EHR, electronic health record.
Citation: Journal of the National Comprehensive Cancer Network 20, 13; 10.6004/jnccn.2021.7088
Patient-generated health data model.
Abbreviation: EHR, electronic health record.
Citation: Journal of the National Comprehensive Cancer Network 20, 13; 10.6004/jnccn.2021.7088
Patient-generated health data model.
Abbreviation: EHR, electronic health record.
Citation: Journal of the National Comprehensive Cancer Network 20, 13; 10.6004/jnccn.2021.7088
Our initial survey assessed the current state of PGHD collection at NCCN Member Institutions. It was piloted by 5 Member Institutions to ensure content accuracy and question clarity. In May 2019 the survey was distributed to all 28 (at the time) EHR Oncology Advisory Group members, using an electronic survey tool (SurveyMonkey). Each institution designated a single respondent responsible for completing the survey, and data were collected over a 4-week period.
Upon review of the initial survey responses, the PRO Workgroup requested a follow-up survey to gain a greater understanding of “point-of-care” (POC) collection, EHR integration of validated survey instruments, barriers to EHR integration, and governance structures. This second survey was administered in August 2019 and response data were collected over a 4-week period through SurveyMonkey.
Data from the surveys were presented to the NCCN EHR Oncology Advisory Group on October 21, 2019, and approved for publication.
Results
The full results are available in supplemental eTables 1 and 2, available with this article at JNCCN.org.
Survey 1: PGHD
A total of 23 of 28 NCCN Member Institutions responded to the survey (82% response rate). Some survey respondents did not answer all questions, and therefore the total number of responses varied slightly per question.
Collection
Of the respondents, 96% reported that their institutions collect PGHD, 95% use >1 method of collection, and 57% use ≥3 collection methods. The type and frequency distribution of PGHD collected is shown in Figure 2. The most common purposes for PGHD collection included active symptoms or events monitoring (86%) and screening (81%). The most widely used instrument to collect PGHD at NCCN Member Institutions was the Patient Health Questionnaire (PHQ-2 or PHQ-9; 85%). PGHD are collected within the research setting according to 81% of respondents, and in the standard-of-care setting according to 57%. A total of 67% indicated that PGHD are collected at POC and 62% reported the data are collected remotely.
Type of PGHD collected at NCCN Member Institutions (n=21).
Abbreviation: PGHD, patient-generated health data.
Citation: Journal of the National Comprehensive Cancer Network 20, 13; 10.6004/jnccn.2021.7088
Type of PGHD collected at NCCN Member Institutions (n=21).
Abbreviation: PGHD, patient-generated health data.
Citation: Journal of the National Comprehensive Cancer Network 20, 13; 10.6004/jnccn.2021.7088
Type of PGHD collected at NCCN Member Institutions (n=21).
Abbreviation: PGHD, patient-generated health data.
Citation: Journal of the National Comprehensive Cancer Network 20, 13; 10.6004/jnccn.2021.7088
Integration
Among responding centers, 90% embed at least some PGHD into the EHR, whereas only 4 indicated that all PGHD are included in the EHR. A total of 61% reported that the center’s clinical decision support system alerts the care team about concerning responses or scores above or below a threshold, and 67% reported that their care teams receive referrals or activities assigned to them based upon PGHD responses.
Governance
Responses regarding the degree to which centers have a governance process are shown in Figure 3. In each of the 3 governance areas, only 33% to 43% reported having a fully implemented governance process (Figure 3). A total of 48% responding centers reported that patients participate in the design of PGHD collection methods and processes.
Degree to which centers have a governance body or process (n=21).
Abbreviation: PGHD, patient-generated health data.
Citation: Journal of the National Comprehensive Cancer Network 20, 13; 10.6004/jnccn.2021.7088
Degree to which centers have a governance body or process (n=21).
Abbreviation: PGHD, patient-generated health data.
Citation: Journal of the National Comprehensive Cancer Network 20, 13; 10.6004/jnccn.2021.7088
Degree to which centers have a governance body or process (n=21).
Abbreviation: PGHD, patient-generated health data.
Citation: Journal of the National Comprehensive Cancer Network 20, 13; 10.6004/jnccn.2021.7088
Survey 2: PGHD Follow-Up
This survey was distributed to the 23 NCCN Member Institutions that completed survey 1, and 17 responded. Some survey respondents did not answer all questions, and therefore the total number of responses varied slightly per question.
Collection
Respondents indicated that POC collection of PGHD occurs primarily in the outpatient waiting room (94%), but also in the outpatient examination room (41%) and the outpatient infusion room (18%), and one center collects PGHD at POC in the inpatient room (6%). Centers varied in the percent of PGHD that is collected at POC versus remotely/at home, with 75% of responding centers collecting ≥60% of PGHD at the POC, and 3 indicating that 100% of PGHD are collected at POC.
Integration
Most centers indicated that PGHD are integrated into their EHR via different methods across their center, and several centers used multiple methods. A total of 88% of centers reported that some PGHD electronically integrates into the EHR as discrete data; however, 29% of those centers also have some PGHD pushed into the EHR as consolidated information in one document. Additionally, 53% of responding centers still scan paper documents containing PGHD into the EHR (without discrete or summary integration). Although almost half (47%) indicated PGHD trigger automatic actions in the EHR based on thresholds, 88% of centers also use validated PGHD instruments that are not integrated into the EHR.
Governance
Responding centers described PGHD governance structures that vary widely and feature unique staffing models. Examples include departmental governance led by administrative and physician leaders; oncology stakeholder governance group lead by physicians, directors, disease line administrators, registered nurses, and information technologists; and oversight/advisory committee staffed by executive operations and research leadership with additional multidisciplinary membership. One center reported efforts to decentralize a steering group, whereas another reported efforts to centralize for an entire university health system.
Discussion
NCCN has conducted the first assessment of the state of the science regarding PGHD collection and management across its Member Institutions. This study extends prior work by Zhang et al19 in 2015 with additional insights, both for uptake and integration. Our study focused on a different population of oncology centers (NCCN Member Institutions; n=28), whereas Zhang et al focused on Quality Oncology Practice Initiative (QOPI) groups (11 of 28 NCCN Member Institutions are QOPI-certified). Like Zhang et al, we confirmed that cancer centers are using different types of PGHD (not just symptoms or outcomes data), thus forming the basis of our recommendation for a more inclusive conceptual model for PGHD that is not limited to PROs alone (Figure 1). We also observed a higher percentage of cancer centers collecting PGHD (96% vs 69%). This may reflect temporal trends since 2015, or more rapid uptake among comprehensive cancer centers specifically, or other factors not measured in this analysis. We observed a different distribution of survey types, including all the types reported by Zhang et al, plus social history, family history, social determinants of health, medications, preferences and values, implanted devices, and images. Our study demonstrated a higher rate of collection electronically, and at home in between visits. Our extended model of PGHD could be of value to various standards organizations and researchers who are working to draw insights from PGHD, or for comparative effectiveness studies.
Our major finding is that adoption of the collection of PGHD is widespread among survey respondents (96%). NCCN Member Institutions currently collect a wide variety of PGHD content types, and it varies by institution (Figure 2). Ninety percent of respondents reported inclusion of at least some of their data into the EHR, although methods to store and visualize PGHD in the EHR varied from discrete data to scanned documents, and varied within each respondent’s center based on what PGHD were being collected. Challenges remain, as evidenced by 88% of respondents reporting the use of numerous instruments that are not integrated into the EHR and only 4 of the responding centers having all PGHD included in the EHR. Given that, according to Basch et al,8,9 the evidence for PGHD collection is potentially associated with improved survival only if also monitoring and responding to these data, how one ingests and visualizes PGHD may be important. Subjective comments suggest factors such as copyright regulations, competing priorities, and technical challenges impede centers’ ability to integrate all PGHD into their commercial EHR.
A significant percentage of institutions (47%) are leveraging these data for process automation and adherence to best evidence, including generating automated EHR referral orders from the data (such as for psychosocial referrals for positive PHQ-2 or PHQ-9 scores, or smoking cessation). This has the potential to add the patient as an active participant in the care team’s efforts to adhere to best practices, and to reduce clinical burden through automated triage and referrals (though we recognize this could also increase unwanted referrals from the patients’ perspectives).
There is wide variability in the maturity and composition of PGHD governance models, as is shown in Figure 3. Even at comprehensive cancer centers that have widely adopted PGHD, most respondents self-reported that they have not yet fully implemented a governance structure to determine what to collect and how to incorporate PGHD into clinical workflows, and to identify secondary use cases. Fewer than half of the responding centers include patients in the governance process.
The reported variability in data management, EHR integration, and governance suggests that PGHD may not yet have reached its full potential for advanced oncology care delivery. In narrative comments, centers described oncology-specific challenges they have encountered, such as intradepartmental agreement, standardization, governance, personnel support, communication with and identification of eligible patients, volume and timing of surveys, integration into the clinic workflow, display format in EHR, interpretation of results, and ensuring clinical follow-up of PGHD.
These findings represent opportunities for the development of new knowledge regarding required competencies that organizations should employ as they continue to develop and expand their collection and utilization of PGHD. Recommendations for how cancer centers, researchers, and EHR vendors can optimize the collection, integration, and visualization of PGHD are summarized in Table 1.20–27
PGHD Recommendations for Cancer Centers, Researchers, and EHR Vendors
A limitation of our study was not fully assessing the current state of how PGHD are used in clinical care. Although we did ask centers if the EHR triggers alerts and referrals in response to the collected PGHD, further study is needed to understand other ways PGHD are used in clinical care, including whether it is used to facilitate shared decision-making between providers and patients. We also did not obtain the technical or operational specifics regarding how those centers that have had the most success with PGHD integration have accomplished this, meriting further analysis. Additional limitations include not assessing where PGHD responses are being recorded within the EHR (ie, as part of the billable provider note, a separate document, or some combination of both). Furthermore, investigation is needed to understand how centers fund and support their PGHD collection and monitoring activities.
Conclusions
The collection and use of PGHD is widespread among NCCN Member Institutions. Content type and integration touchpoints vary among the members (Figure 2). Governance maturity varies among NCCN Member Institutions (Figure 3). As the adoption of PGHD in oncology continues to expand, opportunities exist to enhance the collection and monitoring process, standardize PGHD instruments, integrate PGHD into vendor designed EHR workflows, develop governance structures, and maximize the state of science around realizing value from PGHD.
Acknowledgements
The authors wish to thank the NCCN EHR Oncology Advisory Group, which is comprised of clinical leaders who oversee the optimization of EHR systems at their respective NCCN Member Institutions.
References
- 1.↑
Basch E, Deal AM, Kris MG, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 2016;34:557–565.
- 2.↑
Kotronoulas G, Kearney N, Maguire R, et al. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol 2014;32:1480–1501.
- 3.↑
Basch E, Artz D, Dulko D, et al. Patient online self-reporting of toxicity symptoms during chemotherapy. J Clin Oncol 2005;23:3552–3561.
- 4.↑
Berry DL, Blumenstein BA, Halpenny B, et al. Enhancing patient-provider communication with the electronic self-report assessment for cancer: a randomized trial. J Clin Oncol 2011;29:1029–1035.
- 5.↑
Chen J, Ou L, Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res 2013;13:211.
- 6.↑
Detmar SB, Muller MJ, Schornagel JH, et al. Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial. JAMA 2002;288:3027–3034.
- 7.↑
Velikova G, Booth L, Smith AB, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol 2004;22:714–724.
- 8.↑
Basch E, Deal AM, Dueck AC, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 2017;318:197–198.
- 9.↑
Denis F, Basch E, Septans AL, et al. Two-year survival comparing web-based symptom monitoring vs routine surveillance following treatment for lung cancer. JAMA 2019;321:306–307.
- 10.↑
Barkley R, Khalil M, Shen P, et al. Feasibility of low-cost accelerometers in measuring functional recovery after major oncologic surgery [published online November 28, 2019]. J Surg Oncol, doi: 10.1002/jso.25789
- 11.↑
Di Meglio A, Michiels S, Jones LW, et al. Changes in weight, physical and psychosocial patient-reported outcomes among obese women receiving treatment for early-stage breast cancer: a nationwide clinical study. Breast 2020;52:23–32.
- 12.↑
Hsueh PY, Cheung YK, Dey S, et al. Added value from secondary use of person generated health data in consumer health informatics. Yearb Med Inform 2017;26:160–171.
- 13.↑
Wood WA, Bennett AV, Basch E. Emerging uses of patient generated health data in clinical research. Mol Oncol 2015;9:1018–1024.
- 14.↑
Coens C, Pe M, Dueck AC, et al. International standards for the analysis of quality-of-life and patient-reported outcome endpoints in cancer randomised controlled trials: recommendations of the SISAQOL Consortium. Lancet Oncol 2020;21:e83–96.
- 15.↑
Grossman LV, Mitchell EG. Visualizing the Patient-Reported Outcomes Measurement Information System (PROMIS) measures for clinicians and patients. AMIA Annu Symp Proc 2018;2017:2289–2293.
- 16.↑
Cohen DJ, Keller SR, Hayes GR, et al. Integrating patient-generated health data into clinical care settings or clinical decision-making: lessons learned from Project HealthDesign. JMIR Human Factors 2016;3:e26.
- 17.↑
Jung SY, Kim JW, Hwang H, et al. Development of comprehensive personal health records integrating patient-generated health data directly from Samsung S-Health and Apple Health apps: retrospective cross-sectional observational study. JMIR Mhealth Uhealth 2019;7:e12691.
- 18.↑
Ancker JS, Mauer E, Kalish RB, et al. Early adopters of patient-generated health data upload in an electronic patient portal. Appl Clin Inform 2019;10:254–260.
- 19.↑
Zhang B, Lloyd W, Jahanzeb M, et al. Use of patient-reported outcome measures in quality oncology practice initiative-registered practices: results of a national survey. J Oncol Pract 2018;14:e602–611.
- 20.↑
Gensheimer SG, Wu AW, Snyder CF, et al. Oh, the places we’ll go: patient-reported outcomes and electronic health records. Patient 2018;11:591–598.
- 21.↑
Belenkaya R, Gurley M, Dymshyts D, et al. Standardized observational cancer research using the OMOP CDM oncology module. Stud Health Technol Inform 2019;264:1831–1832.
- 22.↑
Young-Afat DA. Patient-reported outcomes in routine care-a true innovation but only if used correctly. JAMA Oncol 2019;5:1258–1260.
- 23.↑
Centers for Medicare and Medicaid Services. MACRA funding opportunity: measure development awardees for the Quality Payment Program. Accessed April 30, 2021. Available at: https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/MACRA-MIPS-and-APMs/9-21-18-QPP-Measures-Cooperative-Agreement-Awardees.pdf
- 24.↑
Purchaser Business Group on Health. Patient-reported outcomes – oncology. Accessed April 30, 2021. Available at: https://www.pbgh.org/component/content/article/526
- 25.↑
Electronic Symptom Management (eSyM). What is eSyM? Accessed April 30, 2021. Available at: https://www.esymcancermoonshot.org/about-esym
- 26.↑
Raths D. Health systems study EHR-based symptom management tool. Healthcare Innovation website. Accessed April 30, 2021. Available at: https://www.hcinnovationgroup.com/clinical-it/patient-portals/article/21111205/health-systems-study-ehrbased-symptom-management-tool
- 27.↑
Rodriguez JA, Clark CR, Bates DW. Digital health equity as a necessity in the 21st Century Cures Act era. JAMA 2020;323:2381–2382.
