Background
Oral anticancer agents (OAAs) have transformed the treatment of certain once-fatal malignancies by extending survival and delaying disease progression.1 OAAs have rapidly emerged since they were first introduced, and now comprise >50% of available chemotherapies. OAAs are convenient and offer a more targeted therapy with fewer adverse effects.2 However, near perfect adherence to OAAs (ie, adherence rates >90%) as well as the ability to manage disease and adverse effects of treatment are required for patients to receive the full benefit of these medications.3,4 Many patients do not achieve this level of adherence; in clinical practice, adherence rates drop to approximately 40%, contributing to poor clinical outcomes and increased health care costs.5,6 Innovative models to improve patient knowledge and adherence to OAAs have been previously described.7–9 However, many of them have not been sustainable, with a critical limitation being a lack of clear strategies to improve their adoption, implementation, and maintenance.
Key performance indicators (KPIs) are quantifiable measures of quality used to track the success of programs, including those focused on health promotion. They have rapidly emerged in health care as measures of achievement for organizational objectives.10,11 Benefits of KPIs include enhanced transparency, accountability, and quality of care.12 Additionally, hospital managers and staff perceive KPIs to be advantageous for improving patient and employee safety, operational efficiency, and financial effectiveness.13 Previously reported barriers to the implementation of a structured adherence intervention have included a lack of standardization in documentation and a lack of clarity regarding the responsibilities of clinicians involved in the program.14,15 KPIs offer the optimal method for addressing these barriers for several reasons. First, KPIs create a defined list of expectations and responsibilities for clinicians, which has been associated with improved clarity in roles and improved outcomes. Next, KPIs utilize collected data to help identify program strengths and weaknesses, facilitating clear feedback for everyone involved in the program. Feedback provided by KPIs is associated with improved performance of employees as well as improved performance of entire programs, thereby improving patient care.10,16 Finally, KPIs can help identify gaps in the data documentation process, leading to increased standardization of documentation.
Although KPIs provide valuable feedback regarding program success, limited guidance exists on recommended KPIs for an OAA adherence program. The objective of this study was to identify potential KPIs, corresponding metrics of these KPIs, and target values that indicate improved patient care by an oral chemotherapy adherence program.
Methods
Members of the study team conducted a narrative review of the literature to compile an initial list of KPIs, metrics of the KPIs, and targets for success. Proposed targets for success were separated into 5 categories, defined by “does not meet,” “below standard,” “achieves,” “exceeds,” and “stretch.” An advisory panel with clinicians (n=12), administrators (n=7), and patients (n=2) in 2 settings (an academic and a community cancer center) was assembled to refine the initial list of KPIs, metrics, and targets for success using a 2-round consensus-building process. The clinicians and administrators had extensive experience (≥5 years for most) delivering care or managing units that delivered care to patients receiving OAAs. Of note, the advisory panel was assembled from an integrated health system that has academic and community cancer centers and a medically integrated dispensing specialty pharmacy. Further details on the members of the advisory panel are provided in Table 1.
Members of the Advisory Panel
The first round provided an opportunity for panel members to provide quantitative feedback via a web-based Qualtrics survey. The survey used a 5-point Likert scale with responses ranging from 1 (strongly disagree) to 5 (strongly agree) to evaluate (1) the respondent’s support of the proposed KPI (eg, should time to intended OAA initiation be used as a measure to indicate the success of an adherence program?), (2) the metric for the KPI (eg, should “days” be used as a metric for the KPI of time to OAA initiation?), and (3) the target for each metric (eg, should 3–5 days be an appropriate target for the KPI of time to OAA initiation?). Two open-ended questions were used to allow participants to elaborate on their choices. Participants were also given the opportunity to propose additional KPIs for inclusion. Refinements were made to the KPIs based on participant feedback before the second round.
The second round involved a live, web-based panel discussion to obtain qualitative feedback and final consensus on the KPIs. Before the discussion, the list of updated KPIs based on feedback from the Qualtrics survey was emailed to each participant. During the consensus meeting, each KPI, metric, and list of targets was presented along with the alternative definitions proposed from the first round. After each KPI was presented, discussion was opened to the group until final consensus was reached. Panel discussion was audio recorded, and results from both rounds were used to consolidate measurable KPIs and optimal target values for success.
Results
From the narrative review, we identified an initial list of 12 KPIs (Table 2), metrics of each KPI, and predetermined targets for proposal to the advisory panel. The KPIs were developed by taking into account the importance of key pillars that impact outcomes for patients receiving OAAs, including medication adherence, treatment complications, cost of therapy, and patient satisfaction. During the first round, 8 participants completed the survey gauging agreeability on these 12 KPIs. During the second round, 9 participants participated in the panel discussion to refine the list of KPIs based on results from the first round. After consensus-based discussion, a final list of 11 KPIs (7 clinical indicators, 3 process indicators, and 1 economic indicator) were agreed upon as appropriate for inclusion in an oral chemotherapy program: (1) time to intended OAA initiation (clinical), (2) adherence rate during active treatment (ie, implementation phase of treatment17) (clinical), (3) adverse events (clinical), (4) medication-related financial toxicity (clinical), (5) patient satisfaction (clinical), (6) treatment-related emergency department (ED) visits (clinical), (7) treatment-related hospital admissions (clinical), (8) proportion of patients with adherence, toxicity, and financial barriers assessed (process), (9) proportion of patients referred to social work (process), (10) time spent by patient in each phase of care as defined by the intervention’s standard operating procedure (process), and (11) revenue generated by billing for service (economic). The final list of these KPIs, their metrics, and targets for success is provided in Table 3.
Key Performance Indicators From Narrative Literature Review
Key Performance Indicators After Survey and Consensus-Based Discussion
Key Performance Indicators
Time to Intended OAA Initiation
Time to OAA initiation is an important aspect of adherence because it accounts for preliminary barriers that prevent patients from starting their medication.17 This KPI was initially defined as the time to initiation of OAA (in days) from the date the prescription is dispensed. Survey results from the first round of the process indicated strong agreement on inclusion of this KPI (4.88±0.33), the metric (4.38±0.86), and proposed targets of success (4.14±0.99). During the focus group discussion, the metric of the KPI was changed to time from intended start date to actual start date using dates as defined in the electronic health record (EHR), because the previous definition did not account for prescribers instructing patients to delay initiation of their medication. An intended start date for medications needed as soon as possible was not agreed upon. Therefore, the group agreed to make this KPI exploratory until further validation.
Adherence Rate
A high rate of adherence to OAAs is correlated with improved patient outcomes.3,4 This KPI was defined as the percentage of scheduled doses of OAA taken by the patient based on prescription fill history using the proportion of days covered (PDC) calculation18 when available, and supplemented with a validated self-reported adherence instrument using a Wilson 3-item measure (of note, validation was in an HIV and not a cancer setting).19 Survey results from the first round of the process indicated strong agreement on inclusion of this KPI (4.88±0.33), the metric (4.50±0.50), and proposed targets of success (4.38±0.48). During the focus group discussion, the group agreed to using previously published data when converting the Wilson 3-item measure to an adherence rate when PDC was unavailable. There was consensus on all aspects of this KPI.
Adverse Events
OAA-related adverse events are a common cause of poor adherence, and therefore an important monitoring parameter for an OAA adherence program.20 The adverse event KPI was defined as the number of self-reported doses of OAA not taken by the patient due to adverse effects in the last 30 days. These missed doses did not include doses intentionally held by a clinician. Survey results from the first round of the process indicated strong agreement on inclusion of this KPI (4.71±0.45), the metric (4.25±0.97), and proposed targets of success (4.43±0.49). During the focus group discussion, there was consensus on all aspects of this KPI.
Medication-Related Financial Toxicity
The expense of long-term targeted OAAs may lead to substantial financial burden and associated decreased OAA adherence.20 The medication-related financial toxicity KPI was adapted from the Patient Satisfaction Questionnaire Short-Form (PSQ-18)21 and aimed to monitor this financial burden to quickly address this barrier to adherence. The proposed KPI required reading the following statement to patients: “I have to pay more for my oral chemotherapy than I can afford,” and grading their response using a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Survey results from the first round of the process indicated varying levels of agreement on inclusion of this KPI (4.63±0.48), the metric (4.38±1.32), and proposed targets of success (3.71±0.88). During the focus group discussion, the wording of this question was refined to “I delayed or did not fill my oral chemotherapy prescription due to cost or lack of insurance coverage” to avoid phrasing that may elicit false-positive responses. After this change, there was consensus on all aspects of this KPI.
Patient Satisfaction
Patient satisfaction is a regularly used indicator for quality of care.22 This KPI was adapted from the PSQ-18.21 The proposed KPI measured agreement with the following statement: “I am dissatisfied with some of the service I received for this oral chemotherapy program.” Survey results from the first round of the process indicated varying levels of agreement on inclusion of this KPI (4.71±0.45), the metric (3.86±1.25), and proposed targets of success (3.71±1.03). During the focus group discussion, the wording of this question was refined to “How likely would you be to recommend this program to a newly diagnosed patient?” to prevent bias in response due to the negative tone of the question. It was agreed that “this program” would be replaced with the officially agreed upon program name at launch to further clarify the intent of the question. There was consensus on all updated aspects of this KPI.
Treatment-Related ED Visits
Treatment-related ED visits are not only distressing for patients but also costly, ranging on average from $574 to $918 per visit.23,24 Minimizing treatment-related emergencies is imperative to helping patients maintain adherence and treatment with OAAs. This KPI was defined as the number of visits to the ED in the past 90 days due to problems related to a patient’s OAAs. Initially, the target for achieves was set as 0 to 1 ED visits. Survey results from the first round of the process indicated strong agreement on inclusion of this KPI (4.57±0.73) and the metric (4.57±0.73). However, there was lower agreement on proposed targets of success (3.83±1.21), with the main concern being that the target for achieves should be 0. Therefore, the target for achieves was changed to 0 ED visits. During focus group discussions, there was consensus on all aspects of the updated KPI. NCI attribution guidelines were agreed upon for classifying whether the OAA was the cause of the ED visit, and the KPI will be exploratory until this incorporation is validated.
Treatment-Related Hospitalizations
Similar to ED visits, treatment-related hospitalizations are expensive and complicate treatment for patients receiving OAAs.23,24 This KPI was defined as the number of hospitalizations in the past 180 days due to problems related to a patient’s OAAs. The target for achieves was initially set as 0 to 1 hospitalizations. Survey results from the first round of the process indicated strong agreement on inclusion of this KPI (4.83±0.37) and the metric (4.50±0.50). However, there was lower agreement on proposed targets of success (3.50±1.26), with the main concern being that the target for achieves should be 0. Therefore, the target for achieves was updated to 0 hospitalizations. During focus group discussions, there was consensus on inclusion of the KPI, the decision to rely on NCI attribution guidelines for determining whether the hospitalization was medication-related, and on making the KPI exploratory initially until further validated.
Proportion of Patients With Adherence, Toxicity, and Financial Barriers Assessed
The KPI for the proportion of patients with adherence, toxicity, and financial barriers assessed was designed to ensure that these items are being assessed at each patient visit. Survey results for this KPI indicated strong agreement on inclusion of this KPI (4.83±0.37), the metric (4.86±0.35), and proposed targets of success (4.43±0.73). During the focus group discussion, there was consensus on all aspects of this KPI.
Proportion of Patients Referred to Social Work
The proportion of patients referred to social work was intended to be an exploratory KPI designed to measure the interdisciplinary nature of the adherence program. Survey results indicated strong agreement on inclusion of this KPI (4.33±0.75), the metric (4.60±0.80), and the exploratory nature of the KPI (4.00±1.00). There was concern regarding whether a target for success could ever be set. During the focus group discussion, there was consensus on including the KPI for observational purposes.
Time Spent by the Patient in Each Phase of the Standard Operating Procedure
The standard operating procedure (SOP) for this program is divided into 3 phases: initiation phase (eg, addressing barriers to adherence before the start of OAA therapy), early-treatment phase (eg, initiating OAAs and helping patients initially achieve high adherence), and maintenance phase (eg, helping patients maintain high adherence established in earlier phases). The proportion of patients followed up in accordance with the SOP for the OAA adherence program was intended to measure how broadly the SOP is known and how closely it is followed. Survey results indicated strong agreement on inclusion of this KPI (4.43±1.05), the metric (4.57±1.05), and the exploratory nature of the KPI (4.29±0.88). During focus group discussions, there was consensus on all aspects of this KPI.
Revenue Generated by Billing for Service
Pharmacist billing for service was proposed as an exploratory KPI to measure the financial benefit of the adherence program. It was defined as the total revenue generated by patient appointments within the program. Survey results indicated strong agreement on inclusion of this KPI (4.00±0.76), the metric (4.00±0.76), and the exploratory nature of the KPI (4.14±0.83). During the focus group discussion, there was consensus on all aspects of this KPI.
Discussion
Using a systematic approach and an interprofessional advisory panel, our team identified 11 KPIs, metrics, and target values that can be utilized to evaluate the success of an oral chemotherapy adherence intervention. Consensus on these KPIs was achieved through both quantitative surveys and qualitative focus group discussions. These KPIs allow for monitoring 3 aspects of patient care within an OAA adherence intervention: (1) clinical indicators, (2) process indicators, and (3) economic indicators.
Several oncology professional organizations have advocated for monitoring and promoting medication adherence for patients with cancer. However, there continues to be a paucity of literature regarding which KPIs should define metrics for success for health promotion programs that aim to improve care delivery to patients receiving OAAs. ASCO’s Quality Oncology Practice Initiative (QOPI) metrics provide cancer centers with an objective approach to measuring and reporting quality metrics. These metrics also serve as the criteria for QOPI certification for cancer centers. Although these metrics are comprehensive, they do not specifically define KPIs for an OAA-specific care delivery program. Similarly, guidelines that address OAA care delivery focus on elements of a coordinated program (eg, patient education, proactive monitoring, financial toxicity) without discussing KPIs that would define the success metrics.25–29 In the context of evolving value-based payment models, it is essential to establish a specific set of metrics and targets for OAA care delivery, which plays a critical role in monitoring the value-added by a program. Given the high cost and price of OAAs for everyone within the health care system, clinical programs established to improve outcomes should be held accountable for delivering on predefined KPIs. Prior to adoption of our recommended KPIs, there needs to be robust prospective validation in diverse settings across both academic and nonacademic cancer centers.
There were several limitations to this study. First, these KPIs have yet to be clinically implemented. Application of these metrics and targets should be considered with caution until further validation work is completed. However, these KPIs have been validated by an advisory panel of clinicians, administrators, and patients and are seen as beneficial to improving an OAA program. Second, due to the lack of feasibility of creating a nationwide expert panel, all stakeholders involved in the creation of these KPIs were located within the same statewide health system that shares the same EHRs. As a result, the generalizability of certain aspects of these KPIs on a nationwide level may be limited.
Conclusions
The 11 KPIs we identified offer a comprehensive tool for monitoring and improving an OAA adherence intervention. We plan to use these KPIs to aid in the adoption, implementation, and maintenance of an OAA adherence pilot moving forward. Further investigation is required to determine the impact of these KPIs on the success of this OAA adherence intervention.
References
- 1.↑
Druker BJ, Guilhot F, O’Brien SG, et al. Five-year follow-up of patients receiving imatinib for chronic myeloid leukemia. N Engl J Med 2006;355:2408–2417.
- 2.↑
Bedell CH. A changing paradigm for cancer treatment: the advent of new oral chemotherapy agents. Clin J Oncol Nurs 2003;7:5–9.
- 3.↑
Kong JH, Go TH, Lee JY, et al. Adherence to tyrosine kinase inhibitor affects overall survival in adult Korean chronic myeloid leukemia patients; based on the National Health Information Database. Blood 2018;132(Suppl 1):1745.
- 4.↑
Collins J 4th, Stump SE, Heiling H, et al. Impact of adherence to ibrutinib on clinical outcomes in real-world patients with chronic lymphocytic leukemia. Leuk Lymphoma 2022;63:1823–1830.
- 5.↑
Marin D, Bazeos A, Mahon FX, et al. Adherence is the critical factor for achieving molecular responses in patients with chronic myeloid leukemia who achieve complete cytogenetic responses on imatinib. J Clin Oncol 2010;28:2381–2388.
- 6.↑
Partridge A, Kato P, DeMichele A Adherence to oral cancer therapies: challenges and opportunities. Am Soc Clin Oncol Educ Book, 2009;124–128.
- 7.↑
Muluneh B, Schneider M, Faso A, et al. Improved adherence rates and clinical outcomes of an integrated, closed-loop, pharmacist-led oral chemotherapy management program. J Oncol Pract 2018;14:e324–334.
- 8.↑
Zerillo JA, Goldenberg BA, Kotecha RR, et al. Interventions to improve oral chemotherapy safety and quality: a systematic review. JAMA Oncol 2018;4:105–117.
- 9.↑
Rosenberg SM, Petrie KJ, Stanton AL, et al. Interventions to enhance adherence to oral antineoplastic agents: a scoping review. J Natl Cancer Inst 2020;112:443–465.
- 10.↑
Pritchard RD, Harrell MM, DiazGranados D, et al. The productivity measurement and enhancement system: a meta-analysis. J Appl Psychol 2008;93:540–567.
- 12.↑
Doucette D, Millin B. Should key performance indicators for clinical services be mandatory? Can J Hosp Pharm 2011;64:55–57.
- 13.↑
Gu X, Itoh K. Performance indicators: healthcare professionals’ views. Int J Health Care Qual Assur 2016;29:801–815.
- 14.↑
Muluneh B, Muir MA, Collins JB, et al. Barriers and facilitators associated with implementing interventions to support oral anticancer agent adherence in academic and community cancer center settings. PLoS One 2023;18:e0286630.
- 15.↑
Hill M, Gluyas H, Sandy M, et al. Healthcare managers’ perceptions of managing poor performance. J Health Organ Manag 2018;32:416–427.
- 17.↑
Vrijens B, De Geest S, Hughes DA, et al. A new taxonomy for describing and defining adherence to medications. Br J Clin Pharmacol 2012;73:691–705.
- 18.↑
Sattler EL, Lee JS, Perri M 3rd Medication (re)fill adherence measures derived from pharmacy claims data in older Americans: a review of the literature. Drugs Aging 2013;30:383–399.
- 19.↑
Wilson IB, Lee Y, Michaud J, et al. Validation of a new three-item self-report measure for medication adherence. AIDS Behav 2016;20:2700–2708.
- 20.↑
Geynisman DM, Wickersham KE. Adherence to targeted oral anticancer medications. Discov Med 2013;15:231–241.
- 21.↑
Marshall GN, Hays RD. The patient satisfaction questionnaire short-form (PSQ-18). Accessed December 15, 2023. Available at: https://www.rand.org/content/dam/rand/pubs/papers/2006/P7865.pdf
- 23.↑
Loerzel VW, Hines RB, Deatrick CW, et al. Unplanned emergency department visits and hospital admissions of older adults under treatment for cancer in the ambulatory/community setting. Support Care Cancer 2021;29:7525–7533.
- 24.↑
Carlotto A, Hogsett VL, Maiorini EM, et al. The economic burden of toxicities associated with cancer treatment: review of the literature and analysis of nausea and vomiting, diarrhoea, oral mucositis and fatigue. Pharmacoeconomics 2013;31:753–766.
- 25.↑
Mackler E, Segal EM, Muluneh B, et al. 2018 Hematology/Oncology Pharmacist Association Best Practices for the Management of Oral Oncolytic Therapy: pharmacy practice standard. J Oncol Pract 2019;15:e346–355.
- 26.↑
Dillmon MS, Kennedy EB, Anderson MK, et al. Patient-centered standards for medically integrated dispensing: ASCO/NCODA standards. J Clin Oncol 2020;38:633–644.
- 27.↑
Neuss MN, Polovich M, McNiff K, et al. 2013 updated American Society of Clinical Oncology/Oncology Nursing Society chemotherapy administration safety standards including standards for the safe administration and management of oral chemotherapy. Oncol Nurs Forum 2013;40:225–233.
- 28.↑
Mislang AR, Wildes TM, Kanesvaran R, et al. Adherence to oral cancer therapy in older adults: the International Society of Geriatric Oncology (SIOG) taskforce recommendations. Cancer Treat Rev 2017;57:58–66.
- 29.↑
Belcher SM, Mackler E, Muluneh B, et al. ONS guidelines to support patient adherence to oral anticancer medications. Oncol Nurs Forum 2022;49:279–295.
