Background
For patients with cancer, experiencing multiple concurrent symptoms is common.1–4 This concept has been described in a variety of ways, such as symptom clusters, symptom burden, or complex symptoms. Although these terms have similar meanings, the term “symptom complexity” is used in this article to capture the continuum of symptom experiences from mild/no symptoms to numerous concurrent severe symptoms. Patients who experience higher levels of symptom complexity tend to experience lower functioning and quality of life.2,5,6 Therefore, it is critical to identify patients experiencing severe symptom complexity so that timely, targeted symptom management can be delivered.
With research showing benefits of gathering outcome data directly from patients, it has become more common for standardized, validated questionnaires to be integrated into routine clinical practice.7 In 2012, Alberta Heath Services (AHS) Cancer Care Alberta (CCA) began implementing the routine use of a standardized patient symptom questionnaire called “Putting Patients First” (PPF) at all 17 cancer facilities in order to raise clinician awareness of patients’ concerns and symptoms. The PPF includes 2 standardized measures: the Revised Edmonton Symptom Assessment System (ESAS-r)8,9 and the Canadian Problem Checklist (CPC),10 selected in alignment with the national reporting criteria established by the Canadian Partnership Against Cancer (CPAC).11
In the process of integrating these patient-reported outcomes (PROs) into routine clinical practice, we encountered several barriers, such as a perceived lack of time to address symptoms/concerns and issues with the interpretation of PROs. Although clinicians appreciated the ability to see PROs for each patient, they reported feeling overwhelmed by the number of PRO data points to assimilate into their clinical assessment. This feedback aligned with published research on barriers to using PROs in routine clinical practice that include interpretation and application issues.7 As a result, in 2017 a group of epidemiologists, statisticians, and clinicians gathered to develop a pragmatic PRO-driven symptom complexity algorithm based on the existing data already being collected in the Alberta practice environment. The intention was to design visual flags to overcome the clinical challenge of identifying which patients are experiencing numerous concurrent severe symptoms. Three visual flags (green, yellow, red) were developed to signify different categories of symptom complexity that could be reported by a patient at an encounter. Based on symptom complexity level, clinicians can easily access the full view of each individual’s trended PRO symptoms to tailor their clinical response further as needed. The development process included a variety of actual patient symptom scenarios and case reviews with clinical teams to assess the fit of the complexity score with clinical reality, leading to further refinement of the algorithm.
As interest in using the symptom complexity level spread more broadly in the organization, a formal validation of the algorithm was required. This study was designed to validate the algorithm by (1) assessing the known-group validity, (2) calculating the accuracy indices for correctly identifying patients with cancer with different levels of symptom complexity, and (3) comparing the algorithm’s overall screening capacity to an alternative algorithm, generated by 2-step cluster analysis.
Methods
Data Source
This is a retrospective study of symptom complexity in patients with cancer undergoing treatment in Alberta, Canada. This study uses the data from the Alberta Cancer Registry and the electronic medical records (EMRs) within CCA known as ARIA. This study did not require written consent from participants because this was part of a larger provincial quality improvement project to improve clinical capacity to identify, assess, and manage symptoms. Data were gathered during chart audits conducted to understand the magnitude of clinic redesign required to increase capacity for targeted symptom management.
This project complies with the Helsinki Declaration and the Alberta Research Ethics Community Consensus Initiative guidelines for quality improvement and evaluation. The ethical principles established by the Alberta Research Ethics Community Consensus Initiative screened this project and waived the need for full Research Ethics Board review. No harm was anticipated or reported in relation to this project.
Study Population
The research sample included a cohort of patients with advanced-stage cancer attending a palliative radiation therapy clinic at a cancer facility in Alberta, Canada, from February 2016 through November 2017. Criteria for inclusion were (1) patients with cancer aged >18 years, (2) Alberta residents, and (3) PRO data from the ESAS-r, CPC, and EuroQol 5-level EQ-5D (EQ-5D-5L) in their EMR. A total of 1,466 patients with cancer were identified and 64 (4%) were excluded due to incomplete data, leaving a final study cohort of 1,402 patients.
Algorithms
The symptom complexity algorithm considers the unique combination of symptoms and concerns the patient has identified on their preclinic PPF questionnaire. It rates the self-reported severity of symptoms and the number of concerns indicated at a single visit and assigns a symptom complexity score (low/green, moderate/yellow, or severe/red) for the encounter (Figure 1). This summary score alerts the clinician to the level of symptom complexity the patient is experiencing so the clinical team can tailor their time allocation and care to best meet the patient’s needs as displayed on their individual trended PRO graph. For this study, the moderate/yellow and low/green categories of symptom complexity were collapsed into a single category to facilitate comparison, even though it represents a potential loss of information. This combined group will be referred to as the “low-complexity group” in this article.
Symptom complexity algorithm.
Abbreviations: CPC, Canadian Problem Checklist; ESAS-r, Revised Edmonton Symptom Assessment System.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 18, 11; 10.6004/jnccn.2020.7586
Symptom complexity algorithm.
Abbreviations: CPC, Canadian Problem Checklist; ESAS-r, Revised Edmonton Symptom Assessment System.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 18, 11; 10.6004/jnccn.2020.7586
Symptom complexity algorithm.
Abbreviations: CPC, Canadian Problem Checklist; ESAS-r, Revised Edmonton Symptom Assessment System.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 18, 11; 10.6004/jnccn.2020.7586
An alternative algorithm using a 2-step cluster analysis was used for the purposes of comparison. This alternate algorithm divides patients with cancer into groups with distinct levels of symptom severity (high vs low) by applying a 2-step cluster analysis to classify symptom level, as proposed by Shi et al12 in 2011. This type of exploratory analysis has been used to categorize homogenous groups when the grouping is not previously known.13
Measurements
The ESAS-r is a 10-item PRO measure of prevalent evidence-based symptoms experienced by patients with cancer. Each item is rated on a 0 to 10 scale of severity. The scale of “well-being” is not reported on because the scale caused confusion, in that some patients counted 0 as the worst possible score, whereas others counted 10 as the worst possible score. The “other” category was also not included because it could be used to report on a large variety of symptoms.
The CPC is an evidence-based, 54-item self-report checklist designed to identify common concerns that patients with cancer experience. It is used in conjunction with the ESAS-r and tracks concerns since the patient’s last visit.
The EQ-5D-5L is a standardized screening tool that measures health status and quality of life and is extensively validated in different studies.14,15 The EQ-5D-5L has 5 dimensions: mobility, self-care, usual activities, pain/discomfort, and anxiety/depression. Each dimension has 5 levels: no problems, slight problems, moderate problems, severe problems, and extreme problems. Patients make a selection to identify the best description of their current health status (Table 1). The EQ-5D-5L misery index score is an overall health status score, which combines the scores within all 5 dimensions. A score of 25 represents the poorest health status possible.16
EuroQol 5-Level EQ-5D
Karnofsky performance status (KPS) is a validated tool widely used by oncology clinicians to assess the functional status of their patients.17,18 It allows for classification and stratification into 3 functional classifications based on KPS score ranges (100–80 = normal functional status; 70–50 = compromised functional status; and <50 = poor functional status).19,20 Previous studies have broadly used KPS as an outcome/standard in the validation of the ESAS.21–24 Furthermore, researchers have shown KPS, when combined with ESAS, to be useful for recoding symptom prevalence and intensity in patients with advanced cancer.25 Consequently, KPS was used as a comparator for symptom complexity scores in this study. Table 2 lists the KPS definition rating (%) criteria.
Karnofsky Performance Status Rating (%) Criteria
Statistical Analysis
Different types of validity were assessed. We first evaluated the known-group validity by using the independent t test and the Cohen d effect size to compare the EQ-5D-5L misery index score between the severe- and low-complexity groups. The magnitude of effect size was based on the value of d, with 0.2 considered a small effect size, 0.5 considered a medium effect size, and 0.8 considered a large effect size.26 Moreover, using KPS as a comparator, we computed the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy for the symptom complexity algorithm by 2 × 2 frequency tables for a subset of the sample who also had a KPS completed at the same visit (n=583). To determine whether the complexity algorithm performed consistently across demographic subgroups, we further stratified the population by sex and age groups and repeated the same analysis. Last, we compared the overall screening capacity of the 2 algorithms by using the area under the curve (AUC) of receiver operating characteristic (ROC). Data were exported into SPSS Statistics, version 25 (IBM Corp) for analysis and statistical significance was set a priori at P<.05.
Results
Sample Characteristics
Mean age at diagnosis was 63 years (SD, 13.3; range, 12–97; median, 63 years), and 50.3% were female. The most common tumor grouping was genitourinary (25.5%), followed by breast (22.8%), lung (19.5%), gastrointestinal (12.8%), hematologic (6.3%), gynecologic (5.8%), head and neck (2.1%), sarcoma (1.5%), unknown primary (1.4%), endocrine (1.0%), cutaneous (0.9%), and central nervous system (0.6%).
Summary of Symptoms
The mean score on the ESAS-r as a whole was 2.78, demonstrating that patients generally score at the low end of the response range. The most outstanding symptom among this sample was tiredness (mean [M], 4.60; SD, 2.87; range, 0–10; interquartile range [IQR], 2.00–7.00), followed by pain (M, 3.61; SD, 3.02; range, 0–10; IQR, 1.00–6.00) and drowsiness (M, 3.14; SD, 2.93; range, 0–10; IQR, 0.00–5.00). The mildest symptom reported was nausea (M, 1.10; SD, 2.07; range, 0–10; IQR, 0.00–1.00). Figure 2 presents the distribution of symptom intensity based on the ESAS-r for this cohort by established cutoffs (0, 1–3, 4–6, and 7–10).
Distribution of symptom intensity based on the Revised Edmonton Symptom Assessment System.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 18, 11; 10.6004/jnccn.2020.7586
Distribution of symptom intensity based on the Revised Edmonton Symptom Assessment System.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 18, 11; 10.6004/jnccn.2020.7586
Distribution of symptom intensity based on the Revised Edmonton Symptom Assessment System.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 18, 11; 10.6004/jnccn.2020.7586
The CPC has 7 domains: emotional, social/family/spiritual, practical, physical, mobility, nutrition, and informational, and each domain has up to 15 more specific concerns. Of the total number of 54 specific concerns, the most frequently endorsed concern was walking/mobility (n=559), followed by fears/worries (n=443) and strength (n=431). The least endorsed concern was “How much alcohol you drink” (n=11). Table 3 lists the top 10 most endorsed concerns of this sample.
Top 10 Endorsed Concerns Based on the Canadian Problem Checklist
Prevalence of Symptom Complexity
For all 1,402 patients with cancer included in this study, a symptom complexity score was determined according to the proposed algorithm (Figure 1): severe (n=543; 38.7%; 99% CI, 35.4%–42.2%) and low (n=859; 61.3%; 99% CI, 57.8%–64.6%). Table 4 outlines the overall prevalence of symptom complexity levels by patient characteristics.
Sample Characteristics by Patient Symptom Level
Known-Group Validity
Known-group (discriminant) validity was assessed by determining whether the symptom complexity algorithm could differentiate among groups known to differ in health status. We compared health status between the 2 groups using the EQ-5D-5L misery index score as an indicator. The severe-complexity group reported significantly higher mean misery index scores than the low-complexity group (14.8 [95% CI, 14.4–15.1] vs 9.91 [95% CI, 9.64–10.2]; P<.001). A Cohen d of 1.2 indicated that the magnitude of difference in health status between the groups is large.
Accuracy Indices
Overall performance (sensitivity, specificity, PPV, NPV, and accuracy) was calculated using a subgroup of patients who also had a clinician-generated KPS score (n=583). This study chose a score <50% as the cutoff, which indicates the patient has low functionality and substantial care needs,27–29 and classified the patients into 2 groups—more complex (KPS score <50%) versus less complex (KPS score ≥50%)—to correspond with the dichotomous classification by our complexity algorithm: severe symptoms versus low symptoms. Table 5 shows the validity indices obtained for the symptom complexity algorithm with 95% confidence intervals. We further stratified the cohort by sex and age and conducted the same calculations.
Validity Indices
Comparison Between Algorithms
To make this comparison, we generated an alternative 2-step cluster analysis. With the 8 symptoms on ESAS-r as variables (“well-being” and “other” were excluded), the clustering solution was obtained using log-likelihood estimation, and the number of groups fixed (n=2). The 2-step cluster analysis generated a low-symptom group (n=926; 67.3%) and a severe-symptom group (n=449; 32.7%).
The ROC curve plots the probability of true-negative (specificity) versus the probability of false-positive (sensitivity) for various decision criteria, and therefore the larger the AUC, the more valid the diagnostic test is in comparison with the KPS comparator. The measure of the AUC will be between 0.5 for no detectability of the test (chance alone) and 1.0 for perfect detectability. An AUC >0.8 is generally considered adequate.30 As presented in Figure 3A, the results revealed an AUC of 0.824 (95% CI, 0.790–0.858) for the symptom complexity algorithm, which confirmed its screening capacity and surpassed the AUC of 0.721 (95% CI, 0.679 –0.762) for the 2-step cluster analysis algorithm (Figure 3B).
ROC curves: (A) complexity algorithm (AUC, 0.824), and (B) 2-step cluster algorithm (AUC, 0.721).
Abbreviations: AUC, area under the curve; ROC, receiver operating characteristic.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 18, 11; 10.6004/jnccn.2020.7586
ROC curves: (A) complexity algorithm (AUC, 0.824), and (B) 2-step cluster algorithm (AUC, 0.721).
Abbreviations: AUC, area under the curve; ROC, receiver operating characteristic.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 18, 11; 10.6004/jnccn.2020.7586
ROC curves: (A) complexity algorithm (AUC, 0.824), and (B) 2-step cluster algorithm (AUC, 0.721).
Abbreviations: AUC, area under the curve; ROC, receiver operating characteristic.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 18, 11; 10.6004/jnccn.2020.7586
Discussion
Experiencing multiple concurrent symptoms has a profound impact on a patient’s well-being, functional status, and quality of life. Unmanaged complex symptoms can also negatively impact adherence to recommended cancer treatments and clinical outcomes.31 Therefore, enhancing a clinician’s ability to identify patients experiencing numerous concurrent severe symptoms increases the likelihood that targeted symptom management and supportive care will be provided within the clinical encounter.2–4 This algorithm is not meant to stand alone, as the complexity level is a clinical flag for the healthcare team to review and address patients’ individual reported symptoms and/or concerns. Nevertheless, this algorithm provides a simple, practical way to interpret and summarize a variety of PROs that aligns with recommendations that the interpretation of PROs should be simple and descriptive (eg, none, mild, moderate, severe).7
This validation study aimed to evaluate whether the PRO-derived symptom complexity algorithm could accurately and consistently identify patients with cancer experiencing distinct levels of complex symptoms. The reliability of this complexity score has immediate implications for clinical practice. At the micro level, in the clinical care process, patients with varying degrees of symptom complexity are easily identified in the preclinic phase so that the care team can make real-time decisions regarding which level of provider and which route of care (in-person vs virtual) is most suited for the level of symptom management the patient requires. At the meso level, administrators can adjust clinic templates and staffing levels in response to the distribution of symptom complexity among different clinical populations. At the macro level, program funding for symptom management and supportive care could be allocated to areas in which patient needs are prevalent. Based on these system-level impacts, the need for a robust validation study became apparent.
Known-group validity is a standard method of construct validation and is an important characteristic of a good PRO measure.32 Known-group analysis indicated the discriminant capacity of the PRO-derived symptom complexity algorithm for this population with cancer. Mean scores for the EQ-5D-5L misery index score were significantly higher for the severe-complexity subgroup, with a large effect size supporting that this algorithm effectively differentiated the severe-complexity subgroup from the low-complexity subgroup in regard to health status.
Using KPS as a clinical-derived comparator, the PRO-derived symptom complexity algorithm shows satisfactory sensitivity, specificity, predictive value, and accuracy for discriminating patients with severe symptom complexity from those with low symptom complexity. These results confirm the validity of the PRO-derived symptom complexity algorithm and is the basis for further validation work.
To make a comparison, we generated an alternative algorithm classification through 2-step cluster analysis.12,13 The AUC for our PRO-derived symptom complexity algorithm is greater than the alternative algorithm. This suggests the PRO-derived symptom complexity algorithm is superior to the alternative algorithm at detecting symptom complexity. This finding has an important implication: even though a statistical perspective is emphasized in many subject-matter areas, symptom complexity requires proper contextualization of the patient.33
This study is not without limitations. The pragmatic design of our symptom complexity algorithm required retrospective validation, and therefore we had to select from the available measures in our clinical EMR. We found KPS was the best possible assessment tool among available measures, because it has been widely used in ESAS validation studies.21–24 Further studies should be conducted with different reference standards and weighting of symptoms, and to measure the impact that changes to the algorithm would have on the algorithm’s validity. Additionally, the study sample is composed of patients with advanced-stage cancer, which may impede generalizability.
Despite the limitations, this study has a few notable strengths, including sample size and lack of selection bias. The large number of participants in this study includes patients with cancer from both sexes, different ages, and different tumor groups, which increases the samples representativeness. Furthermore, these data were collected as part of routine clinical practice, rather than by research volunteers, which increases external validity.
We believe our study makes 2-fold contributions to current PRO-based research. It has validated an algorithm for ascertaining cases of high symptom complexity among patients with cancer based on their self-reported symptom burden, offering an alternative approach to identify individuals who require complex symptom management. Furthermore, even though PROs have been widely included in cancer research over the past decade, providers still express concerns about the reliability of patient outcomes and continue to show professional resistance to the engagement and empowerment of patients.34–36 This study confirms the concordance between PROs and clinical assessment and highlights the value of patients’ input in describing their own symptoms.
Conclusions
This study demonstrated the validity of a PRO-derived algorithm designed to classify symptom complexity among patients with cancer with good discrimination and calibration. This algorithm is generalizable to similar cancer populations. As healthcare reform continues, use of a symptom complexity algorithm, in conjunction with patients’ individual trended PROs, could help differentiate patient care needs based on their levels of symptom complexity. We propose this PRO-derived algorithm be adopted for future complexity-related studies, refinement, and enhancement.
References
- 1.↑
Shippee ND, Shah ND, May CR, et al.. Cumulative complexity: a functional, patient-centered model of patient complexity can improve research and practice. J Clin Epidemiol 2012;65:1041–1051.
- 2.↑
Pask S, Pinto C, Bristowe K, et al.. A framework for complexity in palliative care: a qualitative study with patients, family carers and professionals. Palliat Med 2018;32:1078–1090.
- 4.↑
Karabulut N, Erci B, Ozer N, et al.. Symptom clusters and experiences of patients with cancer. J Adv Nurs 2010;66:1011–1021.
- 5.↑
Barsevick AM, Whitmer K, Nail LM, et al.. Symptom cluster research: conceptual, design, measurement, and analysis issues. J Pain Symptom Manage 2006;31:85–95.
- 7.↑
Snyder C, Brundage M, Rivera YM, et al.. PRO-cision Medicine Methods Toolkit to address the challenges of personalizing cancer care using patient-reported outcomes: introduction to the supplement. Med Care 2019;57(Suppl 1):S1–7.
- 8.↑
Watanabe S, Nekolaichuk C, Beaumont C, et al.. The Edmonton Symptom Assessment System—what do patients think? Support Care Cancer 2009;17:675–683.
- 9.↑
Watanabe SM, Nekolaichuk C, Beaumont C, et al.. A multicenter study comparing two numerical versions of the Edmonton Symptom Assessment System in palliative care patients. J Pain Symptom Manage 2011;41:456–468.
- 10.↑
Bultz BD, Groff SL, Fitch M, et al.. Implementing screening for distress, the 6th vital sign: a Canadian strategy for changing practice. Psychooncology 2011;20:463–469.
- 11.↑
Canadian Partnership Against Cancer. The 2012 cancer system performance report. Accessed August 18, 2019. Available at: https://www.partnershipagainstcancer.ca/topics/2012-cancer-system-performance-report/
- 12.↑
Shi Q, Smith TG, Michonski JD, et al.. Symptom burden in cancer survivors 1 year after diagnosis: a report from the American Cancer Society’s studies of cancer survivors. Cancer 2011;117:2779–2790.
- 13.↑
Statistics Solutions. Conduct and interpret a cluster analysis. Accessed June 18, 2019. Available at: https://www.statisticssolutions.com/cluster-analysis-2/
- 14.↑
Rabin R, de Charro F. EQ-5D: a measure of health status from the EuroQol Group. Ann Med 2001;33:337–343.
- 15.↑
Grandy S, Fox KM. EQ-5D visual analog scale and utility index values in individuals with diabetes and at risk for diabetes: findings from the Study to Help Improve Early evaluation and management of risk factors Leading to Diabetes (SHIELD). Health Qual Life Outcomes 2008;6:18.
- 16.↑
Augustovski F, Rey-Ares L, Irazola V, et al.. An EQ-5D-5L value set based on Uruguayan population preferences. Qual Life Res 2016;25:323–333.
- 17.↑
Schag CC, Heinrich RL, Ganz PA. Karnofsky performance status revisited: reliability, validity, and guidelines. J Clin Oncol 1984;2:187–193.
- 18.↑
Yates JW, Chalmer B, McKegney FP. Evaluation of patients with advanced cancer using the Karnofsky performance status. Cancer 1980;45:2220–2224.
- 19.↑
Crooks V, Waller S, Smith T, et al.. The use of the Karnofsky performance scale in determining outcomes and risk in geriatric outpatients. J Gerontol 1991;46:M139–144.
- 20.↑
Hollen PJ, Gralla RJ, Kris MG, et al.. Measurement of quality of life in patients with lung cancer in multicenter trials of new therapies. Psychometric assessment of the Lung Cancer Symptom Scale. Cancer 1994;73:2087–2098.
- 21.↑
Chang VT, Hwang SS, Feuerman M. Validation of the Edmonton Symptom Assessment Scale. Cancer 2000;88:2164–2171.
- 22.↑
Moro C, Brunelli C, Miccinesi G, et al.. Edmonton symptom assessment scale: Italian validation in two palliative care settings. Support Care Cancer 2006;14:30–37.
- 23.↑
Carvajal A, Centeno C, Watson R, et al.. A comprehensive study of psychometric properties of the Edmonton Symptom Assessment System (ESAS) in Spanish advanced cancer patients. Eur J Cancer 2011;47:1863–1872.
- 24.↑
Paiva CE, Manfredini LL, Paiva BS, et al.. The Brazilian version of the Edmonton Symptom Assessment System (ESAS) is a feasible, valid and reliable instrument for the measurement of symptoms in advanced cancer patients. PLoS One 2015;10:e0132073.
- 25.↑
Portz JD, Kutner JS, Blatchford PJ, et al.. High symptom burden and low functional status in the setting of multimorbidity. J Am Geriatr Soc 2017;65:2285–2289.
- 26.↑
Cohen J. The statistical power of abnormal-social psychological research: a review. J Abnorm Soc Psychol 1962;65:145–153.
- 27.↑
Péus D, Newcomb N, Hofer S. Appraisal of the Karnofsky performance status and proposal of a simple algorithmic system for its evaluation. BMC Med Inform Decis Mak 2013;13:72–79.
- 28.↑
Tandon P, Reddy KR, O’Leary JG, et al.. A Karnofsky performance status-based score predicts death after hospital discharge in patients with cirrhosis. Hepatology 2017;65:217–224.
- 29.↑
Thuluvath PJ, Thuluvath AJ, Savva Y. Karnofsky performance status before and after liver transplantation predicts graft and patient survival. J Hepatol 2018;69:818–825.
- 30.↑
Sandanger I, Moum T, Ingebrigtsen G, et al.. Concordance between symptom screening and diagnostic procedure: the Hopkins Symptom Checklist-25 and the Composite International Diagnostic Interview I. Soc Psychiatry Psychiatr Epidemiol 1998;33:345–354.
- 31.↑
Basch E, Deal AM, Kris MG, et al.. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 2016;34:557–565.
- 32.↑
Broderick JE, Schneider S, Junghaenel DU, et al.. Validity and reliability of patient-reported outcomes measurement information system instruments in osteoarthritis. Arthritis Care Res (Hoboken) 2013;65:1625–1633.
- 33.↑
Goldstein H, Spiegelhalter DJ. League tables and their limitations: statistical issues in comparisons of institutional performance. J R Stat Soc Ser A 1996;159:385–443.
- 34.↑
Selby P, Velikova G. Taking patient reported outcomes centre stage in cancer research – why has it taken so long? Res Involv Engagem 2018;4:25–29.
- 35.↑
Hutchings HA, Alrubaiy L. Patient-reported outcome measures in routine clinical care: the PROMise is a better future? Dig Dis Sci 2017;62:1841–1843.
- 36.↑
Gilbert A, Sebag-Montefiore D, Davidson S, et al.. Use of patient-reported outcomes to measure symptoms and health related quality of life in the clinic. Gynecol Oncol 2015;136:429–439.
