Because of increasing pressure to provide evidence-based medical care and increasing healthcare costs,1 clinical practice guidelines (CPGs) have been increasingly used over the past decade worldwide.2–4 CPGs aim to interpret and consolidate evidence from the expansive clinical research literature to offer standardized approaches to high-quality care. However, it has been suggested that the quality of many published CPGs is unsatisfactory and that substantial heterogeneity exists among guideline recommendations.2,5,6 Additionally, clinicians' lack of adherence to CPGs may be a result of distrust in guideline development processes and recommendations.7 Therefore, systematically developed guidelines providing transparent recommendations are required.
The Appraisal of Guidelines for Research & Evaluation (AGREE) instrument is a validated, generic tool for the systematic appraisal of CPG methodological development and quality.8 The AGREE project was established in 1998 by a group of international guideline developers and researchers; the original AGREE instrument was published in 2003 and the updated AGREE II instrument was released in 2010.8 AGREE has become the standard tool for evaluation and development of CPGs, with the aim of improving guideline quality and likelihood of broad endorsement.9–11
Nasopharyngeal carcinoma (NPC) is a unique head and neck (H&N) cancer with an uneven distribution: the age-standardized incidence rate ranges from 20 to 50 per 100,000 men in southern China to 0.5 per 100,000 in mainly Caucasian populations.12 Despite originating from similar cell or tissue lineages, NPC is distinct to other epithelial H&N tumors. Radiotherapy (RT) is the primary and only curative modality, and combined chemoradiotherapy (chemoRT) is a pivotal treatment for locoregionally advanced NPC.13 However, the optimal protocols remain elusive. The value of adjuvant chemotherapy (AC) or neoadjuvant chemotherapy (NACT) in addition to concurrent chemoRT (CCRT) are uncertain14,15; these issues need to be addressed clearly using reliable and high-quality CPGs. However, the quality and recommendations of NPC CPGs have not been specifically evaluated. Therefore, we aimed to critically appraise the quality of CPGs for NPC using the AGREE II instrument and assessed the consistency of their recommendations.
Methods
CPG Selection
CPGs were defined as systematically developed statements aiming to assist practitioner and patient decisions regarding appropriate healthcare for specific circumstances.16 We focused on CPGs providing recommendations on NPC diagnosis and management published in English. When >1 set of documents was produced by the same organization, the most up-to-date version was considered. We excluded guidelines that (1) focused exclusively on H&N cancer without specific recommendations for NPC; (2) focused entirely on unique techniques, such as RT planning; and (3) concentrated only on recurrent or metastatic NPC.
Two reviewers (Y.P.C. and L.C.) identified relevant studies using keyword searches of MEDLINE, ESMO, NCCN, NCI, National Guideline Clearing-house (NGC), Cancer Care Ontario, the National Institute for Health and Care Excellence, the Scottish Intercollegiate Guidelines Network, and the Guidelines International Network. The following terms were used: “nasopharyngeal neoplasm,” “nasopharyngeal tumors,” or “nasopharyngeal cancers,” and “clinical practice guidelines,” “recommendations,” or “consensus.” The last search was performed in December 2015.
CPG Data Extraction
For each CPG, 2 reviewers (Y.P.C. and L.C.) independently collected guideline characteristics (eg, year of publication, authoring organization, country or region, level of development, funding source, whether it is distinct to H&N cancer), and key recommendations stated (eg, recommendations on diagnosis, management, and/or surveillance of NPC). The individually recorded details were compared, and any disagreement was resolved by consensus.
Quality Assessment
Mean Item Scores for Clinical Practice Guidelines Assessed Using the AGREE-II Instrument (Range, 1–7)
Results
Characteristics of Selected CPGs
Eligible CPGs (Figure 1) were from 7 organizations: Asia Pacific H&N Cancer expert panel (APHNCEP), Chinese H&N Cancer expert panel (CHNCEP), ESMO, NCCN, NCI, NGC, and Spanish Society of Medical Oncology (SEOM)18–28; their characteristics are summarized in Table 2. All guidelines were published between 2012 and 2015. Two were developed in multiple countries: APHNCEP was approved in the Asia and Pacific area18 and ESMO in Europe.20 The remaining CPGs were produced in China,19 the United States,24–27 or Spain.24 A total of 5 guidelines were produced by professional organizations
Flowchart of selection process for clinical practice guidelines.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 15, 3; 10.6004/jnccn.2017.0033
CPG Appraisal
The domain scores (%) for each CPG, as assessed using the AGREE II instrument, are shown in Figure 2 and Table 3. Of the 6 domains, applicability (domain 5) scored consistently low across CPGs (range, 13.5%–30.2%); no CPG scored >50% in all 6 domains. Scope and purpose (domain 1; ≥73.6% for 4 CPGs) and editorial independence scored highest (domain 6; ≥75.0% for 6 CPGs). With the exception of domain 5, a total of 4 NPC CPGs scored >50% in the remaining 5 domains: APHNCEP (range, 50.0%–86.1%), NCCN (range, 51.6%–87.5%), NCI (range, 56.3%–85.4%), and NGC (range, 62.5%–97.2%). NGC scored highest on domains 1 and 3 (97.2% and 70.3%), and NCCN and NCI scored highest on domain 2 (both, 66.7%). NCCN also ranked first on domain 6 (87.5%), ESMO scored highest on domain 4 (87.5%), and CHNCEP scored highest on domain 5 (30.2%).
Table 1 presents the mean scores and standard deviations for the 23 items for each CPG. A substantial, good, or perfect agreement was observed for all items. On averaging the item scores for each of the 7 CPGs, 9 items scored fewer than half of the points available (<3.5); the most frequently poorly reported items were items 5, 7, 8, 13, 14, and 18 to 21 (Table 3). Unsurprisingly, the mean scores for all CPGs were low for items in domain 5 (items 18–21), indicating that applicability did not attract enough attention and was performed poorly (Table 3). The mean scores for items 5 (range, 2.00–3.00) and 13 (range, 2.00–5.50) were low, indicating that the guidelines seldom provide the views or preferences of the target population, or were externally reviewed by experts before publication. The remaining items with low scores (items 7, 8, and 14) were in domain 3 (rigor of development) and were covering evidence searching, selection methods, and procedures for updating the guideline.
Key Recommendations of Guidelines
Supplemental eTable 1 (available with this article at JNCCN.org) presents the key recommendations
Characteristics of the NPC CPGs
Regarding nonmetastatic/recurrent NPC, all guidelines advised RT alone for stage I. However, the recommended treatments for stage II NPC varied: APHNCEP,18 ESMO,20 and NGC26 recommend CCRT, whereas NCCN24 and NCI25 advocate CCRT followed by AC (category 2A for NCCN). In addition, NCCN24 lists CCRT alone (category 2B) or NACT plus CCRT (category 3) as choices, and NCI25 recommends RT alone for stage II NPC. SEOM27 recommends RT alone, whereas CHNCEP19 recommends RT alone for N0 and RT with or without concurrent chemotherapy for N1 NPC. For locoregionally advanced NPC, APHNCEP,18 ESMO,20 NGC,26 and SEOM27 recommend CCRT alone. NCCN24 recommendations are the same as those for stage II NPC. NCI25 advocates altered fractionation RT as an alternative, and both APHNCEP18 and CHNCEP19 focus on additional cetuximab. The agents recommended were relatively consistent for concurrent chemotherapy and AC, but inconsistent for NACT. For metastatic NPC, platinum-based combination chemotherapy was the most common basic therapeutic scheme, whereas individualized combinations of surgery, RT, and chemotherapy were recommended for recurrent NPC.
CHNCEP,19 NCCN,24 NGC,26 and SEOM27 clearly define the interval and duration of surveillance. Only APHNCEP,18 CHNCEP,19 NCCN,24 and NCI25 consider monitoring of EBV DNA after treatment.
Discussion
This study sought to assess the quality and consistency of recommendations in CPGs involving the diagnosis and management of NPC, to assist clinicians when selecting appropriate guidelines. Using the AGREE II instrument, CPGs achieved consistently low scores in applicability (domain 5). The guidelines developed by APHNCEP, NCCN, NCI, and NGC achieved scores >50% in the remaining 5 domains. Of the 23 AGREE II items, 9 were inadequately reported. Additionally, heterogeneity in treatment recommendations exists.
Developers of CPGs often fail to meet widely endorsed standards devised to ensure high-quality guidelines.29,30 Although oncology guidelines appear
Comparison of the quality of the clinical practice guidelines using the AGREE II instrument.
Abbreviations: APHNCEP, Asia Pacific Head and Neck Cancer expert panel; CHNCEP, Chinese Head and Neck Cancer expert panel; NGC, National Guideline Clearinghouse; SEOM, Spanish Society of Medical Oncology.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 15, 3; 10.6004/jnccn.2017.0033
AGREE II enables the systematic evaluation of CPG quality and has been validated and endorsed by WHO as an accepted standard for guideline development.33,34 Based on the rationale that a high methodological quality reflects CPG integrity, reproducibility, and transparency, we also assessed guideline development methods. Methodological quality was optimal for “scope and purpose” and “editorial independence,” (domains 1 and 6, respectively) whereas “applicability” (domain 5) received the lowest scores. In terms of individual items, applicability domain items (items 18–22) were of low quality, as expected. Additionally, most guidelines lacked explicit statements on whether the views and preferences of the target population were sought (item 5), whether the guideline had been externally reviewed by experts before publication (item 13), and the procedure for updating the guideline (item 14). Search methods and selection criteria (items 7 and 8) in for rigor of development (domain 3) were also inadequately reported. Lack of applicability may indicate that, despite investment of vast resources during development, little consideration is given to translation of CPG recommendations into practice.
For a guideline to be effective, it should take resource implications into account, describe potential facilitators and barriers, and provide clear advice on implementation. More effort is needed to improve conceptualization of implementation during development. Because patients may not have the same views as CPG developers on the outcomes they care most about (eg, severe side effects vs survival benefits), it is important to consider the views and preferences of patients. Similarly, opinions from external experts are also important to ensure validity. The low scores for these 2 items may be explained by the difficulties in or lack of resources available for involving patients or external experts in guideline development. Certain items in the rigor of development domain (domain 3; items
Domain Scores for NPC CPGs Assessed Using the AGREE-II Instrument
Despite the differences in the rigor of development domain scores, the recommendations of the CPGs were generally consistent in terms of diagnosis and follow-up for NPC. All guidelines advocated biopsy of the primary site for diagnosis. However, only 3 guidelines (NCCN, ESMO, SEOM) provided recommendations on neck biopsy, and no firm recommendations were made. EBV DNA testing is of clinical significance for its valid effects on diagnosis, risk stratification, prognostication, and relapse supervision for NPC.36 Only APHNCEP, ESMO, NCCN, and NCI recommend both pretreatment and posttreatment EBV DNA testing. For surveillance of NPC after treatment, NCCN provides relatively detailed recommendations.
Major distinctions across CPGs are mainly related to therapeutic strategies of combined chemoRT. RT alone and CCRT with or without AC were the main choices for stage II NPC. Although the phase III trial by Chen et al37 found that CCRT provides a considerable survival benefit in stage II NPC, these patients underwent conventional 2-dimensional RT. As the advent of intensity-modulated RT (IMRT) has improved locoregional control and resulted in improved treatment outcomes, the benefit offered by additional chemotherapy is speculated to be smaller in patients treated with IMRT.38,39 This may explain why some CPGs recommended RT alone or CCRT without AC for stage II NPC. For locoregionally advanced NPC, CCRT is the basic therapeutic regimen; short-term evaluation of the phase III trial by Chen et al40 found that additional AC did not significantly improve survival. However, NCCN still prefers CCRT plus AC for locoregionally advanced NPC, whereas APHNCEP, ESMO, NGC, and SEOM prefer CCRT alone. ESMO indicates that the efficacy of additional AC remains uncertain and that potential benefits and side effects should be considered.20 Moreover, the efficacy of additional NACT followed by CCRT has not been confirmed, and was recommended in some cases based on individualized factors (APHNCEP), in patients with problematic RT planning (ESMO), or in highly symptomatic patients (SEOM); however, NACT should not negatively affect the optimal administration of CCRT (ESMO) in any case.18,20,28 The value of additional AC and NACT require further research. Therapeutic approaches for metastatic/recurrent NPC were inconclusive, and combinations of RT, chemotherapy, and/or surgery were advised based on the specific condition of each patient. Generally speaking, both NCCN and ESMO do well in their recommendations regarding the diagnosis and staging, and follow-up for NPC. In terms of treatment for NPC, the optimal combined chemoRT strategies remain inconclusive, and inconsistency among CPGs may result from variations in processes for selecting evidence, and methods of interpreting data and formulating recommendations.11
Use of AGREE II to evaluate methodologic quality depends on complete reporting of the guideline development process.41 We evaluated the methodologic quality of the CPG publications, which are the most accessible to end-users, rather than the entire spectrum of guideline documents (eg, consensus statements containing specific policies and procedures for guideline development). If such complementary documents were not readily identifiable from the appraised CPG or its Web site, they were not assessed. For example, the ESMO guideline is a relatively brief summary compared with other guidelines, which may explain its relatively low scores. Although the supplementary materials represent a potentially valuable resource, these documents were not cited in the publications and the end user was not informed about these statements or their contents.32,35 Supplementary materials should be referenced and made more accessible in future versions to help users better understand how the guidelines were developed and increase trust in the recommendations. Moreover, explicit reporting of a guideline does not guarantee optimal recommendations; it is possible that a well-reported guideline contains defective recommendations. Conversely, a guideline that is not reported comprehensively or developed systematically may provide sound, evidence-based advice. Still, based on our results, we found that the NCG and NCCN guidelines received relatively high scores in development quality, whereas the NCCN and ESMO guidelines provided fairly practical recommendations. Therefore, generally NCCN recommendations may be a good choice to follow in clinical practice. However, it should be noted that no guideline is completely satisfactory, and some guidelines may be more suitable for the clinical practice in certain regions.
There are several limitations in this study. First, we relied on materials reported in published CPGs; reporting quality could affect these findings, which potentially places reporting quality under scrutiny. Secondly, AGREE II was used to evaluate CPGs as a whole, and is not specific for individual recommendations within guidelines. However, a global appraisal of the guideline development process may reflect the validity of the recommendations to a certain extent. Finally, only CPGs written in English were included; guidelines in other languages may play an important role in guiding clinical practice in local regions.
Because NPC is relatively rare and has an extremely unbalanced geographic distribution, it is necessary to promote international collaboration for guideline development and create CPGs with regional features. For example, guideline development organizations could share interpretation of evidence, while accounting for the organizational or cultural diversity of organizations and/or countries to provide insight into the challenges in different areas to adjust guideline recommendations accordingly. Through such efforts, development of high-quality, regionally relevant CPGs for NPC will be achieved.
Conclusions
Increased efforts are required to make comprehensive resources available to guide healthcare providers and enhance delivery of high-quality, evidence-based care for NPC. Guideline developers should evaluate both methodological processes and the resulting end product to improve quality.
The authors have disclosed that they have no financial interests, arrangements, affiliations, or commercial interests with the manufacturers of any products discussed in this article or their competitors.
This work was supported by grants from the National Science & Technology Pillar Program during the Twelfth Five-year Plan Period (2014BAI09B10); the Health & Medical Collaborative Innovation Project of Guangzhou City, China (201400000001); the Planned Science and Technology Project of Guangdong Province (2013B020400004); and the Science and Technology Project of Guangzhou City, China (14570006).
See JNCCN.org for supplemental online content.
References
- 1.↑
Woolf SH. Practice guidelines: a new reality in medicine. I. Recent developments. Arch Intern Med 1990;150:1811–1818.
- 2.↑
Grilli R, Magrini N, Penna A et al.. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 2000;355:103–106.
- 3.
Woolf SH, Grol R, Hutchinson A et al.. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ 1999;318:527–530.
- 4.↑
Smith TJ, Hillner BE. Ensuring quality cancer care by the use of clinical practice guidelines and critical pathways. J Clin Oncol 2001;19:2886–2897.
- 5.↑
Graham ID, Beardall S, Carter AO et al.. What is the quality of drug therapy clinical practice guidelines in Canada? CMAJ 2001;165:157–163.
- 6.↑
Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 1999;281:1900–1905.
- 7.↑
Sox HC. Assessing the trustworthiness of the guideline for management of high blood pressure in adults. JAMA 2014;311:472–474.
- 8.↑
Brouwers M, Kho ME, Browman GP et al.. AGREE II: advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J 2010;182:E839–842.
- 9.↑
Sinuff T, Patel RV, Adhikari NK et al.. Quality of professional society guidelines and consensus conference statements in critical care. Crit Care Med 2008;36:1049–1058.
- 10.
Qaseem A, Snow V, Shekelle P et al.. Screening for HIV in health care settings: a guidance statement from the American College of Physicians and HIV Medicine Association. Ann Intern Med 2009;150:125–131.
- 11.↑
Huang TW, Lai JH, Wu MY et al.. Systematic review of clinical practice guidelines in the diagnosis and management of thyroid nodules and cancer. BMC Med 2013;11:191.
- 14.↑
Chen YP, Wang ZX, Chen L et al.. A Bayesian network meta-analysis comparing concurrent chemoradiotherapy followed by adjuvant chemotherapy, concurrent chemoradiotherapy alone and radiotherapy alone in patients with locoregionally advanced nasopharyngeal carcinoma. Ann Oncol 2015;26:205–211.
- 15.↑
Chen YP, Guo R, Liu N et al.. Efficacy of the additional neoadjuvant chemotherapy to concurrent chemoradiotherapy for patients with locoregionally advanced nasopharyngeal carcinoma: a Bayesian network meta-analysis of randomized controlled trials. J Cancer 2015;6:883–892.
- 16.↑
Field MJ, Lohr KN, eds. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academies Press; 1990.
- 18.↑
D'Cruz A, Lin T, Anand AK et al.. Consensus recommendations for management of head and neck cancer in Asian countries: a review of international guidelines. Oral Oncol 2013;49:872–877.
- 19.↑
Lang J, Gao L, Guo Y et al.. Comprehensive treatment of squamous cell cancer of head and neck: Chinese expert consensus 2013. Future Oncol 2014;10:1635–1648.
- 20.↑
Chan AT, Gregoire V, Lefebvre JL et al.. Nasopharyngeal cancer: EHNS-ESMO-ESTRO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol 2012;23(Suppl 7):vii83–85.
- 21.
Pfister DG, Ang KK, Brizel DM et al.. NCCN Clinical Practice Guidelines in Oncology: Head and Neck Cancer, Version 2.2011. To view the most recent version of these guidelines, visit NCCN.org.
- 22.
Pfister DG, Ang KK, Brizel DM et al.. NCCN Clinical Practice Guidelines in Oncology: Head and Neck Cancer, Version 2.2013. To view the most recent version of these guidelines, visit NCCN.org.
- 23.
Pfister DG, Spencer S, Brizel DM et al.. NCCN Clinical Practice Guidelines in Oncology: Head and Neck Cancer, Version 2.2014. To view the most recent version of these guidelines, visit NCCN.org.
- 25.↑
SEER-Medicare: Brief Description of the SEER-Medicare Database. National Cancer Institute Web site. Available at: http://healthservices.cancer.gov/seermedicare/overview/. Accessed February 13, 2017.
- 26.↑
Nasopharyngeal Cancer Treatment (PDQ®)-Health Profession Version. National Cancer Institute Web site. Available at: http://www.cancer.gov/types/head-and-neck/hp/nasopharyngeal-treatment-pdq. Accessed December 1, 2015.
- 27.↑
National Guideline Clearinghouse. Nasopharyngeal cancer treatment. Available at: https://www.guideline.gov/summaries/summary/48140. Accessed February 13, 2017.
- 28.↑
Mesia R, Pastor M, Grau JJ, del Barco ESEOM. SEOM clinical guidelines for the treatment of nasopharyngeal carcinoma 2013. Clin Transl Oncol 2013;15:1025–1029.
- 29.↑
Reames BN, Krell RW, Ponto SN, Wong SL. Critical evaluation of oncology clinical practice guidelines. J Clin Oncol 2013;31:2563–2568.
- 30.↑
Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet institute of medicine standards: two more decades of little, if any, progress. Arch Intern Med 2012;172:1628–1633.
- 31.↑
Burgers JS, Fervers B, Haugh M et al.. International assessment of the quality of clinical practice guidelines in oncology using the Appraisal of Guidelines and Research and Evaluation Instrument. J Clin Oncol 2004;22:2000–2007.
- 32.↑
Abdelsattar ZM, Reames BN, Regenbogen SE et al.. Critical evaluation of the scientific content in clinical practice guidelines. Cancer 2015;121:783–789.
- 33.↑
AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003;12:18–23.
- 34.↑
Grol R, Cluzeau FA, Burgers JS. Clinical practice guidelines: towards better quality guidelines and increased international collaboration. Br J Cancer 2003;89(Suppl 1):S4–8.
- 35.↑
Cervantes A, Pentheroudakis G, Arnold D. Critical evaluation of the scientific content in clinical practice guidelines. Cancer 2015;121:1907–1908.
- 36.↑
Leung SF, Chan KC, Ma BB et al.. Plasma Epstein-Barr viral DNA load at midpoint of radiotherapy course predicts outcome in advanced-stage nasopharyngeal carcinoma. Ann Oncol 2014;25:1204–1208.
- 37.↑
Chen QY, Wen YF, Guo L et al.. Concurrent chemoradiotherapy vs radiotherapy alone in stage II nasopharyngeal carcinoma: phase III randomized trial. J Natl Cancer Inst 2011;103:1761–1770.
- 38.↑
Lee AW, Lau WH, Tung SY et al.. Preliminary results of a randomized study on therapeutic gain by concurrent chemotherapy for regionally-advanced nasopharyngeal carcinoma: NPC-9901 Trial by the Hong Kong Nasopharyngeal Cancer Study Group. J Clin Oncol 2005;23:6966–6975.
- 39.↑
Lai SZ, Li WF, Chen L et al.. How does intensity-modulated radiotherapy versus conventional two-dimensional radiotherapy influence the treatment results in nasopharyngeal carcinoma patients? Int J Radiat Oncol Biol Phys 2011;80:661–668.
- 40.↑
Chen L, Hu CS, Chen XZ et al.. Concurrent chemoradiotherapy plus adjuvant chemotherapy versus concurrent chemoradiotherapy alone in patients with locoregionally advanced nasopharyngeal carcinoma: a phase 3 multicentre randomised controlled trial. Lancet Oncol 2012;13:163–171.
- 41.↑
Hayward RS, Wilson MC, Tunis SR et al.. Users' guides to the medical literature. VIII. How to use clinical practice guidelines. A. Are the recommendations valid? The Evidence-Based Medicine Working Group. JAMA 1995;274:570–574.
