Over the past 12 years, I have had the unusual privilege of hearing firsthand how NCCN Guidelines panel members make decisions. I have noticed a surprising degree of nuance in how data are evaluated and applied to decisions regarding whether or not to recommend a particular intervention.
Because oncology addresses such a diverse group of diseases, in which the extent and quality of data vary across and even within disease sites, the process of coming to consensus can be involved and sometimes even contentious. The sheer volume of data that must be considered for a guideline that covers the continuum of care from diagnosis through palliation or addressing the issues relating to screening and early detection for a given disease is daunting. A single guideline may have hundreds of separate decision points that can include a thousand or more possible interventions. Furthermore, available data may support a variety of different approaches. As clinician researchers, panel members are acutely aware that patient-specific disease and comorbidity factors, as well as patient preferences, can influence therapy selection for the individual. Therefore, they try to be as inclusive as possible in their recommendations—within existing data—to allow physicians the latitude to customize treatment in ways that are supported by the evidence. How the panels decide which of the universe of data are relevant and persuasive may be of interest to readers of JNCCN.
For example, in the process of assigning a category of evidence and consensus to a recommendation, panels consider the extent of the data: do several studies address the question, or only a single large or small study, or even clinical experience if no data have been published? The panels also look at the consistency of the data: are results consistent across trials or do results conflict? And finally, what is the quality of the data? Does the discussion rely on one or more randomized clinical trials, a meta-analysis or systematic review, nonrandomized clinical trials, or clinical experience? Is the panel trying to extrapolate from a setting in which high-quality data exist for a similar clinical situation? Panel members also consider whether data have been published in peer-reviewed literature or presented with abstract at a major meeting.
When evaluating clinical trials data, the panels look critically at the trial design to determine whether the results are persuasive. They consider the patient cohort, staging (and staging methods), markers, demographics, comorbid conditions allowed in the study population, performance status of patients, and prior therapies. They also review the treatment arms. Were current comparators used or were the comparators obsolete? How were dosing ranges incorporated, dose adjustments accounted for, and adverse events managed and reported? Were appropriate supportive care measures implemented? What response assessment criteria were used and how were they applied? How long was follow-up, and was it long enough to identify meaningful differences among experimental arms? Was the analytic plan appropriate, and planned analyses targeted to questions of interest? Most importantly, the panels consider all of this information collectively and decide whether the results of the study were persuasive. Interestingly, sometimes panel members look at a high-quality study but disagree among themselves regarding how data are interpreted. This can lead to spirited discussions!
Analysis of systematic reviews and meta-analyses are even more challenging, because not only is it important that the component studies are persuasive, but the knowledge and experience of the authors are also important. In reviewing a systematic review for one disease site, panel members were concerned that several of the component studies were not of sufficient quality to provide relevant results to the analysis and made the entire review less convincing.
Further complicating this task, the data in oncology are not always particularly good. There are many situations in which available data are not optimal, but treating physicians must nevertheless decide on the best intervention for their individual patients. The NCCN Guidelines panels use their expertise to recommend appropriate management based on the available evidence. The process of integrating evidence with consensus helps fill important gaps when evidence is suboptimal or lacking to apply to real-life clinical scenarios that require a solution. However, the gaps are also important to recognize, because they represent areas where future research could be directed.
Poonacha and Go1 reviewed the data within several NCCN Guidelines in an article in the Journal of Clinical Oncology last year and found that only 6% of the recommendations were category 1 (high-quality data with uniform consensus), with most of those recommendations relating to initial therapy. The vast majority of recommendations were category 2A (lower level evidence with uniform consensus). The message in this paper is not that panels ignore high-quality data, but rather that we in the oncology community must find better ways to get more patients into more high-quality trials.
The additional message is that those in positions to affect public policy must support these efforts. This support is needed both in messaging to the public and in dollars directed to basic, translational, and clinical research. In rare diseases or subsets of more common cancers, generating robust data will remain difficult until the diseases are understood better or a specific marker pointing to a specific intervention is identified. Fortunately, new data are being published each month, and the NCCN Guidelines are updated continuously to reflect these new data. Such attention to almost real-time updates is a rarity in medicine but is needed to address the evolving treatments in life-threatening diseases like cancer.
Hartzband and Groopman2 noted that although the data in guidelines may be objective in themselves, when guidelines authors consider recommendations, the interpretation and application of those data are subjective. Certainly, this has been the experience of the NCCN Guidelines panels, wherein expert consensus is used to evaluate existing data and to supplement the evidence base if high-quality data are lacking. Further, NCCN expects that clinicians using the NCCN Guidelines will, in turn, apply their own clinical judgement to individual patient circumstances.
Over the past 3 years, NCCN has taken steps to make this decision-making process more transparent to the user community. Readers of JNCCN and users of the NCCN Guidelines in JNCCN and online have expressed a strong interest in how panel decisions are made. After considerable reflection, NCCN has decided to share more of the deliberations with our user community via the new NCCN Guidelines Insights series, the first of which appears in this issue. The goal of the NCCN Guidelines Insights is to help readers understand how and why decisions were made rather than just relaying the data the decisions are based on.
The NCCN Guidelines Insights highlight the important updates and changes in the newest version of an NCCN Guideline. The NCCN Guidelines Insights include only a few algorithm pages—those that contain important updates—with these updates themselves shown in colored ink. The NCCN Guidelines Insights also include specially written discussions summarizing, in broad terms, the panel’s discussion, while highlighting the evidence used to make the appropriate decisions.
In making these decisions, the panels consider the consistency, extent, and quality of the data. Many of the issues considered by the panels are clear, requiring only limited discussion of fairly convincing data. Others are more ambiguous. In some cases, the data are limited or inconsistent; in others, interpretation of the data varies. In these cases, panel members review the total picture of the disease site and clinical situation. If other effective and safe options exist, a panel may decide not to recommend an intervention until more conclusive data are available. However, for clinical situations with no effective treatment options, even limited data may be acceptable. After all, clinicians treating patients with rare illnesses, for which few data exist, must still apply the best knowledge available to try to help. The NCCN Guidelines Insights are designed to assist readers in understanding some of the issues the panel members face and the ways they apply data to make recommendations.
We believe that these changes to JNCCN’s editorial format will increase the value of the NCCN Guidelines, both the full versions online and the Guidelines Insights in JNCCN, and that they will enhance NCCN and JNCCN’s ongoing mission of improving care for people with cancer.
What do you think? E-mail us at JNCCN@NCCN.org. We look forward to your feedback on this new feature.
Poonacha TK, Go RS. Level of scientific evidence underlying recommendations arising from the National Comprehensive Cancer Network clinical practice guidelines. J Clin Oncol 2011;29:186–191.