Presented here is an initial high-level attempt to take stock of the current evidence underlying guidelines for the treatment of patients with cancer. The primary aim in discussing these data is to spur ideation and hypothesis generation, as well as inform scientific and policy discussions concerning the state of clinical science in oncology.

The method used is a simple count of references included in the most recent version (either version 2011 or 2012) of each of the 38 disease-specific NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines).¹ This excludes NCCN Guidelines for detection, prevention, risk reduction, and supportive care. Admittedly, a simple count of references is a crude metric, which does not consider the heterogeneity of value among the data underlying specific NCCN Guidelines recommendations. For example, data from practice-changing phase III trials are arguably more valuable to a provider regarding therapy evaluation than data from a trial with less definitive results. However, even the latter example may contain important data on adverse events or drug dosing that inform a physician’s therapy recommendations.

Reliance on the citation list associated with an NCCN Guideline is dependent on the guideline itself having a comprehensive review of the data that is relevant to clinical decision-making. The process for evidence review and inclusion in the NCCN Guidelines is described in detail elsewhere,² but the general goal is to provide reliable and valid data that are most applicable to making decisions about patient care. The NCCN Guidelines are widely considered to represent the standard of care for patients with cancer in the United States and are produced by large multidisciplinary volunteer panels composed of leading disease-specific experts. Therefore, it is reasonable to expect that the data cited are most relevant to making patient care decisions.

Notably, many studies with results that may not directly inform patient care will be excluded. For example, a 2007 systematic review that examined gemcitabine treatment for bladder cancer identified 28 phase II studies,³ of which 4 studies are referenced and discussed in version 2.2012 of the NCCN Guidelines for Bladder Cancer (to view the most recent version of these or any other guidelines, visit the NCCN Web site at NCCN.org). Each study cited likely had supporting early-phase trials and considerable basic science and translational research that informed the cited reference’s design and execution. In addition, studies with null results that do not inform clinical decision-making are not likely to be cited. The magnitude of the difference between studies referenced in the NCCN Guidelines and the totality of research in oncology may be best exemplified in a recent bibliometric review of oncology research output, which identified more than 63,000 cancer-related PubMed references published in 2007 alone.⁴ This is in contrast to the approximate 8000 aggregate references cited across the NCCN Guidelines.

Considerable variation is observed in reference numbers per NCCN Guideline, with a 20-fold difference between the NCCN Guidelines for Dermatofibrosarcoma Protuberans (DFSP) and Non-Hodgkin’s Lymphomas (NHL; Figure 1). Some of this variation is attributable to the heterogeneity of disease covered within each guideline. For example, the 831 references included in the NCCN Guidelines for NHL include citations for well over a dozen distinct NHL histologies that are each associated with a unique treatment pathway. In addition, solid tumors, such as breast or non–small cell lung cancers (NSCLC), may be classified into subtypes of disease based on biomarkers and histology.^5,6 This is in contrast to diseases such as DFSP, which is understood to be a fairly uniform, highly treatable rare disease.⁷

View Full Size

Number of references included in each of the disease-specific NCCN Clinical Practice Guidelines in Oncology (versions 2011 or 2012).

Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 10, 5; 10.6004/jnccn.2012.0067

An additional root cause of variation is the research activity associated with different cancer types. Research is driven partly by need; therefore, the fact that breast cancer, NSCLC, and colon cancer top this list is not surprising, because these 3 cancer types are estimated to account for 34% of new cancers and 44% of cancer deaths in 2012.⁸ Furthermore, these diseases account for 21% of the $25 billion of National Institutes of Health cancer research funding allocated between 2008 and 2011.⁹

Each of the top 4 disease sites represented by the NCCN Guidelines showed a large increase in references between 2005 and 2011. The increase in reference numbers ranged in magnitude from a 3.9-fold increase in the NCCN Guidelines for Breast Cancer (133 to 523 references) to a 9.4-fold increase in those for NHL (88 to 831 references). This increase is partly from the expansion of recommendations for clinical situations, such as breast cancer during pregnancy, or the addition of new NHL subtypes, such as cutaneous B-cell lymphoma. However, much of this increase can also be attributed to the growth in the evidence that informs NCCN Guidelines recommendations. For example, 65% of the approximate 2100 references included in these 4 NCCN Guidelines to date were published since 2005.

These data have implications on the practice of oncology, namely the apparent variation in the evidence supporting cancer care and the growth in complexity of cancer care. The latter highlights the importance of effective and up-to-date continuing education for oncology providers. Furthermore, these data speak to a growing value associated with information resources and tools that aid in the identification and synthesis of a rapidly growing body of clinically relevant literature.

Additional work will be required to better understand the variation in evidence both between and within a disease site represented by an NCCN Guideline. This includes qualifying the level of evidence associated with each reference to oncology care and cross-referencing individual references with specific decision nodes along the continuum of care. This is critical to identifying points in care where evidence is lacking and that may benefit from clinical research in the form of clinical trials or observational or comparative effectiveness studies.