In spring of 2011, 11 NCCN Member Institutions were invited to participate in an opportunity to use the NCCN Oncology Outcomes Database to identify opportunities for improvement (OFI) in the care of patients with breast cancer and to implement measures that would target such improvement. The measures identified focused on concordance with specific recommendations in the NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines) for Breast Cancer (Table 1) or timely access to and administration of care. Each institution chose their project, based on opportunities specific to that institution.
Specific project assignments included oversight of project activities by a primary investigator at each institution, participation in multiple meetings and conference calls from 2011 through September 2013, and review of and response to nonconcordant reports.
Data Security
All aggregate, de-identified data were housed in the NCCN Breast Oncology Outcomes Database via individual institutional review board (IRB)-approved protocols, with an institution-specific identifier key known only to that institution. Patient-level performance data on individual measures were provided to the institution in a “locked” state so that only members of the individual institution, under an IRB-approved protocol, could “unlock” patient-specific data to identify measure-specific performance.
Methods
Baseline data were provided to the institution as indicated in Table 2 in spring 2011. Because the results at the institution were less than the targeted concordance rate of 85%, the measures chosen for attention were inva4a, inva9a, and JVBis1. The OFI group normalized definitions for guideline nonconcordance, as noted in the following sections.
Physician Discussed, Patient Declined
For each case of a patient decision to decline guideline-concordant care, there should be 1) source documentation of the provider’s recommendation of the guideline-concordant care, and 2) documentation that the patient or patient’s guardian declined care. If possible, a record of the discussion with the patient should be documented and reviewed to ensure adequate patient education was provided.
Physician-Related
Treatment Not Recommended by Physician: For each case of physician recommendations that are not guideline concordant, the recommendation for nonconcordant care should be based on source documentation that adequately addresses the reason for not recommending care. However, if the physician recommendation is based solely on advanced patient age or comorbid conditions, this reason should not be selected as a reason for noncondordance.
NCCN Guidelines Recommendations Targeted for Performance Improvement
Comorbidity or Advanced Age Alone: For each case of a patient not receiving recommended treatment because of comorbid clinical conditions or advanced age, source documentation should indicate that this was the sole factor on which the therapy decision was based.
Patient Received Non-Guideline Recommended Treatment: For each case in which a patient receives treatment that is outside the guideline-recommended options for therapy, notes from the treating physician should be included (but are not required), detailing recommendation for care and adequately addressing the reason for administering the therapy.
System-Related
Not Referred or Seen: This relates to each case for whom there is no documented patient referral or visit to the proper specialist for guideline-concordant care.
Inadequate Documentation in Chart: Cases in which the patient’s medical record does not contain enough information to determine the reason for nonconcordance fall under this explanation.
Too Early to Determine Whether Treatment Was Administered: This category encompasses each case in which the patient has not yet reached the point at which they should receive the recommended care. Reviewers may want to note the current treatment status and an estimate of the date at which the patient may receive the care for the purpose of revisiting the case to determine concordance.
Recommended Treatment Received, but Started Later Than Required: This category is used when a timing component is associated with the concordance measure and treatment is given outside of the window.
Database Issues
Patient Not Eligible for Treatment Due to Data Entry Error: In each case in which a patient, based on demographic and clinical characteristics, seems to not be eligible for a particular treatment because of an identified data entry or interpretation error, the error should be noted and the data in the database should be corrected.
Patient Not Eligible for Treatment Due to Unknown Error: This designation is used for cases in which a patient, based on their demographic and clinical characteristics, is not eligible for a particular treatment but the reason for the discrepancy cannot be identified (such as with analytical errors). The error should be noted and the appropriate personnel notified to work with project management to determine the underlying reason.
Data Entry Error: This category is used for cases believed to be concordant but for which a data error results in a nonconcordant status. The error should be noted and appropriate personnel notified to correct the data.
Institution-Specific Concordance Rates After Initial Quality Reviewa
Unknown or Analytic Error: This category is used for each case deemed to be concordant when the reason the case was classified as nonconcordant is unknown. The error should be noted and the personnel notified to work with project management to determine the underlying reason.
Results
Detailed individual chart review was performed by experienced data abstractors (Table 2), following the previously noted standardized definitions. A third review was performed by 2 medical oncologists with experience in reviewing data for quality measurement to better understand the reasons for nonconcordance.
Because of the lack of structured clinical data entry at this institution, the volume of clinical documents, the formatting of documentation both internally (inpatient notes) and externally (referral and follow-up notes) into scanned documents, and the multiple data sources required to review some of the data elements, such as order entry and administration (for the bisphosphonate measure), the OFI team felt this review should be performed by qualified physicians. This review required 30 to 45 minutes for each case previously scored as nonconcordant.
The intensive, focused, physician-only review of all documentation from all sources in the medical record showed a “real” concordance rate across all measures of 91% (range, 82%-97%). The 3 largest contributors to the discordance rate originally reported were 1) data entry error (N=43), an indication of the difficulty in locating the correct data source in unstructured medical record sources; 2) patient choice (N=16), which seemed, from the medical record review, more of an indication of informed patient choice rather than a true error in treatment recommendation or communication; and 3) comorbidity or advanced age (N=11), which, reflects that the NCCN Guidelines are written as guidelines and must be interpreted in light of other competing medical issues.
Conclusions
Although measuring performance against standardized norms and looking for relevant opportunities to improve are important, current data limitations must also be considered. Despite the formal definitions used in this effort and the involvement of experienced data abstractors with a strict database structure, subsequent clinical review showed results that were only approximations to reality. At this institution, a 5% to 15% variance was noted between initial review and intensive subsequent review, mostly due to system limitations. Patient choice and comorbidities will also contribute to variation concordance.
Given these limitations, seeking 100% concordance with any guideline is probably unrealistic. The appropriate target of concordance may be very situation-specific.
