Background
Malignant or metastatic spinal cord compression of the thecal sac is a devastating medical emergency presented by 5% to 20% of patients with spinal metastases.1 It can be caused by vertebral collapse, but is usually provoked by soft tissue causing epidural spinal cord compression (ESCC).2
Clinical symptoms and the ESCC grade are the major determinants in the decision to operate or irradiate.3,4 The ESCC score system is a 6-point scale for diagnosing and reporting ESCC based on imaging findings (Figure 1).3 It was developed by oncologic spine surgeons and proven to be reliable among a small sample of these specialists.3 However, managing spinal cord compression requires a multidisciplinary approach,5,6 and the lack of nomenclature standardization prevents agreement in decision-making,7 delays appropriate treatment, and hinders treatment effectiveness.8,9
Therefore, the purpose of this study was to assess intraobserver and interobserver agreement in identification of spine level involved in each patient and in the calculation of the ESCC score among a large sample of clinicians from different specialties with varied degrees of experience and working in different settings and locations.
Methods
This study was approved by the Institutional Review Boards of the participating hospitals, and complied with the Guidelines for Reporting Reliability and Agreement Studies (GRRAS).10
The 6-point ESCC classification according to Bilsky et al.3 (A) Grade 0 indicates bone-only disease. (B) Grade 1a, epidural impingement, without deformation of the thecal sac. (C) Grade 1b, deformation of the thecal sac, without spinal cord abutment. (D) Grade 1c, deformation of the thecal sac with spinal cord abutment, but without cord compression. (E) Grade 2, spinal cord compression, but with cerebrospinal fluid (CSF) visible around the cord. (F) Grade 3, spinal cord compression, no CSF visible around the cord.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 14, 1; 10.6004/jnccn.2016.0008
The 6-point ESCC classification according to Bilsky et al.3 (A) Grade 0 indicates bone-only disease. (B) Grade 1a, epidural impingement, without deformation of the thecal sac. (C) Grade 1b, deformation of the thecal sac, without spinal cord abutment. (D) Grade 1c, deformation of the thecal sac with spinal cord abutment, but without cord compression. (E) Grade 2, spinal cord compression, but with cerebrospinal fluid (CSF) visible around the cord. (F) Grade 3, spinal cord compression, no CSF visible around the cord.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 14, 1; 10.6004/jnccn.2016.0008
The 6-point ESCC classification according to Bilsky et al.3 (A) Grade 0 indicates bone-only disease. (B) Grade 1a, epidural impingement, without deformation of the thecal sac. (C) Grade 1b, deformation of the thecal sac, without spinal cord abutment. (D) Grade 1c, deformation of the thecal sac with spinal cord abutment, but without cord compression. (E) Grade 2, spinal cord compression, but with cerebrospinal fluid (CSF) visible around the cord. (F) Grade 3, spinal cord compression, no CSF visible around the cord.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 14, 1; 10.6004/jnccn.2016.0008
Selection of Hospital Departments and Clinicians
Sample size was calculated assuming that the prevalence of ESCC in a particular spinal level would be 10%, and that at least 5 readers would be recruited per medical specialty and 5 per hospital category. In order to ensure that, should κ values reflect an “almost perfect” agreement (κ=0.81), the the lower limit of the 95% CI would lie within the range classified as reflecting a substantial agreement (κ=0.61–0.80), the sample size was established at 90 patients. Sample size calculations were performed using κ size package of the R library.11
The 61 hospital departments specializing in radiology, medical oncology, radiation oncology, neurosurgery, and orthopedic surgery, which had previously participated in studies undertaken by the Spanish Back Pain Research Network (REIDE) or had expressed interest in doing so, were invited to participate in this study. Twelve departments were located in 6 private hospitals and the other 49 in 38 nonprofit hospitals, belonging to or working for the Spanish National Health Service (SNHS). The SNHS is the tax-funded, government-owned organization that provides free health care to every resident in Spain.
The SNHS classifies hospitals into 5 categories based on the size of the catchment area; number of beds; number of clinicians; availability of high-tech medical equipment and procedures; education, training, and academic activity; and clinical complexity of the cases treated (ie, being the reference hospital for specific diseases or procedures).12 Category 1 is the simplest and category 5 is the most complex. Departments invited to participate in this study were located in hospitals belonging to categories 2, 3, 4, and 5.
All clinicians who had finished their residency and worked at the participating departments were invited to act as readers in this study, and asked to report the number of years they had been working in clinical practice after their residency. The departments and clinicians did not receive any compensation for participating in this study.
Patients and Images Selection
A radiologist at a category 4 hospital, who did not act as reader, was responsible for selecting patients and images for study inclusion. He identified patients who had undergone an MRI in his department for spinal cord compression and whose ESCC scores had been rated by a tumor board.
The tumor board comprised a medical oncologist, a radiation oncologist, a radiologist, a pathologist, an orthopedic surgeon, and a neurosurgeon. None of its members acted as readers for the study.
For each case, demographic data, histopathology, and a pain description with an emphasis on neurologic signs were provided to simulate information typically provided to any physician in routine practice. All MR imaging had been performed with a 1.5-T unit (Magnetom Symphony; Siemens, Erlangen, Germany) with a spinal matrix coil. The recruiting radiologist selected 2 images per patient: a sagittal T2-weighted turbo spin-echo sequence (4000/115; section thickness, 4 mm) and an axial T2-weighted turbo spin-echo sequence (4500/110; section thickness, 5 mm) at maximal ESCC grade.3 The sagittal MRI image included at least 2 spine segments.13
The first 90 cases that complied with inclusion criteria were selected. Inclusion criterion was presentation with stage IV (AJCC classification, 7th Edition, 2010) metastatic spine disease confirmed with biopsy. Exclusion criteria included clinical history lacking data required to assess ESCC or imaging of insufficient quality to assess the spinal levels affected.
Procedure
The recruiting radiologist prepared an information pack corresponding to each patient, comprising 2 images and a clinical vignette that included the patient's age, oncologic history, clinical signs, and symptoms (Figure 2).3 Patient identity was masked and a code was assigned to each information pack. All of the information packs were uploaded to an online platform designed for this study (http://www.typeform.com/).
Each reader was provided with a personal password to assess the information packs online. Readers were asked to indicate all the spinal segments in which they identified metastases for each patient (cervical, thoracic, lumbar, and/or sacral) and to calculate the ESCC score. They were only provided with definitions included in the ESCC (Figure 1). No attempt was made to further define or standardize the meaning of the terms included in the scoring systems or to homogenize the diagnostic criteria, and readers did not receive any instructions regarding the interpretation of images. They were told to use their own clinical judgment when in doubt.
Readers assessed the information pack on their own and uploaded the resulting report directly onto the online platform. They assessed the same clinical sets twice, with a minimum interval period of 6 weeks. The platform software ensured that the minimum interval period was observed, and that readers had no access to their own previous reports or to their colleagues' uploaded reports.
Data introduced into the platform were automatically converted into a spreadsheet. The software engineer in charge of developing the platform cross-checked that the data in the database
An example of the information pack provided to readers for each patient.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 14, 1; 10.6004/jnccn.2016.0008
An example of the information pack provided to readers for each patient.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 14, 1; 10.6004/jnccn.2016.0008
An example of the information pack provided to readers for each patient.
Citation: Journal of the National Comprehensive Cancer Network J Natl Compr Canc Netw 14, 1; 10.6004/jnccn.2016.0008
Statistical Analysis
At the analysis phase, grades 1a, 1b, and 1c were grouped, resulting in a 4-point ESCC: 0, 1 (including 1a, 1b, and 1c), 2, and 3. Data on the spine level affected in each patient was classified as cervical, thoracic, lumbar, or sacral, and rated as yes or no.
To assess intraobserver and interobserver agreement, ratings from each reader were cross-tabulated and the Fleiss κ statistic was calculated. A weighted κ approach, with a bisquared weighting scheme, was used to analyze the agreement when using the ESCC scoring system. Kappa values were categorized as “almost perfect” (0.81–1.00), “substantial” (0.61–0.80), “moderate” (0.41–0.60), “fair” (0.21–0.40), “slight” (0.00–0.20), and “poor” (<0.00).14
To assess intraobserver agreement for each variable (ESCC score and levels involved), a κ index was calculated for each of the 83 readers, and median, 5th and 95th percentiles values were calculated.
To assess interobserver agreement, the corresponding κ index was calculated and the 95% CI was determined following the jackknife resampling method.15
Subgroup analyses for each variable were performed, in which κ values were calculated separately depending on medical specialty, hospital category, and professional experience. Professional experience was classified as “junior” (≤7 years in practice, after residency), “experienced” (8–13 years), and “senior specialist” (≥14 years).
The ESCC scores established by the tumor board were subsequently classified into grades 0, 1, 2, and 3. These grades were used as the gold standard to assess overall agreement. The agreement between this gold standard and the median score for each image among the 83 readers was calculated through the κ statistic.
Stata 13 software was used (StataCorp 2013; Stata Statistical Software: Release 13, College Station, TX).
Results
Of the 132 clinicians invited to act as readers, 83 (62.87%) participated in this study. The first 90 patients selected by the recruiting radiologist (51 women and 39 men; mean age, 60.8 years) complied with the inclusion criteria, and none were excluded. These 90 patients presented metastases in 182 spinal segments. Table 1 shows sample characteristics.
Sample Characteristics
There were more than 5 readers for each specialty and degree of professional experience. However, only 3 readers worked at category 2 hospitals; therefore, agreement for this subgroup was not calculated (Tables 2 and 3).
Regarding the identification of spine levels showing ESCC, intraobserver and interobserver agreements were substantial (κ value: median, 0.772 [5th, 95th percentiles: 0.541, 0.948], and κ value: 0.610 [95% CI, 0.531, 0.696], respectively) (Table 2). Subgroup analyses showed that interobserver agreement was only moderate among junior specialists, those working in category 3 hospitals, and in all specialties except radiation oncology (Table 2).
Regarding ESCC score, intraobserver agreement was almost perfect (κ value: median, 0.819 [5th, 95th percentiles: 0.636, 0.923]), and interobserver agreement was substantial (κ value: 0.635 [95% CI, 0.578, 0.699]). Subgroup analyses showed that intraobserver agreement was only substantial among junior specialists, orthopedic surgeons, medical oncologists, and radiation oncologists, whereas interobserver agreement was only moderate among junior and experienced specialists, orthopedic surgeons, and medical oncologists, and among those working in category 3 and 4 hospitals (Table 3).
The agreement between the median of the scores calculated by the readers, and the ESCC grades based on the scores established by the tumor board, was 0.713 (95% CI, 0.596–0.835). Classification by readers and by the tumor board coincided in all of the 31 patients in whom compressive findings were observed (ESCC 2 and 3) (Table 4).
Discussion
Results from this study show that there is a substantial interobserver agreement in determining the ESCC score. These results are generally consistent; differences across specialties, number of years of experience, and type of hospital are small. Results from this study show that there is substantial interobserver agreement in determining the ESCC score. Although some differences in κ values across hospitals, specialties, and number of years of experience were documented, the 95% CI of these values overlap, and differences are small and likely to be clinically meaningless (Tables 2 and 3).10
Intraobserver and Interobserver Agreement on Level of Cord Compression
The substantial agreement in identifying the spinal level showing ESCC based on MRI is reassuring, because clinical symptoms are unreliable for selecting the target level.13
This study assessed the reliability of the ESCC score across the different specialties involved in the management of ESCC, in conditions as close as possible to routine practice. All patients showed lesions at 2 or more spine levels and clinicians had to identify the target vertebra based on clinical judgment, as is often the case in clinical practice.16 A high number of readers participated, and they had different
Intraobserver and Interobserver Agreement on Spinal Cord Compression Score
Cross-Tabulation of Scores Determined by ESCC Board Tumor, and Median Categorization of Readers
The similarity of results obtained by physicians, irrespective of years of experience and hospital category, is also generally consistent with previous studies and supports current routine practice19; patients with cancer in whom ESCC is suspected undergo MRI at their hospital and are referred to surgery when deemed appropriate.1 It is reassuring that all of the patients who experienced compressive grades of ESCC (grades 2 and 3), and therefore required urgent clinical management, were correctly identified in this study (Table 4).19,22 This is consistent with previous studies that showed the sensitivity of the ESCC scoring system for detecting such cases.3 However, this does not necessarily imply a perfect external validity, because gold standards are difficult to define for metastatic spine disease,23,24 and discrepancies between imaging and real surgical outcomes do exist.7
Good communication among clinicians involved in the management of spine metastatic disease leads to consistency of care, which is a prerequisite for effectiveness.22 For instance, good communication between surgeons and radiation oncologists facilitates rapid identification of patients with epidural disease in whom surgical resection improves results from postoperative stereotactic body radiotherapy (SBRT).4 Ensuring that the diagnostic instruments used are reliable, is probably the most effective means of decreasing inappropriate variability in health care.25 Results from this study suggest that using the ESCC score can be useful to ensure accurate communication among multidisciplinary team members and, therefore, should be used routinely.6 However, it should be kept in mind that the intrinsic characteristics of certain tumors make it impossible to reach good agreement when it comes to their assessment and management, even after repeated training.26 Furthermore, agreement when using a scoring system does not necessarily mean that the resulting recommended treatment is appropriate, because clinicians sometimes agree on measures that are not evidence-based or effective,27 and an improvement in the quality of oncologic care does not necessarily translate immediately into better clinical results or improved survival rates.28 In fact, no current scoring system is robust enough to establish a solid prognosis for all patients with spinal metastases.29
This study has some limitations. Readers were only provided with 2 selected images per case. It is possible that providing all of the readers with all of the images available for each patient might have changed the degree of agreement. However, providing a selection of images ensures that all of the readers assess the same ones, and is consistent with the procedure followed by high-quality studies assessing reliability.3,30 Readers were volunteers from each of the hospital departments participating in this study, and were not randomly selected. Therefore, selection bias may exist; it is possible that physicians who agreed to participate in this study were the most motivated or interested in metastatic spine disease.31 However, clinicians involved in management of spinal metastases in routine clinical practice are usually highly specialized, and this does not challenge the results from this study. The prevalence of patients with grades 1a, 1b, and 1c ESCC made it necessary to merge these categories into a single category (grade 1). Maintaining the 3 subcategories would have led to groups too small for the κ statistic to reliably reflect the degree of agreement.32 In fact, subscale analysis of grade 1 ESCC was not performed in this study or in its original design.3 Moreover, there are no guidelines on the dosage of irradiation suitable for such cases19; for instance, more careful planning of radiotherapy is required for grade 1c ESCC than for grades 1a and 1b in order to avoid reaching the dosage above which the risk of spinal cord radiation overdose and myelopathy increases significantly.3,33
Conclusions
This study suggests that there is substantial agreement among radiologists, medical oncologists, radiation oncologists, orthopedic surgeons, and neurosurgeons when identifying the spinal level affected by metastases and when using the ESCC scoring system. Therefore, although there is room for improvement, the use of the ESCC score in clinical practice could improve communication among specialists involved in the management of spine metastases.
Acknowledgments
The authors wish to thank the following investigators of Spanish Back Pain Research Network Task Force for the improvement of interdisciplinary management of spinal metastasis (see supplemental eAppendix 1). The authors are also grateful to Prof. David Moratal for his continuous collaboration.
See JNCCN.org for supplemental online content.
References
- 1.↑
Sutcliffe P, Connock M, Shyangdan D et al.. A systematic review of evidence on malignant spinal metastases: natural history and technologies for identifying patients at high risk of vertebral fracture and spinal cord compression. Health Technol Assess 2013;17:1–274.
- 2.↑
Schmidt MH, Klimo P, Vrionis FD. Metastatic spinal cord compression. J Natl Compr Cancer Netw 2005;3:711–719.
- 3.↑
Bilsky MH, Laufer I, Fourney DR et al.. Reliability analysis of the epidural spinal cord compression scale. J Neurosurg Spine 2010;13:324–328.
- 4.↑
Al-Omair A, Masucci L, Masson-Cote L et al.. Surgical resection of epidural disease improves local control following postoperative spine stereotactic body radiotherapy. Neuro Oncol 2013;15:1413–1419.
- 5.↑
L'espérance S, Vincent F, Gaudreault M et al.. Treatment of metastatic spinal cord compression: CEPO review and clinical recommendations. Curr Oncol 2012;19:e478–490.
- 6.↑
Arana E, Kovacs FM, Royuela A et al.. Agreement in the assessment of metastatic spine disease using scoring systems. Radiother Oncol 2015;115:135–140.
- 7.↑
Putz C, Gantz S, Bruckner T et al.. Preoperative scoring and limits of prognostication: functional outcome after surgical decompression in metastatic spinal cord compression. Oncology 2014;86:177–184.
- 8.↑
Brooks FM, Ghatahora A, Brooks MC et al.. Management of metastatic spinal cord compression: awareness of NICE guidance. Eur J Orthop Surg Traumatol 2014;24(Suppl 1):S255–259.
- 9.↑
McGivern UM, Drinkwater KJ, Clarke JI, Locke I. A Royal College of Radiologists national audit of radiotherapy in the treatment of metastatic spinal cord compression and implications for the development of acute oncology services. Clin Oncol (R Coll Radiol) 2014;26:453–460.
- 10.↑
Kottner J, Audigé L, Brorson S et al.. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol 2011;64:96–106.
- 11.↑
Donner A, Rotondi MA. Sample size requirements for interval estimation of the kappa statistic for interobserver agreement studies with a binary outcome and multiple raters. Int J Biostat 2010;6:Article 31.
- 12.↑
[Spanish hospitals classification by cluster analysis.] 2007. Available at: http://www.icmbd.es/docs/resumenClusterHospitales.pdf. Accessed December 7, 2015.
- 13.↑
Venkitaraman R, Sohaib SA, Barbachano Y et al.. Frequency of screening magnetic resonance imaging to detect occult spinal cord compromise and to prevent neurological deficit in metastatic castration-resistant prostate cancer. Clin Oncol 2010;22:147–152.
- 14.↑
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159.
- 15.↑
Efron B. Nonparametric estimates of standard error: the jackknife, the bootstrap and other methods. Biometrika 1981;68:589–99.
- 16.↑
Keil S, Barabasch A, Dirrichs T et al.. Target lesion selection: an important factor causing variability of response classification in the Response Evaluation Criteria for Solid Tumors 1.1. Invest Radiol 2014;49:509–517.
- 17.↑
Obuchowski NA. How many observers are needed in clinical studies of medical imaging? AJR Am J Roentgenol 2004;182:867–869.
- 18.↑
Fisher CG, Schouten R, Versteeg AL et al.. Reliability of the Spinal Instability Neoplastic Score (SINS) among radiation oncologists: an assessment of instability secondary to spinal metastases. Radiat Oncol 2014;9:69.
- 19.↑
Cox BW, Spratt DE, Lovelock M et al.. International Spine Radiosurgery Consortium consensus guidelines for target volume definition in spinal stereotactic radiosurgery. Int J Radiat Oncol Biol Phys 2012;83:e597–605.
- 20.↑
Wu DH, Mayr NA, Karatas Y et al.. Interobserver variation in cervical cancer tumor delineation for image-based radiotherapy planning among and within different specialties. J Appl Clin Med Phys 2005;6:106–110.
- 22.↑
Terezakis SA, Heron DE, Lavigne RF et al.. What the diagnostic radiologist needs to know about radiation oncology. Radiology 2011;261:30–44.
- 23.↑
Bankier AA, Levine D, Halpern EF, Kressel HY. Consensus interpretation in imaging research: is there a better way? Radiology 2010;257:14–17.
- 24.↑
Harel R, Angelov L. Spine metastases: current treatments and future directions. Eur J Cancer 2010; 46: 2696–707.
- 25.↑
Tuijn S, Janssens F, Robben P, van den Bergh H. Reducing interrater variability and improving health care: a meta-analytical review. J Eval Clin Pract 2012;18:887–895.
- 26.↑
Rock JB, Washington MK, Adsay NV et al.. Debating deposits: an interobserver variability study of lymph nodes and pericolonic tumor deposits in colonic adenocarcinoma. Arch Pathol Lab Med 2014;138:636–642.
- 27.↑
Dea N, Fisher CG. Evidence-based medicine in metastatic spine disease. Neurol Res 2014;36:524–529.
- 28.↑
Keating NL, Landrum MB, Lamont EB et al.. Tumor boards and the quality of cancer care. J Natl Cancer Inst 2013;105:113–121.
- 29.↑
Tokuhashi Y, Uei H, Oshima M, Ajiro Y. Scoring system for prediction of metastatic spine tumor prognosis. World J Orthop 2014;5:262–271.
- 30.↑
Fourney DR, Frangou EM, Ryken TC et al.. Spinal instability neoplastic score: an analysis of reliability and validity from the spine oncology study group. J Clin Oncol 2011;29:3072–3077.
- 31.↑
Lim C, Cheung MC, Franco B et al.. Quality improvement: an assessment of participation and attitudes of medical oncologists. J Oncol Pract 2014;10:e408–414.
- 32.↑
Lantz CA, Nebenzahl E. Behavior and interpretation of the kappa statistic: Resolution of the two paradoxes. J Clin Epidemiol 1996;49:431–434.
- 33.↑
Sahgal A, Weinberg V, Ma L et al.. Probabilities of Radiation Myelopathy Specific to Stereotactic Body Radiation Therapy to Guide Safe Practice. Int J Radiat Oncol 2013;85:341–347.
