Metastatic Versus Osteoporotic Vertebral Fractures on MRI: A Blinded, Multicenter, and Multispecialty Observer Agreement Evaluation

View More View Less
  • a Department of Radiology, Fundación Instituto Valenciano de Oncología, Valencia;
  • b Spanish Back Pain Research Network, Kovacs Foundation, Palma de Mallorca;
  • c Unidad de la Espalda Kovacs, Hospital Universitario HLA-Moncloa, Madrid;
  • d Clinical Biostatistics Unit, Instituto de Investigación Sanitaria Puerta de Hierro-Segovia de Arana, Madrid; CIBERESP;
  • e Department of Radiology, Hospital Universitario Regional de Málaga, Málaga;
  • f Department of Radiology, Hospital de Manacor, Mallorca;
  • g School of Biomedical Engineering, Universitat Politècnica de Valencia, Valencia;
  • h Department of Radiology, Hospital de Galdakao, Galdakao, Bizkaia;
  • i Department of Radiology, Hospital Universitario Puerta de Hierro, Madrid; and
  • j Department of Radiology, Hospital Son Llàtzer, Palma de Mallorca, Spain.
Restricted access

Background: MRI is assumed to be valid for distinguishing metastatic vertebral fractures (MVFs) from osteoporotic vertebral fractures (OVFs). This study assessed (1) concordance between the image-based diagnosis of MVF versus OVF and the reference (biopsy or follow-up of >6 months), (2) interobserver and intraobserver agreement on key imaging findings and the diagnosis of MVF versus OVF, and (3) whether disclosing a patient’s history of cancer leads to variations in diagnosis, concordance, or agreement. Patients and Methods: This retrospective cohort study included clinical data and imaging from 203 patients with confirmed MVF or OVF provided to 25 clinicians (neurosurgeons, radiologists, orthopedic surgeons, and radiation oncologists). From January 2018 through October 2018, the clinicians interpreted images in conditions as close as possible to routine practice. Each specialist assessed data twice, with a minimum 6-week interval, blinded to assessments made by other clinicians and to their own previous assessments. The kappa statistic was used to assess interobserver and intraobserver agreement on key imaging findings, diagnosis (MVF vs OVF), and concordance with the reference. Subgroup analyses were based on clinicians’ specialty, years of experience, and complexity of the hospital where they worked. Results: For diagnosis of MVF versus OVF, interobserver agreement was fair, whereas intraobserver agreement was substantial. Only the latter improved to almost perfect when a patient’s history of cancer was disclosed. Interobserver agreement for key imaging findings was fair or moderate, whereas intraobserver agreement on key imaging findings was moderate or substantial. Concordance between the diagnosis of MVF versus OVF and the reference was moderate. Results were similar regardless of clinicians’ specialty, experience, and hospital category. Conclusions: When MRI is used to distinguish MVF versus OVF, interobserver agreement and concordance with the reference were moderate. These results cast doubt on the reliability of basing such a diagnosis on MRI in routine practice.

Submitted July 7, 2019; accepted for publication October 7, 2019.

Author contributions: Study concept: Arana, Kovacs. Study design: Arana, Kovacs, Royuela. Data collection: Asenjo, Nagib, Pérez-Aguilera, Dejoz, Cabrera-Zubizarreta, García-Hidalgo, Estremera. Quality assessment and logistics: Arana, Dejoz. Data analysis: Royuela. Guidance on clinical practice of vertebral fractures: Arana. Writing—original draft and revision: Arana, Kovacs, Royuela. Writing—review and editing: All authors. Critical revisions: All authors.

Disclosures: The authors have disclosed that they have not received any financial consideration from any person or organization to support the preparation, analysis, results, or discussion of this article.

Funding: Funding for this work was provided by The Kovacs Foundation.

Correspondence: Estanislao Arana, MD, MHE, PhD, Servicio de Radiología, Fundación Instituto Valenciano de Oncología, C/ Beltrán Báguena, 19, 46009 Valencia, Spain. Email: Estanis.Arana@ext.uv.es

View associated content

Supplementary Materials

    • Supplemental Materials (PDF 493.70 KB)
  • 1.

    Kato S, Hozumi T, Yamakawa K, . META: an MRI-based scoring system differentiating metastatic from osteoporotic vertebral fractures. Spine J 2015;15:15631570.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2.

    Thawait SK, Kim J, Klufas RA, . Comparison of four prediction models to discriminate benign from malignant vertebral compression fractures according to MRI feature analysis. AJR Am J Roentgenol 2013;200:493502.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3.

    Wang KC, Jeanmenne A, Weber GM, . An online evidence-based decision support system for distinguishing benign from malignant vertebral compression fractures by magnetic resonance imaging feature analysis. J Digit Imaging 2011;24:507515.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4.

    Thawait SK, Marcus MA, Morrison WB, . Research synthesis: what is the diagnostic performance of magnetic resonance imaging to discriminate benign from malignant vertebral compression fractures? Systematic review and meta-analysis. Spine (Phila Pa 1976) 2012;37:E736744.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 5.

    Kottner J, Audigé L, Brorson S, . Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol 2011;64:96106.

  • 6.

    Departamento de Métodos Cuantitativos en Economía y Gestión U de LP de GC. Clasificación de hospitales públicos españoles mediante el uso del análisis cluster [Internet], 2007. Accessed October 10, 2019. Available at: http://www.icmbd.es/docs/resumenClusterHospitales.pdf.

  • 7.

    Bilsky MH, Laufer I, Fourney DR, . Reliability analysis of the epidural spinal cord compression scale. J Neurosurg Spine 2010;13:324328.

  • 8.

    Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159174.

  • 9.

    Curtis JR, Taylor AJ, Matthews RS, . “Pathologic” fractures: should these be included in epidemiologic studies of osteoporotic fractures? Osteoporos Int 2009;20:19691972.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10.

    Shankar PR, Kaza RK, Al-Hawary MM, . Impact of clinical history on maximum PI-RADS version 2 score: a six-reader 120-case sham history retrospective evaluation. Radiology 2018;288:158163.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 11.

    Levine D, Bankier AA, Halpern EF. Submissions to radiology: our top 10 list of statistical errors. Radiology 2009;253:288290.

  • 12.

    Elmore JG, Longton GM, Carney PA, . Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA 2015;313:11221132.

  • 13.

    Sherbeck JP, Zhao L, Lieberman RW. High variability in lymph node counts among an international cohort of pathologists: questioning the scientific validity of node counts. J Natl Compr Canc Netw 2018;16:395401.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 14.

    Itri JN, Patel SH. Heuristics and cognitive error in medical imaging. AJR Am J Roentgenol 2018;210:10971105.

  • 15.

    Urrutia J, Besa P, Morales S, . Does the META score evaluating osteoporotic and metastatic vertebral fractures have enough agreement to be used by orthopaedic surgeons with different levels of training? Eur Spine J 2018;27:25772583.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16.

    Mauch JT, Carr CM, Cloft H, . Review of the imaging features of benign osteoporotic and malignant vertebral compression fractures. AJNR Am J Neuroradiol 2018;39:15841592.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17.

    Sung JK, Jee WH, Jung JY, . Differentiation of acute osteoporotic and malignant compression fractures of the spine: use of additive qualitative and quantitative axial diffusion-weighted MR imaging to conventional MR imaging at 3.0 T. Radiology 2014;271:488498.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 18.

    Besa P, Urrutia J, Campos M, . The META score for differentiating metastatic from osteoporotic vertebral fractures: an independent agreement assessment. Spine J 2018;18:20742080.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 19.

    Shank CD, Lepard JR, Walters BC, . Towards evidence-based guidelines in neurological surgery. Neurosurgery 2019;85:613621.

  • 20.

    Loy CT, Irwig L. Accuracy of diagnostic tests read with and without clinical information: a systematic review. JAMA 2004;292:16021609.

  • 21.

    Carney PA, Cook AJ, Miglioretti DL, . Use of clinical history affects accuracy of interpretive performance of screening mammography. J Clin Epidemiol 2012;65:219230.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 22.

    Waite S, Scott J, Gale B, . Interpretive error in radiology. AJR Am J Roentgenol 2017;208:739749.

  • 23.

    Jarvik JG, Deyo RA. Moderate versus mediocre: the reliability of spine MR data interpretations. Radiology 2009;250:1517.

  • 24.

    Wood KB, Khanna G, Vaccaro AR, . Assessment of two thoracolumbar fracture classification systems as used by multiple surgeons. J Bone Joint Surg Am 2005;87:14231429.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 25.

    Arana E, Kovacs FM, Royuela A, . Agreement in the assessment of metastatic spine disease using scoring systems. Radiother Oncol 2015;115:135140.

  • 26.

    Arana E, Kovacs FM, Royuela A, . Agreement in metastatic spinal cord compression. J Natl Compr Canc Netw 2016;14:7076.

  • 27.

    Fisher CG, Versteeg AL, Schouten R, . Reliability of the spinal instability neoplastic scale among radiologists: an assessment of instability secondary to spinal metastases. AJR Am J Roentgenol 2014;203:869874.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28.

    Arana E, Kovacs FM, Royuela A, . Spine instability neoplastic score: agreement across different medical and surgical specialties. Spine J 2016;16:591599.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29.

    Khan L, Mitera G, Probyn L, . Inter-rater reliability between musculoskeletal radiologists and orthopedic surgeons on computed tomography imaging features of spinal metastases. Curr Oncol 2011;18:e282287.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1132 1132 47
PDF Downloads 426 426 13
EPUB Downloads 0 0 0