Enhancing Readability of Online Patient-Facing Content: The Role of AI Chatbots in Improving Cancer Information Accessibility

Authors:
Andres A. Abreu Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX

Search for other papers by Andres A. Abreu in
Current site
Google Scholar
PubMed
Close
 MD
,
Gilbert Z. Murimwa Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX

Search for other papers by Gilbert Z. Murimwa in
Current site
Google Scholar
PubMed
Close
 MD
,
Emile Farah Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX

Search for other papers by Emile Farah in
Current site
Google Scholar
PubMed
Close
 MD
,
James W. Stewart II Department of Surgery, Yale School of Medicine, New Haven, CT

Search for other papers by James W. Stewart II in
Current site
Google Scholar
PubMed
Close
 MD
,
Lucia Zhang Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX

Search for other papers by Lucia Zhang in
Current site
Google Scholar
PubMed
Close
 BS
,
Jonathan Rodriguez Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX

Search for other papers by Jonathan Rodriguez in
Current site
Google Scholar
PubMed
Close
 BS
,
John Sweetenham Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX

Search for other papers by John Sweetenham in
Current site
Google Scholar
PubMed
Close
 MD
,
Herbert J. Zeh III Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX

Search for other papers by Herbert J. Zeh III in
Current site
Google Scholar
PubMed
Close
 MD
,
Sam C. Wang Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX

Search for other papers by Sam C. Wang in
Current site
Google Scholar
PubMed
Close
 MD
, and
Patricio M. Polanco Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX

Search for other papers by Patricio M. Polanco in
Current site
Google Scholar
PubMed
Close
 MD
Restricted access

Background: Internet-based health education is increasingly vital in patient care. However, the readability of online information often exceeds the average reading level of the US population, limiting accessibility and comprehension. This study investigates the use of chatbot artificial intelligence to improve the readability of cancer-related patient-facing content. Methods: We used ChatGPT 4.0 to rewrite content about breast, colon, lung, prostate, and pancreas cancer across 34 websites associated with NCCN Member Institutions. Readability was analyzed using Fry Readability Score, Flesch-Kincaid Grade Level, Gunning Fog Index, and Simple Measure of Gobbledygook. The primary outcome was the mean readability score for the original and artificial intelligence (AI)–generated content. As secondary outcomes, we assessed the accuracy, similarity, and quality using F1 scores, cosine similarity scores, and section 2 of the DISCERN instrument, respectively. Results: The mean readability level across the 34 websites was equivalent to a university freshman level (grade 13±1.5). However, after ChatGPT’s intervention, the AI-generated outputs had a mean readability score equivalent to a high school freshman education level (grade 9±0.8). The overall F1 score for the rewritten content was 0.87, the precision score was 0.934, and the recall score was 0.814. Compared with their original counterparts, the AI-rewritten content had a cosine similarity score of 0.915 (95% CI, 0.908–0.922). The improved readability was attributed to simpler words and shorter sentences. The mean DISCERN score of the random sample of AI-generated content was equivalent to “good” (28.5±5), with no significant differences compared with their original counterparts. Conclusions: Our study demonstrates the potential of AI chatbots to improve the readability of patient-facing content while maintaining content quality. The decrease in requisite literacy after AI revision emphasizes the potential of this technology to reduce health care disparities caused by a mismatch between educational resources available to a patient and their health literacy.

Submitted September 26, 2023; final revision received December 19, 2023; accepted for publication December 21, 2023. Published online May 15, 2024.

Author contributions: Study concept & design: Abreu, Murimwa, Polanco. Data acquisition: Abreu, Farah, Zhang, Rodriguez. Data analysis: Abreu. Data interpretation: All authors. Writing—original draft: Abreu, Murimwa, Wang, Polanco. Writing—review & editing: All authors. Final approval of manuscript: All authors.

Disclosures: Dr. Zeh has disclosed serving as a scientific advisor for Surgical Safety Technologies. Dr. Polanco has disclosed serving as a consultant for Iota Biosciences and Palisade Bio; and serving as a proctor for Intuitive Surgical. The remaining authors have not received any financial consideration from any person or organization to support the preparation, analysis, results, or discussion of this article.

Supplementary material: Supplementary material associated with this article is available online at https://doi.org/10.6004/jnccn.2023.7334. The supplementary material has been supplied by the author(s) and appears in its originally submitted form. It has not been edited or vetted by JNCCN. All contents and opinions are solely those of the author. Any comments or questions related to the supplementary materials should be directed to the corresponding author.

Correspondence: Patricio M. Polanco, MD, Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, 5323 Harry Hines Boulevard, Dallas, TX 75390. Email: patricio.polanco@utsouthwestern.edu

Supplementary Materials

    • Supplemental Materials (PDF 3.61 MB)
  • Collapse
  • Expand
  • 1.

    Baker L, Wagner TH, Singer S, et al. Use of the internet and e-mail for health care information: results from a national survey. JAMA 2003;289:24002406.

  • 2.

    Cohen RA, Stussman B. Health information technology use among men and women aged 18–64: early release of estimates from the National Health Interview Survey, January–June 2009. Accessed October 10, 2023. Available at: https://www.cdc.gov/nchs/data/hestat/healthinfo2009/healthinfo2009.pdf

    • PubMed
    • Export Citation
  • 3.

    van de Poll-Franse LV, van Eenbergen M. Internet use by cancer survivors: current use and future wishes. Support Care Cancer 2008;16:11891195.

  • 4.

    Weiss BD, Schwartzberg JG, Davis TC, et al. Health literacy: a manual for clinicians. Accessed October 5, 2023. Available at: http://lib.ncfh.org/pdfs/6617.pdf

  • 5.

    Weiss BD, Blanchard JS, McGee DL, et al. Illiteracy among Medicaid recipients and its relationship to health care costs. J Health Care Poor Underserved 1994;5:99111.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6.

    Hickson GB, Clayton EW, Githens PB, et al. Factors that prompted families to file medical malpractice claims following perinatal injuries. JAMA 1992;267:13591363.

  • 7.

    Vincent C, Young M, Phillips A. Why do people sue doctors? A study of patients and relatives taking legal action. Lancet 1994;343:16091613.

  • 8.

    Schillinger D, Grumbach K, Piette J, et al. Association of health literacy with diabetes outcomes. JAMA 2002;288:475482.

  • 9.

    Gilliland J. The concept of readability. Literacy 1968;2:2429.

  • 10.

    Ochiai E, Kigenyi T, Sondik E, et al. Healthy People 2030 leading health indicators and overall health and well-being measures: opportunities to assess and improve the health and well-being of the nation. J Public Health Manag Pract 2021;27(Suppl 6):S235241.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11.

    Brach C, ed. AHRQ health literacy universal precautions toolkit: 3rd edition. Accessed January 1, 2024. Available at: https://www.ahrq.gov/sites/default/files/publications2/files/health-literacy-universal-precautions-toolkit-3rd-edition.pdf

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 12.

    Murimwa GZ, Stewart JW 2nd, Zhang L, et al. Readability and non- English language resources of hepatobiliary cancer center websites in the USA. Ann Surg Oncol 2023;30:38983900.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13.

    Lamb LR, Baird GL, Roy IT, et al. Are English-language online patient education materials related to breast cancer risk assessment understandable, readable, and actionable? Breast 2022;61:2934.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 14.

    Beas R, Cabanillas-Ramirez C, Izquierdo-Veraza D, et al. How good is online information for patients on the treatment for luminal gastrointestinal cancers? A comprehensive evaluation in English and Spanish. J Cancer Educ 2023;38:18011807.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 15.

    Owens B. How Nature readers are using ChatGPT. Nature 2023;615:20.

  • 16.

    OpenAI. ChatGPT; 2023. Accessed January 1, 2024. Available at: https://chat.openai.com/chat

    • PubMed
    • Export Citation
  • 17.

    Ayers JW, Poliak A, Dredze M, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med 2023;183:589596.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 18.

    Eppler MB, Ganjavi C, Knudsen JE, et al. Bridging the gap between urological research and patient understanding: the role of large language models in automated generation of layperson’s summaries. Urol Pract 2023;10:436443.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 19.

    Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159174.

  • 20.

    Fry E. A readability formula that saves time. J Read 1968;11:513–516; 575–578.

  • 21.

    Flesch R. A new readability yardstick. J Appl Psychol 1948;32:221233.

  • 22.

    Gunning R. The Technique of Clear Writing. McGraw-Hill; 1952.

  • 23.

    McLaughlin GH. SMOG grading—a new readability formula. J Read 1969;12:639646.

  • 24.

    Zhou S, Jeong H, Green PA. How consistent are the best-known readability equations in estimating the readability of design standards? IEEE Transactions Prof Comm 2017;60:97111.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 25.

    Rooney MK, Santiago G, Perni S, et al. Readability of patient education materials from high-impact medical journals: a 20-year analysis. J Patient Exp 2021;8:2374373521998847.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 26.

    Spadero DC. Assessing readability of patient information materials. Pediatr Nurs 1983;9:274278.

  • 27.

    Friedman DB, Hoffman-Goetz L, Arocha JF. Health literacy and the world wide web: comparing the readability of leading incident cancers on the internet. Med Inform Internet Med 2006;31:6787.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28.

    Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav 2006;33:352373.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29.

    Sheppard ED, Hyde Z, Florence MN, et al. Improving the readability of online foot and ankle patient education materials. Foot Ankle Int 2014;35:12821286.

  • 30.

    Hall JC. The readability of original articles in surgical journals. ANZ J Surg 2006;76:6870.

  • 31.

    Sokolova M, Lapalme G. A systematic analysis of performance measures for classification tasks. Inf Process Manage 2009;45:427437.

  • 32.

    Singhal A. Modern information retrieval: a brief overview. IEEE Data Eng Bull 2001;24:3543.

  • 33.

    Tan PN, Steinbach M, Kumar V. Introduction to Data Mining, 1st ed. Addison-Wesley; 2005

  • 34.

    Kadhim AI, Cheah YN, Ahamed NH, et al. Feature extraction for co-occurrence-based cosine similarity score of text documents. IEEE 2014;1:14.

  • 35.

    Charnock D, Shepperd S, Needham G, et al. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999;53:105111.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 36.

    Remer SL, Connelly TM, Clancy C, et al. The quality and content of hyperthermic intraperitoneal chemotherapy information available to patients: an evaluation of North American hyperthermic intraperitoneal chemotherapy websites. Surgery 2023;174:3035.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 37.

    Cocci A, Pezzoli M, Lo Re M, et al. Quality of information and appropriateness of ChatGPT outputs for urology patients. Prostate Cancer Prostatic Dis 2024;27:103108.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 38.

    Goodman M, Finnegan R, Mohadjer L, et al. Literacy, numeracy, and problem solving in technology-rich environments among U.S. adults: results from the Program for the International Assessment of Adult Competencies 2012. Accessed January 1, 2024. Available at: https://nces.ed.gov/pubs2014/2014008.pdf

    • PubMed
    • Export Citation
  • 39.

    Coskun B, Ocakoglu G, Yetemen M, et al. Can ChatGPT, an artificial intelligence language model, provide accurate and high-quality patient information on prostate cancer? Urology 2023;180:3558.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 40.

    Van Bulck L, Moons P. What if your patient switches from Dr. Google to Dr. ChatGPT? A vignette-based survey of the trustworthiness, value and danger of ChatGPT-generated responses to health questions. Eur J Cardiovasc Nurs 2024;23:95–98.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 41.

    Chen S, Kann BH, Foote MB, et al. Use of artificial intelligence chatbots for cancer treatment information. JAMA Oncol 2023;9:14591462.

  • 42.

    Wright JP, Edwards GC, Goggins K, et al. Association of health literacy with postoperative outcomes in patients undergoing major abdominal surgery. JAMA Surg 2018;153:137142.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 43.

    Davis TC, Arnold C, Berkel HJ, et al. Knowledge and attitude on screening mammography among low-literate, low-income women. Cancer 1996;78:19121920.

  • 44.

    McGreevey JD 3rd, Hanson CW 3rd, Koppel R. Clinical, legal, and ethical aspects of artificial intelligence-assisted conversational agents in health care. JAMA 2020;324:552553.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 45.

    Cacciamani GE, Chen A, Gill IS, et al. Artificial intelligence and urology: ethical considerations for urologists and patients. Nat Rev Urol 2024;21:50–59.

  • 46.

    US National Cancer Institute. NCI-designated cancer centers. Accessed January 1, 2024. Available at: https://www.cancer.gov/research/infrastructure/cancer-centers

    • PubMed
    • Export Citation

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1764 1764 819
PDF Downloads 1180 1180 540
EPUB Downloads 0 0 0