Background
Use of secure messaging (SM)—private patient and physician communications conducted through a website or app-based portal—has drastically increased since its implementation in the late 1990s,1,2 around the same time email became widely used by the public. In recent years, accelerated by the COVID-19 pandemic, health care has become more inclusive of remote care delivery options such as remote medical visits and SM.3
Physicians find SM is frequently used as another form of remote health care, often comparable to phone or video appointments. Studies have also shown that this transition to “desktop medicine,” where physicians spend increasing time on electronic forms of health care while maintaining their direct patient care responsibilities, is a risk factor for physician burnout and increased physician stress.4–6
Despite these documented trends in the use of SM and their impact on physician well-being, there is scarce data on content of SM sent by patients with cancer. Herein, we report on utilization of SM and assess the content of SM in oncology practice, which may inform expected staffing requirements and projected cost savings from the implementation of more efficient strategies to manage the oncologist inbox.
Methods
Study Setting
This population-based retrospective cohort study examined all patient-initiated SMs sent to a Kaiser Permanente Northern California (KPNC) medical oncologist or fellow over a 1-year period (June 1, 2021–May 31, 2022) at 21 oncology practices. KPNC is an integrated health care system serving >4.7 million members, with 262 medical offices, 21 hospitals, and 21 community cancer centers across the region. KPNC members are highly representative of the local populations.7
This study was reviewed by the Institutional Review Board of the KPNC region and was exempt from ethics review or patient consent in accordance with US federal regulations. Funding for this study was provided by KPNC Graduate Medical Education, Kaiser Foundation Hospitals.
Exclusion Criteria
Patients were excluded if they were <18 years of age or did not have a documented gender. SM within the randomized sample were excluded if the initial message was initiated by the provider or was sent to a provider who was not a medical oncologist, or if the patient did not have a cancer diagnosis.
Study Variables
Patient-specific data were extracted from the electronic medical record (EMR), including patient demographics (self-reported gender, race/ethnicity, age) and number of SMs initiated (Table 1, Figure 1). The timestamp data of the initial patient message and oncologist response were recorded and categorized as inside or outside of business hours (Monday through Friday, 8 am–6 pm). The difference in timestamp data between patients’ and oncologists’ initial messages was calculated to report the time until oncologists’ response.
Number of Patients by Demographic Characteristics and Utilization Level
| Low (<3 SMs) n (%) |
Medium (3–9 SMs) n (%) |
High (≥10 SMs) n (%) |
Total n | |
|---|---|---|---|---|
| Gender | ||||
| Male | 5,432 (34.0) | 6,248 (39.1) | 4,282 (26.8) | 15,962 |
| Female | 8,963 (35.4) | 9,657 (38.2) | 6,690 (26.4) | 25,310 |
| Age | ||||
| <45 y | 1,654 (34.7) | 1,785 (37.4) | 1,334 (27.9) | 4,773 |
| 45–64 y | 5,189 (34.0) | 5,851 (38.3) | 4,223 (27.7) | 15,263 |
| 65–79 y | 5,709 (34.8) | 6,338 (38.7) | 4,348 (26.5) | 16,395 |
| ≥80 y | 1,843 (38.1) | 1,931 (39.9) | 1,067 (22.0) | 4,841 |
| Race/Ethnicity | ||||
| White | 8,652 (35.1) | 9,488 (38.5) | 6,536 (26.5) | 24,676 |
| Hispanic | 1,793 (36.4) | 1,898 (38.5) | 1,237 (25.1) | 4,928 |
| Black | 912 (35.2) | 1,024 (39.5) | 657 (25.3) | 2,593 |
| Asian/PI | 2,292 (32.4) | 2,753 (38.9) | 2,028 (28.7) | 7,073 |
| Other/Unknown/Multi | 746 (37.3) | 742 (37.1) | 514 (25.7) | 2,002 |
Abbreviations: Multi, multiracial; PI, Pacific Islander; SM, secure message.
Distribution of SMs by demographic group and utilization category.
Abbreviations: Multi, multiracial; PI, Pacific Islander; SM, secure message.
Citation: Journal of the National Comprehensive Cancer Network 2024; 10.6004/jnccn.2024.7055
Message Content Analysis
Of the total study sample of SMs, 500 were randomly selected for individualized review by a trained resident physician researcher (B. Anderson). Prior to SM review, the study team defined provider categories and restricted responses to align with each providers’ scope of practice (B. Anderson, L. Lyon, M. Lee, R. Liu). The categories for response included oncologist, registered nurse (RN), medical assistant (MA), primary care physician (PCP), or subspecialist. Within each response category, broad types of messages were established to filter patient SMs into each response category, as outlined in Table 2. Within each SM, the number of messages exchanged was counted and the content type of the initial SM was analyzed and sorted into a category for response. Messages that did not fit directly into one category, either because an SM contained multiple topics or because the content was unique, were discussed between the study team to assign the most appropriate response category based on consensus (B. Anderson, L. Lyon, M. Lee, D. Kumar, R. Liu).
Provider Categories for SM Response Delegation With Corresponding SM Content Examples
| Oncologist | RN | MA | PCP | Subspecialist |
|---|---|---|---|---|
|
|
|
|
|
Abbreviations: MA, medical assistant; PCP, primary care physician; RN, registered nurse; SM, secure message.
To compare the potential impact of different clinical personnel in managing SMs, we first extrapolated the percentages of SMs that could ideally be managed by each provider type (oncologist, MA, RN), as assigned by the study team, to the total volume of SMs. Projected hours were obtained by multiplying the reported physician response time to SM (2.3 min/SM)8 by the total volume of SMs in the study. National estimates of typical hourly compensation by provider types9,10 were multiplied by the projected hours needed of each provider type. Total estimated cost of management in each of the 2 potential staffing models was compared with projected potential health care system savings associated with implementing basic SM triaging strategies.
Analysis of each SM was performed to assess whether the medical care provided via SM was significant (ie, would typically require an appointment), either in-person or via telehealth (B. Anderson). These SMs handled complex medical counseling, cancer progression, or treatment changes. Messages where the utility of an appointment was unclear were discussed between the study team (B. Anderson, L. Lyon, M. Lee, D. Kumar, R. Liu). The total annual reduction in appointments due to the use of SMs was calculated by multiplying the proportion of SMs that provided significant medical care in our selected review by the total study sample. CMS billing codes 99211–99215 for established patient appointments were used to estimate the total cost of unbilled medical care provided via SM.10 An average copayment fee for subspecialty appointments nationally in 202211 was used to estimate the total savings to patients in potentially avoided copayment fees typically associated with appointments. Messages where the physicians’ initial response included a recommendation for the patient to obtain evaluation in the emergency department (ED) were marked as “emergency.” These SMs underwent detailed medical record review to evaluate follow-up and resolution of the stated medical concern (B. Anderson). Messages marked as emergency were not included as being typically better served by an appointment, and therefore the financial implications of the emergency SM were not considered in our calculations.
Statistical Analysis
Descriptive statistics (counts and percentages) were used to report the SM distributions for patient age at index date, gender, and self-reported race/ethnicity. Patients were grouped as low (quartile 1), medium (quartiles 2 and 3), or high (quartile 4) utilizers based on number of SMs initiated. The total number of SMs for each utilization level were reported as frequencies and percentages. We used chi-square tests to compare the demographic characteristics and the number of SMs sent by utilization level.
All analyses were performed using Microsoft Excel or SAS 9.4 (SAS Institute Inc), and report P<.05 as significant.
Results
Patient Characteristics
During the 1-year study period, we found 41,272 patients initiated 334,053 unique SMs to 132 medical oncologists and oncology fellows. In our selected review, the average number of messages exchanged between patient and provider was 2.6, with a mode of 1, and a maximum of 36. On average, each oncologist received 2,530 SMs during the 1-year period. The most SMs received by an individual oncologist was 6,832. Oncologists were found to respond to 22% of SMs outside of business hours.
Most patients self-identified as female (61.3%) (Table 1). Median age was 65 years (IQR, 54–74 years). Most patients were White (59.8%), followed by Asian or Pacific Islander (PI; 17.1%), Hispanic (11.9%), Black (6.3%), and other/unknown/multiracial (4.9%). The proportion of high utilizers (≥10 SMs) decreased with age, from 27.9% among patients aged <45 years to 22.0% among those aged ≥80 years. White and Asian/PI patients were more likely to be high utilizers (26.5% and 28.7%, respectively), whereas Black, Hispanic, and other/unknown/multiple race patients were more often low utilizers (35.2%, 36.4%, and 37.3%, respectively). Low utilizers (34.9% of patients) sent 6.0% of the total SM, medium utilizers (38.5%) sent 23.1%, and high utilizers (26.6%) sent 70.9%.
Message Characteristics
In the analysis of 500 random SMs, only 26.8% were categorized as needing review by an oncologist, whereas the remaining 73.2% may have been efficiently managed by another staff member based on our study criteria (Table 2). The remaining SMs could have been managed by an RN (38.2%), MA (28.4%), PCP (5.4%), or other subspecialty provider (1.2%) (Figures 2 and 3).
Appropriate levels of service designation by provider type.
Abbreviations: MA, medical assistant; PCP, primary care physician; RN, registered nurse.
Citation: Journal of the National Comprehensive Cancer Network 2024; 10.6004/jnccn.2024.7055
Percentage of total patient messages sent by the hour of the weekday that they were sent.
Citation: Journal of the National Comprehensive Cancer Network 2024; 10.6004/jnccn.2024.7055
In 24.4% of the SM reviewed, oncologists provided medical care via SM that typically would require an appointment based on our study criteria. When extrapolated to the size of our entire cohort, we estimated a total of $7.6 million in medical care was provided to patients through SM. Patients saved an estimated $3.6 million in avoided out-of-pocket copayment costs (Figure 4). This cost does not include the effort the providers took to triage the remaining 75.6% of SM.
Estimated value identified in work provided and costs avoided via secure messaging. Copays refers to the total savings to patients from avoided copayments typically associated with traditional (in-person or telehealth) visits. Forgone billing accounts for the total estimated loss of reimbursement for care designated to typically require a traditional visit.
Citation: Journal of the National Comprehensive Cancer Network 2024; 10.6004/jnccn.2024.7055
In 2.4% of SMs reviewed, the patient’s initial message prompted the oncologist to recommend urgent evaluation in the ED. Most of these patients (61.5%) were subsequently evaluated in the ED. The remaining 38.5% received outpatient evaluation, including laboratory and imaging tests or scheduling of an in-person evaluation. Examples of the emergencies included symptoms of low blood counts (eg, fatigue, petechiae), fever in patients with known neutropenia, debilitating pain, and shortness of breath. The average response time to these messages was 14.9 hours (range, 4 minutes to 85.8 hours). Most emergency SMs were sent outside of business hours (53.8%).
Across the 132 oncologists, we estimate that each spent an average of 97 hours over the year solely on responses to patient SMs. With the assistance of MAs and RNs triaging messages, each oncologist across the study would save approximately 71 hours per year. Accounting for hiring MA and RN staff, the system would still potentially save $1.1 million.
Discussion
The advent and widespread adoption of patient-to-physician SMs has fundamentally changed how patients interact with their health care team.12–14 Once limited by in-person visits, wait times, and copays, patients can now reach their doctor anywhere and anytime. In cancer and other disease states, patient interaction with SMs has been shown to have meaningful clinical benefits,15,16 with increased and sustained use.3,12,14 At the same time, SM has led to additional health care disparities,3 and remains largely unreimbursed. SM workload has repeatedly been associated with burnout,17–20 in some studies affecting the majority of the workforce,21,22 impacting work–life boundaries and increasing anxiety regarding unlimited inbox volume.4 Our study provides the first assessment of the content of patient-initiated SMs in oncology to inform more efficient and sustainable management strategies of the physician inbox.
We showed a high number of patients and all oncologists participating in SM, with each oncologist receiving on average 211 SMs monthly. A large study published in 2015 across an array of adult and pediatric specialties found their providers received only 14 SMs per month on average.12 SMs in oncology are more medically complex,23 and this drastic increase in their frequency increases the risk of burnout. One study found that the relationship between SM volume and burnout is likely directly proportional, with the physicians with the highest volume having almost 4 times the likelihood of experiencing burnout as their colleagues with the lowest volume.19 In a survey of 163 oncologists in 2018, EMR responsibilities caused stress in 67% of physicians, and 79% worked on EMRs outside of clinic time.24 Importantly, physicians who complete their SM work outside of business hours have been shown to have longer average stress durations when compared with physicians who complete their inbox work during business hours.25 Our finding that SMs are sometimes emergent and are being sent after hours and in ever-increasing volumes highlights a need for more flexibility with individual schedule management and dedicated resources for secure message management to reduce overall burnout. With almost 25% of medical oncologists leaving the workforce in the last decade,26 exacerbating a workforce already in crisis,27 solutions to the growing SM burden are urgently needed.
Examples of these solutions are emerging. Team-based inbox management has been cited as a viable solution, including delegation to MAs, pharmacists, and RNs, or use of templated Smartphases to reply to common administrative questions.5 We estimated that by implementing a team-based approach, where an MA surveys an inbox and forwards messages to an RN or physician, each oncologist in our health system could see an annual reduction of 71 hours in their workload. Machine learning models have also been used to determine whether patient messages involve medical decision-making, and could play a key role in SM management given the potential for cost savings, streamlined inbox management, and similar or improved outcomes compared with usual care.28,29 Chatbots were also shown to be noninferior to physicians and associated with better symptom management and self-care advice compared with a nurse-led model tested among patients with breast cancer.30,31 Our study discovered that although patients were asked not to use SM for acute care, 2.4% still used SM for urgent issues. Proactive monitoring of patient needs through cost-effective measures such as patient-reported outcomes,32 lay health workers,33 or artificial intelligence (AI) with a teams-based triaging system may effectively capture after-hours and emergency SM. A recent study using AI to generate draft responses to SM did not reduce time spent on SM management but significantly reduced self-reported provider burnout.34 Given their potential, AI platforms are likely to be integrated into clinical practice, although work needs to be done to manage misinformation and increase actionability.35
There are new billing models for SM to help support a sustainable workforce and reduce burnout. As of 2020, the CMS created new billing codes to reimburse inbox-related work. Some large academic institutions have begun charging patients for SM use. After beginning to charge for SM, one institution saw a roughly 15% decrease in monthly SM volume.36 Restructuring reimbursement to recognize this new form of work in a way that accounts for the potential variability in work needed to reply to a particular SM is vital to make this form of health care delivery sustainable. Given increased health disparities associated with the SM form of medical care, new health care costs to patients must be considered when creating policy changes.
Our study found disparities in utilization of SM by traditionally underserved populations, such as Black, Hispanic, and multiracial patients. Previous studies have shown that although underrepresented minorities have enrollment rates to the online portal similar to those of other demographic groups, utilization of the services offered within the portal, such as SM, are underutilized when compared with other demographic groups.3 One study found that most of the apparent disparities in use of SM noted within underserved communities could be explained by patients’ access to internet and preferences in care medium.37 Another found that patients with limited English proficiency to be significantly less likely to participate in SM.38 One unique solution may include leveraging the abilities of AI for direct end-to-end translation services within the SM portal itself, improving access to people of all backgrounds.39
High utilizers sent the vast majority of SM across all demographic categories in our study. A 2024 primary care study reported a similar pattern, with high-utilizers (≥3 SM/year) accounting for 77% of SM over a comparable 1-year period.40 That study characterized high utilizers as predominantly older, English-speaking, nonminority patients with more comorbidities and higher overall health care utilization. Interestingly, unlike the primary care study, our findings revealed that although there were minor differences in the frequency of high utilizers across racial/ethnic groups, the proportion of total messages sent by high utilizers was consistently around 70% across all racial/ethnic categories. This contrasts with the primary care setting, where both the frequency of high utilizers and their message volume varied more significantly by race/ethnicity. Additionally, high utilizers in our study of patients with cancer sent significantly more messages than those in primary care. The consistency in message volume from high utilizers across demographics in cancer care suggests that the nature of oncology treatment might override typical disparities in health care engagement, possibly due to the universally high stakes and complex care needs in this setting. This stark contrast in SM use between primary care and oncology settings aligns with reported burnout rates among oncologists related to this workload, underscoring the urgent need for systemic changes.
Our study had several limitations. Kaiser Permanente is an integrated health system with many unique features including the ability to use SMs to contact any provider on the patient’s care team free of charge, and no copay for most telehealth visits. The financial calculations used in this study do not reflect KPNC’s capitated system. In the study we examined a random sample of the total SMs and used these findings to estimate the financial burden of SM. In doing so, the total financial burden of SM may be more or less than our estimates. Our study did not directly measure the amount of time providers spent responding to SMs. We suspect the 2.3 minutes per SM response from a small 2011 Norwegian study of PCPs8 varies from the circumstances of our study but was reasonable given that oncology messages are likely to be more complex and time-consuming than primary care. More work is needed in this area to clarify the amount of time spent on physician responses to patient SM. Our study cites 2019 salary data for both RNs and MAs, which we suspect significantly underrepresents the current market value for these positions.
Conclusions
Our study reveals the significant impact of SM in oncology, with oncologists receiving an average of 211 messages monthly, far exceeding previous estimates across specialties. Although only 26.8% of SMs required direct oncologist review, suggesting potential for more efficient management, we also identified disparities in utilization among underserved populations. Interestingly, in the oncology setting, high utilizers across all demographic groups consistently sent approximately 70% of the total SMs, indicating that cancer care may override typical health care engagement disparities. These findings underscore the urgent need for systemic changes in SM management to address growing workload and burnout risk among oncologists, while also highlighting opportunities for cost savings and improved efficiency through team-based inbox management and AI technologies. Future research should focus on evaluating various SM management strategies, investigating factors contributing to utilization patterns in oncology, and developing interventions to address disparities. By addressing these challenges through innovative management strategies and equitable access initiatives, we can ensure the sustainability of this important communication tool while maintaining high-quality patient care and provider well-being.
References
- 1.↑
Huang M, Fan J, Prigge J, et al. Characterizing patient-clinician communication in secure medical messages: retrospective study. J Med Internet Res 2022;24:e17273.
- 2.↑
Crotty BH, Tamrat Y, Mostaghimi A, et al. Patient-to-physician messaging: volume nearly tripled as more patients joined system, but per capita rate plateaued. Health Aff (Millwood) 2014;33:1817–1822.
- 3.↑
Neeman E, Lyon L, Sun H, et al. Future of teleoncology: trends and disparities in telehealth and secure message utilization in the COVID-19 era. JCO Clin Cancer Inform 2022;6:e2100160.
- 4.↑
Lieu TA, Altschuler A, Weiner JZ, et al. Primary care physicians’ experiences with and strategies for managing electronic messages. JAMA Netw Open 2019;2:e1918287.
- 5.↑
Arndt BG, Beasley JW, Watkinson MD, et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017;15:419–426.
- 6.↑
Adler-Milstein J, Zhao W, Willard-Grace R, et al. Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc 2020;27:531–538.
- 7.↑
Davis AC, Voelkel JL, Remmers CL, et al. Comparing Kaiser Permanente members to the general population: implications for generalizability of research. Perm J 2023;27:87–98.
- 8.↑
Kummervold PE, Johnsen JA. Physician response time when communicating with patients over the Internet. J Med Internet Res 2011;13:e79.
- 9.↑
Bourbeau B, Harter D, Towle E. Results from the ASCO 2019 survey of oncology practice operations. JCO Oncol Pract 2020;16:253–262.
- 10.↑
Shanafelt TD, Gradishar WJ, Kosty M, et al. Burnout and career satisfaction among US oncologists. J Clin Oncol 2014;32:678–686.
- 11.↑
Claxton G, Rae M, Damico A, et al. Health benefits in 2022: premiums remain steady, many employers report limited provider networks for behavioral health: study examines employer-sponsored health benefits in 2022. Health Aff (Millwood) 2022;41:1670–1680.
- 12.↑
Cronin RM, Davis SE, Shenson JA, et al. Growth of secure messaging through a patient portal as a form of outpatient interaction across clinical specialties. Appl Clin Inform 2015;6:288–304.
- 13.↑
Shenson JA, Cronin RM, Davis SE, et al. Rapid growth in surgeons’ use of secure messaging in a patient portal. Surg Endosc 2016;30:1432–1440.
- 14.↑
North F, Luhman KE, Mallmann EA, et al. A retrospective analysis of provider-to-patient secure messages: how much are they increasing, who is doing the work, and is the work happening after hours? JMIR Med Inform 2020;8:e16521.
- 15.↑
Heisey-Grove DM, McClelland LE, Rathert C, et al. Associations between patient health outcomes and secure message content exchanged between patients and clinicians: retrospective cohort study. J Med Internet Res 2020;22:e19477.
- 16.↑
Coquet J, Blayney DW, Brooks JD, Hernandez-Boussard T. Association between patient-initiated emails and overall 2-year survival in cancer patients undergoing chemotherapy: evidence from the real-world setting. Cancer Med 2020;9:8552–8561.
- 17.↑
Peccoralo LA, Kaplan CA, Pietrzak RH, et al. The impact of time spent on the electronic health record after work and of clerical work on burnout among clinical faculty. J Am Med Inform Assoc 2021;28:938–947.
- 18.↑
Li C, Parpia C, Sriharan A, Keefe DT. Electronic medical record-related burnout in healthcare providers: a scoping review of outcomes and interventions. BMJ Open 2022;12:e060865.
- 19.↑
Hilliard RW, Haskell J, Gardner RL. Are specific elements of electronic health record use associated with clinician burnout more than others? J Am Med Inform Assoc 2020;27:1401–1410.
- 20.↑
Tai-Seale M, Dillon EC, Yang Y, et al. Physicians’ well-being linked to in-basket messages generated by algorithms in electronic health records. Health Aff (Millwood) 2019;38:1073–1078.
- 21.↑
Tran B, Lenhart A, Ross R, Dorr DA. Burnout and EHR use among academic primary care physicians with varied clinical workloads. AMIA Jt Summits Transl Sci Proc 2019;2019:136–144.
- 22.↑
Shanafelt TD, West CP, Dyrbye LN, et al. Changes in burnout and satisfaction with work-life integration in physicians during the first 2 years of the COVID-19 pandemic. Mayo Clin Proc 2022;97:2248–2258.
- 23.↑
Grant RW, McCloskey J, Hatfield M, et al. Use of latent class analysis and k-means clustering to identify complex patient profiles. JAMA Netw Open 2020;3:e2029068.
- 24.↑
Gajra A, Bapat B, Jeune-Smith Y, et al. Frequency and causes of burnout in US community oncologists in the era of electronic health records. JCO Oncol Pract 2020;16:e357–365.
- 25.↑
Akbar F, Mark G, Prausnitz S, et al. Physician stress during electronic health record inbox work: in situ measurement with wearable sensors. JMIR Med Inform 2021;9:e24014.
- 26.↑
Tuia J, Haslam A, Prasad V. Profile of the oncology physician workforce and the characteristics of attrition. JCO Oncol Pract 2023;19:465–472.
- 27.↑
Duma N, Velazquez AI, Franco I, et al. Dónde están? Latinx/Hispanic representation in the oncology workforce: present and future. JCO Oncol Pract 2022;18:388–395.
- 28.↑
Xu L, Sanders L, Li K, Chow JCL. Chatbot for health care and oncology applications using artificial intelligence and machine learning: systematic review. JMIR Cancer 2021;7:e27850.
- 29.↑
Wang A, Qian Z, Briggs L, et al. The use of chatbots in oncological care: a narrative review. Int J Gen Med 2023;16:1591–1602.
- 30.↑
Tawfik E, Ghallab E, Moustafa A. A nurse versus a chatbot – the effect of an empowerment program on chemotherapy-related side effects and the self-care behaviors of women living with breast cancer: a randomized controlled trial. BMC Nurs 2023;22:102.
- 31.↑
Bibault JE, Chaix B, Guillemassé A, et al. A chatbot versus physicians to provide information for patients with breast cancer: blind, randomized controlled noninferiority trial. J Med Internet Res 2019;21:e15787.
- 32.↑
Basch E, Deal AM, Dueck AC, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 2017;318:197–198.
- 33.↑
Patel MI, Agrawal M, Blayney DW, et al. Engagement of patients with advanced cancer (EPAC) randomized clinical trial: long-term effects on survival and healthcare use. J Clin Oncol 2023;19(Suppl):Abstract 76.
- 34.↑
Garcia P, Ma SP, Shah S, et al. Artificial intelligence–generated draft replies to patient inbox messages. JAMA Netw Open 2024;7:e243201.
- 35.↑
Pan A, Musheyev D, Bockelman D, et al. Assessment of artificial intelligence chatbot responses to top searched queries about cancer. JAMA Oncol 2023;9:1437–1440.
- 36.↑
Ayers JW, Poliak A, Dredze M, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med 2023;183:589–596.
- 37.↑
Graetz I, Gordon N, Fung V, et al. The digital divide and patient portals: internet access explained differences in patient portal use for secure messaging by age, race, and income. Med Care 2016;54:772–779.
- 38.↑
Rodriguez JA, Saadi A, Schwamm LH, et al. Disparities in telehealth use among California patients with limited English proficiency: study examines disparities in telehealth use among California patients with limited English proficiency. Health Aff (Millwood) 2021;40:487–495.
- 39.↑
Mohamed YA, Khanan A, Bashir M, et al. The impact of artificial intelligence on language translation: a review. IEEE Access 2024;12:25553–25579.
- 40.↑
Hansen MA, Hirth J, Zoorob R, Langabeer J. Demographics and clinical features associated with rates of electronic message utilization in the primary care setting. Int J Med Inform 2024;183:105339.
