2
Putting ChatGPT’s Medical Advice to the (Turing) Test
Abstract
Importance: Chatbots could play a role in answering patient questions, but patients’ ability to
distinguish between provider and chatbot responses, and patients’ trust in chatbots’ functions are
not well established.
Objective: To assess the feasibility of using ChatGPT or a similar AI-based chatbot for patient-
provider communication.
Design: Survey in January 2023
Setting: Survey
Participants: A US representative sample of 430 study participants aged 18 and above was
recruited on Prolific, a crowdsourcing platform for academic studies. 426 participants filled out
the full survey. After removing participants who spent less than 3 minutes on the survey, 392
respondents remained. 53.2% of respondents analyzed were women; their average age was 47.1.
Exposure(s): Ten representative non-administrative patient-provider interactions were extracted
from the EHR. Patients’ questions were placed in ChatGPT with a request for the chatbot to
respond using approximately the same word count as the human provider’s response. In the
survey, each patient’s question was followed by a provider- or ChatGPT-generated response.
Participants were informed that five responses were provider-generated and five were chatbot-
generated. Participants were asked, and incentivized financially, to correctly identify the
response source. Participants were also asked about their trust in chatbots’ functions in patient-
provider communication, using a Likert scale of 1-5.
