Summary: When it comes to answering people’s questions about cancer, especially regarding myths and misconceptions, ChatGPT is 97% accurate in providing the correct information. The AI is so accurate, test subjects were unaware whether the answers came from ChatGPT or the National Cancer Institute.
Source: Huntsman Cancer Institute
A study in the Journal of The National Cancer Institute Cancer Spectrum looked at chatbots and artificial intelligence (AI), as they become popular resources for cancer information. They found these resources give accurate information when asked about common cancer myths and misconceptions.
In the first study of its kind, Skyler Johnson, MD, physician-scientist at Huntsman Cancer Institute and assistant professor in the department of radiation oncology at the University of Utah (the U), evaluated the reliability and accuracy of ChatGPT’s cancer information.
Using the National Cancer Institute’s (NCI) common myths and misconceptions about cancer, Johnson and his team found that 97% of the answers were correct. However, this finding comes with some important caveats, including a concern amongst the team that some of the ChatGPT answers could be interpreted incorrectly.
“This could lead to some bad decisions by cancer patients. The team suggested caution when advising patients about whether they should use chatbots for information about cancer,” says Johnson.
The study found reviewers were blinded, meaning they didn’t know whether the answers came from the chatbot or the NCI. Though the answers were accurate, reviewers found ChatGPT’s language was indirect, vague, and in some cases, unclear.
“I recognize and understand how difficult it can feel for cancer patients and caregivers to access accurate information,” says Johnson.
“These sources need to be studied so that we can help cancer patients navigate the murky waters that exist in the online information environment as they try to seek answers about their diagnoses.”
Incorrect information can harm cancer patients. In a previous study by Johnson and his team published in the Journal of the National Cancer Institute, they found that misinformation was common on social media and had the potential to harm cancer patients.
The next steps are to evaluate how often patients are using chatbots to seek out information about cancer, what questions they are asking, and whether AI chatbots provide accurate answers to uncommon or unusual questions about cancer.
Funding: The study was supported by the National Institutes of Health/National Cancer Institute including P30 CA042014 and Huntsman Cancer Foundation.
About this ChatGPT AI and cancer research news
Author: Avery Shrader
Source: Huntsman Cancer Institute
Contact: Avery Shrader – Huntsman Cancer Institute
Image: The image is in the public domain
Original Research: Open access.
“Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information” by Skyler B Johnson et al. Cancer Spectrum
Abstract
Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information
Data about the quality of cancer information that chatbots and other artificial intelligence systems provide are limited.
Here, we evaluate the accuracy of cancer information on ChatGPT compared with the National Cancer Institute’s (NCI’s) answers by using the questions on the “Common Cancer Myths and Misconceptions” web page.
The NCI’s answers and ChatGPT answers to each question were blinded, and then evaluated for accuracy (accurate: yes vs no). Ratings were evaluated independently for each question, and then compared between the blinded NCI and ChatGPT answers.
Additionally, word count and Flesch-Kincaid readability grade level for each individual response were evaluated. Following expert review, the percentage of overall agreement for accuracy was 100% for NCI answers and 96.9% for ChatGPT outputs for questions 1 through 13 (ĸ = ‒0.03, standard error = 0.08).
There were few noticeable differences in the number of words or the readability of the answers from NCI or ChatGPT. Overall, the results suggest that ChatGPT provides accurate information about common cancer myths and misconceptions.
