Summary: ChatGPT outperformed human candidates in a mock Obstetrics and Gynecology (O&G) specialist examination, excelling in areas such as empathetic communication, information gathering, and clinical reasoning.
It scored an average of 77.2%, surpassing human candidates who scored an average of 73.7%. The study found that ChatGPT was able to rapidly generate accurate and contextually relevant answers to complex clinical questions. However, it struggled in scenarios that required open interpretation or involved multiple changes.
Key Facts:
- ChatGPT completed the examination stations in an average of two minutes and 54 seconds, well under the 10-minute limit.
- It was able to generate factually accurate and contextually relevant answers rapidly, a feat that would take an average person over a decade of clinical training.
- While ChatGPT outperformed in several areas, it struggled with complex scenarios involving multiple changes or requiring open interpretation.
Source: National University of Singapore
In a study to determine how the Chat Generative Pre-Trained Transformer or ChatGPT would fare in medical specialist examinations compared to human candidates without additional training, the Artificial Intelligence chatbot performed better than human candidates in a mock Obstetrics and Gynecology (O&G) specialist clinical examination, used to assess the eligibility of individuals to become O&G specialists.
The results from the mock clinical examination detailed that ChatGPT also achieved high scores in empathetic communication, information-gathering and clinical reasoning. The study is published in the American Journal of Obstetrics and Gynecology.
The tabulated results showed that ChatGPT attained a higher average score of 77.2%, compared to the human candidates who scored an average of 73.7%.
It was also recorded that ChatGPT took an average of two minutes and 54 seconds to complete each station, markedly ahead of the stipulated 10 minutes given. Even though ChatGPT completed the stations in record time, ChatGPT did not outperform all the individuals in each cohort.
To minimize bias, the responses of all three candidates were submitted to the examination panel, while concealing the true identity of ChatGPT.
In the study, the team selected seven stations that were in the objective structured clinical examinations (OSCEs) that had been run in the actual mock examinations in the two previous years, all similar in scope and difficulty, with no inclusion of visual interpretations to cater to the current limitations of ChatGPT present at the time of study.
Each station has multiple layers of evolving questions based on initial data presented and subsequent responses from the candidate. The OSCE is a criterion-based assessment, where each candidate is assessed on their clinical competencies by completing a series of circuit stations in a simulated environment.
Given 10 minutes to complete each station, the candidate is introduced to an unfamiliar clinical scenario, coupled with the necessary information which would aid them to make an informed clinical decision.
The candidate is expected to articulate a care plan, while demonstrating expertise such as communication, information gathering, application of clinical knowledge and patient safety within the time limit.
The stations were introduced in an identical format and in the same order to two human candidates—Candidates A and B, and ChatGPT, known as Candidate C.
The study team from the Department of O&G at the Yong Loo Lin School of Medicine (NUS Medicine), led by Associate Professor Mahesh Choolani, Head of the Department of O&G, also conducted an analysis of the answers and found that ChatGPT scored very well in the empathetic communication domain.
It was able to skillfully and rapidly generate factually accurate and contextually relevant answers to evolving clinical questions, based on unfamiliar data in the shortest time possible, a feat that would take an average intelligent person more than 10 years of clinical training to be able to understand the questions in this type of highly-complex examinations and answer them appropriately.
It is laudable that Generative AI, which is at present only in its infancy stage, has the prowess to consolidate and interpret huge chunks of general content quickly, and organize it into coherent and concise conversational-type responses, something that would not come naturally to non-native English speakers or candidates facing examination stress.
Despite best efforts to blind the examination panel, examiners were generally able to identify the responses from ChatGPT, but not in all cases.
From the mix of answers from human candidates and ChatGPT that were transcribed verbatim and assessed by 14 trained clinician examiners, it was also observed that even though English was used throughout, there was also an infusion of Singlish or words loaned from Malay, Tamil and Chinese dialects, that was included extensively by human candidates.
The intonation and unique vocabulary are very familiar and endearing to Singaporeans or long-term residents in Singapore. This method of communication would very well serve as a bridge to initiate closeness and build trust, while helping to ease nervousness in patients, compared to the more articulately-scripted answers of ChatGPT.
The lack of local ethnic knowledge is one of the major limitations of ChatGPT, on top of the lack of up-to-date medical references and data, which in turn causes hallucinations in ChatGPT, compelling it to churn out irrelevant or incorrect answers and conclusions at times.
Crucially, the study results also revealed that ChatGPT is less able to handle subjects that have multiple changes of scenarios, within the question itself, that require open interpretation.
The stations with multiple-changing scenarios would require additional training in context-specific medical knowledge in highly-specialized topics. This would be manageable for a highly-trained human candidate who has cultivated higher-level discernment and flexible reasoning needed to tackle ambiguities within these questions.
ChatGPT was found to outperform human candidates in several knowledge areas, including labor management, gynecologic oncology and postoperative care, topics or stations that largely focused on standard protocol-driven decision-making, but not in highly contextual situations.
“The arrival and increased use of ChatGPT has proven that it can be a viable resource in guiding medical education, possibly provide adjunct support for clinical care in real time, and even support the monitoring of medical treatment in patients.
“In an era where accurate knowledge and information is instantly accessible, and these capabilities could be embedded within appropriate context by Generative AI in the foreseeable future, the need for future generations of medical doctors to clearly demonstrate the value and importance of the human touch is now saliently obvious.
“As doctors and medical educators, we need to strongly emphasize and exemplify the use of soft skills, compassionate communication and knowledge application in medical training and clinical care,” said Associate Professor Mahesh Choolani.
About this AI research news
Author: Mahesh Choolani
Source: National University of Singapore
Contact: Mahesh Choolani – National University of Singapore
Image: The image is credited to Neuroscience News
Original Research: Open access.
“ChatGPT outscored human candidates in a virtual objective structured clinical examination in obstetrics and gynecology” by Sarah W. Li et al. American Journal of Obstetrics and Gynecology
Abstract
ChatGPT outscored human candidates in a virtual objective structured clinical examination in obstetrics and gynecology
Background
Natural language processing is a form of artificial intelligence that allows human users to interface with a machine without using complex codes. The ability of natural language processing systems, such as ChatGPT, to successfully engage with healthcare systems requiring fluid reasoning, specialist data interpretation, and empathetic communication in an unfamiliar and evolving environment is poorly studied. This study investigated whether the ChatGPT interface could engage with and complete a mock objective structured clinical examination simulating assessment for membership of the Royal College of Obstetricians and Gynaecologists.
Objective
This study aimed to determine whether ChatGPT, without additional training, would achieve a score at least equivalent to that achieved by human candidates who sat for virtual objective structured clinical examinations in Singapore.
Study Design
This study was conducted in 2 phases. In the first phase, a total of 7 structured discussion questions were selected from 2 historical cohorts (cohorts A and B) of objective structured clinical examination questions. ChatGPT was examined using these questions and responses recorded in a script. Of note, 2 human candidates (acting as anonymizers) were examined on the same questions using videoconferencing, and their responses were transcribed verbatim into written scripts. The 3 sets of response scripts were mixed, and each set was allocated to 1 of 3 human actors. In the second phase, actors were used to presenting these scripts to examiners in response to the same examination questions. These responses were blind scored by 14 qualified examiners. ChatGPT scores were unblinded and compared with historical human candidate performance scores.
Results
The average score given to ChatGPT by 14 examiners was 77.2%. The average historical human score (n=26 candidates) was 73.7 %. ChatGPT demonstrated sizable performance improvements over the average human candidate in several subject domains. The median time taken for ChatGPT to complete each station was 2.54 minutes, well before the 10 minutes allowed.
Conclusion
ChatGPT generated factually accurate and contextually relevant structured discussion answers to complex and evolving clinical questions based on unfamiliar settings within a very short period. ChatGPT outperformed human candidates in several knowledge areas. Not all examiners were able to discern between human and ChatGPT responses. Our data highlight the emergent ability of natural language processing models to demonstrate fluid reasoning in unfamiliar environments and successfully compete with human candidates that have undergone extensive specialist training.
