A recent study published in JAMA Network Open reports that ChatGPT, an artificial intelligence (AI) program, has the potential to become a reliable source of accurate medical information. The study found that ChatGPT provided mostly correct responses to over 280 medical questions across different specialties. Although not perfect or completely reliable, the program was considered relatively accurate and provided comprehensive information.
The accuracy of ChatGPT’s responses improved when a second AI program was used to review the initial response. This suggests that combining multiple AI programs could further enhance the accuracy of medical information provided. The researchers aimed to test the accuracy and completeness of information provided by ChatGPT across various medical disciplines, recognizing that patients and even physicians might rely on AI programs for searching medical issues.
To assess ChatGPT, 33 physicians from 17 specialties developed 284 questions, categorized into easy, medium, and difficult levels. The program’s average accuracy score was 4.8 out of 6, indicating that it was more correct than incorrect. The program performed better on questions related to common conditions and specific medical issues like melanoma and immunotherapy.
However, ChatGPT did struggle with certain questions. For example, it provided an incomplete response to a question about oral antibiotics for MRSA infections. The researchers suggested that the phrasing of the question might have contributed to the program’s difficulties in grasping the correct answer. They highlighted the importance of crafting questions to optimize the AI’s response.
The study also observed a learning effect in ChatGPT, with initial poor answers improving over time. The AI program quickly grew smarter, suggesting potential for continuous improvement. However, the researchers cautioned against relying solely on AI for medical advice and recommended using it in conjunction with other trusted resources.
Both the researchers and an expert from the American Academy of Family Physicians agreed that AI will not replace physicians but can serve as a valuable tool for doctors and patients. ChatGPT could potentially provide additional information for difficult diagnoses or act as a “health coach” for patients. The expert mentioned the potential for cost-cutting health systems to use AI as a front-line resource, but emphasized the importance of maintaining continuity and coordination of care.
In conclusion, the study shows that while ChatGPT has the potential to provide accurate and comprehensive medical information, it is not yet fully reliable. With further development and optimization, AI chatbots specifically designed for medical use could achieve higher reliability in the future.