Long Island University researchers have uncovered the limitations of OpenAI’s ChatGPT when it comes to addressing medical questions. A recent study found that the AI chatbot provided accurate responses to only a fraction of medication-related queries, raising concerns about the potential risks of relying on such AI for medical advice.
Accuracy of ChatGPT’s responses
In a study conducted by researchers at Long Island University, 39 medication-related questions were posed to ChatGPT, representing real inquiries from the university’s College of Pharmacy drug information service. The responses generated by ChatGPT were then compared to answers provided by trained pharmacists. The results were far from reassuring, with ChatGPT delivering accurate responses to only around 25% of the questions. In the remaining cases, the answers were either incomplete, inaccurate or simply did not address the questions adequately.
Concerns for public health
Given the rapid rise in ChatGPT’s popularity following its release in November 2022, concerns have emerged regarding its use for medical inquiries. Sara Grossman, an associate professor of pharmacy practice at Long Island University and one of the study’s authors, expressed concerns that students, pharmacists, and the general public might turn to ChatGPT for health and medication-related information. Unfortunately, the study revealed that many of the AI’s responses were not only inaccurate but potentially dangerous.
For instance, in one question about potential drug interactions between the Covid-19 antiviral medication Paxlovid and the blood-pressure-lowering medication Verapamil, ChatGPT provided an inaccurate response, suggesting no adverse effects. In reality, combining these medications can lead to a significant drop in blood pressure, causing dizziness and fainting. Such misleading information could put individuals at risk.
Questionable scientific references
The researchers also found that when they asked ChatGPT for scientific references to support its responses, the AI could provide references for only a limited number of questions. More troublingly, the references provided were often fabricated, appearing legitimate at first glance. These fabricated citations listed URLs, proper formatting, and even referenced reputed scientific journals. However, upon investigation, it became clear that ChatGPT had created these references, casting doubt on the reliability of its information.
In one instance, ChatGPT was asked to provide a dose conversion ratio for the muscle spasm medication baclofen when transitioning from spinal injections to oral doses. The AI provided a conversion rate along with references to guidance from medical organizations. However, neither of these organizations had established an official dose conversion ratio, rendering ChatGPT’s information unreliable. Furthermore, the AI made a critical error in the calculation, resulting in a dose recommendation that was 1,000 times lower than required. Such an error could lead to severe withdrawal symptoms in patients if followed by a healthcare professional.
The dangers of misinformation
The Long Island University study is not the first to raise concerns about ChatGPT’s ability to generate fictional citations when responding to medical queries. Previous research has also highlighted this issue, demonstrating that the AI can create deceptive forgeries of scientific references, even attributing them to real authors with prior publications in scientific journals.
Sara Grossman, who had limited experience with ChatGPT before the study, expressed surprise at the AI’s ability to confidently synthesize information instantaneously. The AI’s responses were presented in a professional and sophisticated manner, potentially giving users a false sense of confidence in the accuracy of the information provided.
OpenAI’s response
In response to these findings, OpenAI, the organization behind ChatGPT, advised users not to rely on the AI’s responses as a substitute for professional medical advice or treatment. OpenAI’s usage policies explicitly state that ChatGPT’s models are not fine-tuned to provide medical information and should never be used for diagnostic or treatment services for serious medical conditions.
Seeking reliable medical information
While the convenience of ChatGPT and similar AI tools is undeniable, consumers are urged to exercise caution when seeking medical information online. Sara Grossman recommended using governmental websites like the National Institutes of Health’s MedlinePlus page for reputable information. However, she emphasized that online answers should not replace the guidance of healthcare professionals, as each patient’s case is unique and requires personalized care.
The study conducted by Long Island University serves as a stark reminder of the limitations of AI, particularly when it comes to critical matters like healthcare. While AI can provide valuable information, it should never be a replacement for the expertise and guidance of qualified healthcare professionals. The consequences of relying on inaccurate or fabricated information can be severe, underscoring the need for vigilance and discernment in the digital age.
