AI in Healthcare: Promise, Pitfalls, and the Risk of Misguided Medical Advice
Key Takeaways:
- People using AI for health advice often struggle to interpret and communicate symptoms effectively, leading to incorrect conclusions in many cases.
- Even when AI identifies a condition correctly, it may fail to recommend appropriate urgency, particularly in time-sensitive or complex scenarios.
- Clinicians see value in AI as a supportive tool, but stress that it should complement, not replace, professional medical care.
AI becomes a common source of health information
As technology companies continue to develop platforms tailored for healthcare consultation, artificial intelligence is becoming an increasingly influential part of how people make decisions about their health. According to OpenAI, more than 40 million people use ChatGPT each day to seek health-related information.
However, emerging research suggests that while these tools offer unprecedented access to medical knowledge, they may also mislead users in certain contexts.
Challenges in how people use AI for medical queries
One of the central issues identified by researchers is not only the capability of AI systems, but how individuals interact with them. Many people lack the knowledge required to communicate symptoms accurately or comprehensively.
A recent study published in Nature Medicine attempted to replicate real-world use of AI chatbots. Participants were given medical scenarios and asked to consult AI tools. The results highlighted notable limitations:
- Participants correctly identified the condition only about one-third of the time.
- Just 43% made the correct decision regarding next steps, such as whether to seek emergency care or remain at home.
“People don’t know what they are supposed to be telling the model,” says Andrew Bean, who studies AI systems at Oxford University and was one of the authors on this study.
Bean explains that effective use of AI often depends on precise wording. “Doctors are trained to ask you questions about symptoms you might not have realised you should have mentioned,” says Bean.
Small differences in input can lead to dangerous outcomes
The study demonstrated how subtle differences in language can significantly alter the advice provided by AI systems.
In one example, two individuals described the same clinical scenario slightly differently. One described experiencing “the worst headache I’ve ever had” and was advised to go to the emergency room immediately. The other, who did not include that specific phrasing, was advised to take aspirin and remain at home.
“Turns out this was actually a life-threatening condition,” says Bean.
This highlights a critical limitation: AI systems rely heavily on the information they are given, and may not prompt for missing but clinically important details in the way a trained clinician would.
When AI gets the diagnosis right but the advice wrong
Even when AI tools successfully identify a medical condition, they may still provide inappropriate guidance regarding urgency or next steps.
In a separate study, researchers evaluated how AI systems responded to a range of medical scenarios. They found that in 52% of emergency cases, the tools “under-triaged” – treating conditions as less serious than they actually were.
In one case, the AI failed to direct a hypothetical patient experiencing diabetic ketoacidosis and impending respiratory failure – both life-threatening conditions – to seek emergency care.
“When there was a textbook medical emergency, ChatGPT got it right,” said Girish Nadkarni, a doctor and AI researcher at Mount Sinai who is an author on the study. However, he noted that performance declined in more complex situations, particularly where timing was critical. In such cases, the system often misjudged how urgently care was required.
An OpenAI spokesperson responded by stating that the study did not reflect typical real-world usage and that it evaluated an older version of ChatGPT, which the company says has since been improved to address some of these concerns.
The role of AI in supporting patient understanding
Despite these concerns, many clinicians believe that AI tools can still play a constructive role in healthcare, particularly in improving patient understanding and engagement.
“I encourage patients to use these tools,” says Robert Wachter, a doctor at UC San Francisco and author of the recently published book, A Giant Leap: How AI Is Transforming Health Care and What That Means for Our Future.
Wachter points out that barriers to accessing healthcare – including cost and availability – mean that AI can sometimes provide a useful alternative source of information. “The advice you get from the tools is substantially better than nothing and better than what you would get from your second cousin,” says Wachter.
However, he emphasises that AI should never be viewed as a substitute for professional medical care.
Enhancing, not replacing, the doctor–patient relationship
Experts suggest that AI is most valuable when used alongside traditional healthcare, rather than in place of it.
Adam Rodman, a hospitalist and researcher at Harvard Medical School, advises against using AI tools to assess emergency situations. Instead, he sees their greatest benefit in preparing for or reflecting on medical consultations.
“A good time to use a large language model is when you’re about to go see a doctor – or after you see your doctor,” says Rodman.
He explains that AI can help people better understand their condition, ask more informed questions, and make more effective use of time during appointments. This can support a more collaborative relationship between patients and clinicians.
“There are no downsides to better understanding your health,” says Rodman.
The future of AI in healthcare
Healthcare professionals broadly agree that AI is now firmly embedded within modern medicine and will continue to evolve alongside clinical practice.
“ My hope is that you might see AI as an extension of a human relationship,” says Rodman. He envisions a future in which both clinicians and patients work with AI to improve communication and navigate healthcare systems more efficiently.
However, he also raises concerns about potential unintended consequences. One particular risk is the possibility that people may receive serious or distressing diagnoses – such as cancer – directly from an AI system, rather than from a clinician.
Research suggests that when healthcare becomes more transactional or resembles a marketplace, trust in clinicians may decline.
”What I hope is that this technology can be used in a way that enhances humanity in medicine,” says Rodman “and not in a way that cuts out the doctor-patient relationship.”
Conclusion
Artificial intelligence is rapidly transforming access to health information, offering both opportunities and risks. While these tools can enhance understanding and support more informed decision-making, their limitations – particularly in how they interpret incomplete or imprecise input – mean they must be used with caution.
Ultimately, AI has the potential to strengthen healthcare delivery, but only if it is integrated in a way that supports, rather than replaces, the human relationships at the heart of medicine.
