Physicians May Struggle to Spot AI Errors, Even When Evidence Contradicts the Advice
Key Takeaways:
- In experiments involving decisions about hypothetical patients, physicians tended to trust incorrect advice labelled as artificial intelligence (AI) generated, even when they had the chance to notice that patient recovery data contradicted it.
- Across both experiments, physicians rated the AI system as reliable and did not draw on the recovery data to conclude that its recommendations were wrong; in the second experiment, they failed to notice that the treatment was entirely ineffective.
- The findings, published in the open-access journal PLOS Digital Health by Aranzazu Vinas of the University of the Basque Country and colleagues, point to real challenges for the widely held assumption that a human will reliably catch and correct an algorithm’s mistakes.
What the research examined
New research suggests that physicians may find it difficult to learn from experience when that experience runs counter to advice presented as coming from an AI system. In a series of experiments in which physicians made decisions about treating hypothetical patients, they tended to trust incorrect AI-labelled recommendations, even after being given the opportunity to notice that patient recovery data contradicted those recommendations.
The work was carried out by Aranzazu Vinas of the University of the Basque Country, Spain, together with colleagues, and is published in the open-access journal PLOS Digital Health.
Why AI classification matters in care
AI systems can help physicians categorise patients according to their differing care needs, for example by estimating whether a particular patient is more or less likely to benefit from a given treatment. Because these systems are not perfect, they are intended to be used as suggestions rather than instructions, with any potential errors caught and corrected by the physician using them.
This safeguard rests on an assumption that is easy to take for granted: that a human in the loop will notice when the algorithm is wrong. Prior research, however, has shown that people in general struggle to spot and correct mistakes made by AI. Vinas and colleagues set out to explore how far that difficulty extends to physicians in particular.
How the experiments worked
The researchers analysed data from 223 physicians who took part anonymously in online experiments. Participants were asked to imagine that they had the option to treat patients for a rare disease using a treatment that was not yet proven and still under development. They were told that an AI system had identified which patients were more, or less, likely to benefit from that treatment.
The physicians then chose which patients to treat. After being shown data on how those patients recovered, they rated their perceptions of how reliable the AI system was.
The design contained a deliberate mismatch. The actual effectiveness of the hypothetical treatment did not align with the AI’s recommendations. In the first experiment, the treatment was equally, and moderately, effective for all patients. In the second experiment, it was equally ineffective for everyone. In each case, the recovery data available to physicians should, in principle, have allowed them to see that the AI’s classification did not hold up.
What the physicians did
In both experiments, the physicians tended to rate the AI system as reliable, and they did not appear to use the patient recovery data to conclude that the AI’s recommendations were incorrect. In the second experiment, they did not realise that the treatment was entirely ineffective.
As lead author Aranzazu Vinas notes: “In both experiments, physicians mostly trusted the AI’s classifications and had trouble learning from the feedback. Furthermore, in the second experiment, professionals did not notice that the treatment was completely ineffective.”
Co-author Helena Matute adds: “People tend to say that there is always a human controlling the algorithm, but our experiments show that doctors (as well as anyone else) have problems in learning from the available evidence when it contradicts the suggestions of an algorithm.”
What it means for healthcare
Taken together, the results highlight potential challenges for incorporating AI-based classification into healthcare. If the human overseeing an algorithm cannot readily detect its errors, even when contradicting evidence is in front of them, then the reassurance that a clinician will always catch a mistake may be weaker than commonly assumed.
The authors suggest that future research could build on this study, for instance by developing and testing strategies and protocols designed to strengthen human critical thinking and the detection of AI errors. The aim would be to maximise the benefits of human-AI collaboration while minimising the potential for error.
Co-author Fernando Blanco summarises the wider purpose of this line of enquiry: “It is important to investigate the errors that humans (including doctors) make when working with algorithms, in order to learn how to minimize the problems that arise from them.”
Building the habit of questioning AI
While researchers work on formal protocols, individual clinicians can already sharpen how they interrogate AI output. Knowing when to trust a recommendation, and when to challenge it, is a clinical skill rather than a technical one, and it is one that structured training can help build. Our short course AI Essentials for GPs: Tools, Ethics and Everyday Applications introduces practical frameworks, including the SAFER Evaluation Framework, for spotting errors, fabrications, and outdated recommendations before they reach a patient. For clinicians who want a reliable method for the kind of critical checking this study suggests is all too easy to skip, it is a useful place to start.
Read More