Artificial intelligence (AI) is increasingly being used to help doctors identify which patients are most likely to benefit from specific treatments. However, because AI systems are not infallible, physicians are expected to evaluate their recommendations critically and correct any errors. A new study suggests that this may be more difficult than expected, with doctors continuing to trust inaccurate AI advice even when patient outcomes indicate that the recommendations are wrong.
Researchers led by Aranzazu Vinas of the University of the Basque Country in Spain examined this issue in two online experiments involving 223 physicians. Participants were asked to imagine treating patients with a rare disease using an experimental therapy still under development. They were informed that an AI system had classified patients according to their likelihood of benefiting from the treatment and then decided which patients should receive it. After reviewing patient recovery data, they also rated the reliability of the AI system.
Unknown to the participants, the AI’s recommendations deliberately conflicted with the treatment’s actual effectiveness. In one experiment, the therapy provided the same moderate benefit to every patient regardless of the AI’s classifications. In the second, the treatment was completely ineffective for all patients. The recovery data, therefore, offered physicians clear evidence that the AI’s predictions were inaccurate.
Despite this feedback, most physicians continued to view the AI system as reliable. They generally failed to recognise that the patient outcomes contradicted the algorithm’s recommendations and did not adjust their assessments accordingly. In the second experiment, many participants also failed to realise that the treatment itself had no benefit, suggesting that reliance on the AI prevented them from concluding directly from the clinical evidence.
The findings highlight a potential challenge as AI becomes more deeply integrated into healthcare. Rather than functioning solely as decision-support tools, AI systems may unintentionally discourage doctors from learning from experience when real-world outcomes conflict with algorithmic advice. The researchers suggest that future studies should explore ways to strengthen critical thinking and improve physicians’ ability to detect AI errors, helping maximise the benefits of human-AI collaboration while reducing the risk of inappropriate clinical decisions.
“In both experiments, physicians mostly trusted the AI’s classifications and had trouble learning from the feedback,” said lead author Aranzazu Vinas. “Furthermore, in the second experiment, professionals did not notice that the treatment was completely ineffective.” Co-author Helena Matute added that the results show doctors, like other people, can struggle to learn from evidence that contradicts an algorithm’s suggestions. Co-author Fernando Blanco said that understanding the mistakes humans make when working with AI is essential for developing strategies that minimise these problems.
More information: Aranzazu Vinas et al, Doctors vs. Algorithms: Physicians, too, struggle to learn from evidence that contradicts AI suggestions, PLOS Digital Health. DOI: 10.1371/journal.pdig.0001490
Journal information: PLOS Digital Health Provided by PLOS
