Doctors kept trusting the algorithm even when patients weren't getting better
As artificial intelligence becomes woven into the fabric of clinical medicine, a study of 223 physicians in Spain reveals a quiet but consequential vulnerability: when doctors collaborate with AI systems, they tend to trust the machine's judgment even when patient outcomes plainly contradict it. The research exposes not a failure of intelligence, but a failure of skepticism — the ancient human tendency to defer to perceived authority, now dressed in algorithmic form. What was designed as a safeguard, the physician's critical eye, may be softened by the very tools it was meant to oversee.
- Spanish researchers designed experiments where AI treatment recommendations were demonstrably wrong — yet 223 physicians kept rating those systems as reliable.
- In one scenario the treatment worked for everyone regardless of the AI's predictions; in another it worked for no one — and doctors failed to notice either contradiction.
- The cognitive trap is well-documented: humans struggle to detect and correct machine errors, especially when the machine is framed as an expert authority.
- The intended safeguard — a physician catching algorithmic mistakes — may be functionally broken if doctors interpret patient data through the AI's lens rather than using it to challenge the AI.
- Researchers are now calling for new clinical protocols designed to actively rebuild physician skepticism and teach doctors to treat patient outcomes as a genuine audit of AI performance.
A research team in Spain posed a disquieting question: when a doctor sees an AI recommending a treatment, how rigorously do they actually test that recommendation against reality? Their answer, drawn from experiments with 223 physicians, offers little comfort.
The setup placed doctors in hypothetical scenarios involving a rare disease and an experimental drug. An AI system had supposedly analyzed patient data to identify who would benefit from treatment. Physicians made their decisions, watched outcomes unfold, and then rated how much they trusted the algorithm. The catch: the AI was wrong in both experiments. In the first, the drug worked equally for everyone, making the AI's predictions meaningless. In the second, the drug worked for no one at all. Yet physicians continued to rate the system as reliable, and most failed to recognize that the treatment was entirely useless.
Lead researcher Aranzazu Vinas noted that physicians consistently accepted the AI's classifications and struggled to learn from feedback that contradicted them. Her co-author Helena Matute identified the broader illusion at work: people tend to assume some human expert is always watching the algorithm, catching its errors. These experiments suggest that assumption is dangerously optimistic.
The stakes are real. AI tools are entering clinical practice as aids to physician judgment, not replacements for it — the doctor is supposed to be the last line of defense against algorithmic error. But if that defense dissolves in the presence of a confident-sounding machine, patients may receive ineffective treatments while the evidence that should have raised alarms gets quietly absorbed into trust. The researchers argue that understanding this failure is the necessary first step toward correcting it, through protocols that restore genuine skepticism and teach physicians to treat patient outcomes as a true test of whether their AI partner is actually getting it right.
A team of researchers from Spain set out to answer a troubling question: when a doctor sees that an artificial intelligence system is recommending a treatment, how carefully do they actually check whether that recommendation is sound? The answer, based on experiments with 223 physicians, is not reassuring.
The setup was straightforward. Physicians were placed in a hypothetical scenario where they could choose to treat patients suffering from a rare disease using an experimental treatment. An AI system had supposedly analyzed the patient data and identified which patients would benefit most from the drug and which would not. The doctors then made their treatment decisions, observed what happened to the patients, and rated how much they trusted the AI's judgment.
Here is where the experiment's design became crucial: the AI recommendations were wrong. In the first test, the treatment worked equally well for everyone, regardless of what the algorithm had predicted. In the second test, the treatment did not work at all—it was completely ineffective across the board. Yet in both cases, the physicians continued to rate the AI system as reliable. They did not use the patient outcome data to conclude that the algorithm had steered them wrong. In the second experiment, most doctors failed to recognize that the treatment was entirely useless.
This finding points to a cognitive trap that researchers have documented in other contexts: people struggle to notice and correct errors made by machines, especially when those machines are presented as authoritative. The physicians in this study seemed to interpret the patient recovery data through the lens of the AI's recommendations rather than using the data to test whether the AI was trustworthy. When the evidence contradicted the algorithm, they did not revise their assessment. They kept trusting it.
Aranzazu Vinas, the lead researcher, noted the consistency of the problem across both experiments: physicians mostly accepted the AI's classifications and had difficulty learning from feedback that contradicted the algorithm's suggestions. Her co-author Helena Matute framed the issue more broadly: people often assume that a human being is always watching over the algorithm, catching its mistakes. But the experiments suggest that doctors, like anyone else, struggle to learn from available evidence when that evidence runs counter to what an algorithm has told them.
The implications for healthcare are significant. AI systems are increasingly being integrated into clinical practice to help categorize patients and suggest treatment approaches. These tools are not meant to replace physician judgment—they are supposed to be aids, with doctors catching and correcting any errors. But if physicians have difficulty noticing when an algorithm is wrong, even when the evidence is plain, then the human safeguard may not function as intended. A doctor might implement an ineffective treatment because the AI recommended it, and the patient data that should have triggered skepticism instead gets filtered through trust in the machine.
The researchers emphasize that understanding these human errors is the first step toward preventing them. Future work will need to develop protocols and strategies that strengthen physicians' critical thinking when they work alongside algorithms—ways to help doctors remain genuinely skeptical, to actively question AI recommendations, and to use patient outcomes as a real test of whether the system is performing as claimed. Without such interventions, the collaboration between human and machine in medicine risks becoming a one-way street, with the algorithm leading and the physician following, even when the path is wrong.
Citas Notables
In both experiments, physicians mostly trusted the AI's classifications and had trouble learning from the feedback. Furthermore, in the second experiment, professionals did not notice that the treatment was completely ineffective.— Aranzazu Vinas, lead researcher
People tend to say that there is always a human controlling the algorithm, but our experiments show that doctors have problems in learning from the available evidence when it contradicts the suggestions of an algorithm.— Helena Matute, co-author
La Conversación del Hearth Otra perspectiva de la historia
Why would doctors trust an AI recommendation if the patient data showed it was failing?
Because the data itself becomes hard to interpret once you've been told what to expect. The doctors weren't ignoring the outcomes—they were seeing them through the lens of the algorithm's judgment. It's a kind of cognitive capture.
But these are physicians. They're trained to read evidence and change their minds.
They are, and that training matters. But it turns out that training doesn't automatically protect you from this particular trap. The algorithm carries a kind of authority that makes contradictory evidence feel ambiguous rather than clear.
What would actually help? How do you break that pattern?
That's what the researchers are asking now. You'd need explicit protocols—ways to force doctors to actively test the algorithm's claims rather than passively accept them. Maybe structured review processes, or training that makes the bias visible.
Is this just about doctors, or is it a broader human problem?
Broader. The researchers found that people in general struggle with this. But it matters most in medicine because the stakes are a patient's health. A wrong algorithm recommendation in a spreadsheet is annoying. In a hospital, it can be dangerous.
So the AI isn't the problem here—it's how humans interact with it.
Exactly. The AI did what it was designed to do. The problem is that humans have a blind spot when it comes to trusting machines, and we haven't yet built the safeguards to compensate for that.