AI Outperformed Emergency Doctors in Harvard Triage Study, Raising Questions About the Future of Clinical Decision-Making

Key Takeaways: 

• A Harvard-led study found that an AI reasoning model outperformed emergency physicians in diagnosing patients during hospital triage scenarios using text-based clinical information.
  
• Researchers said the findings represent a major advance in AI clinical reasoning, although they stressed that AI is not ready to replace human doctors.
  
• Experts warned that important concerns remain around accountability, bias, safety, and the risk of clinicians becoming overly reliant on AI systems.
  
  
  

AI shows strong performance in emergency medicine trial

From fictional emergency department heroes such as George Clooney in ER to Noah Wyle in The Pitt, emergency physicians have long been portrayed as the ultimate decision-makers in moments of medical crisis. However, a new Harvard study suggests artificial intelligence may increasingly play a major role in those same high-pressure situations.

Researchers from Harvard Medical School and Beth Israel Deaconess Medical Center found that advanced AI systems outperformed human doctors in emergency medicine triage scenarios, making more accurate diagnoses when presented with limited patient information during the critical early stages of hospital admission.

The findings, published in Science, were described by independent experts as representing “a genuine step forward” in AI clinical reasoning.

According to the study authors, large language models (LLMs) “have eclipsed most benchmarks of clinical reasoning”.

AI versus doctors in emergency room triage

One of the study’s central experiments examined 76 patients who presented to the emergency department of a Boston hospital.

Both the AI system and pairs of human physicians were given identical electronic health record information to assess. This included standard triage details such as:

• Vital signs
• Demographic information
• Brief nursing notes explaining why the patient attended hospital

Using only this text-based information, OpenAI’s o1 reasoning model identified the exact diagnosis or a very close diagnosis in 67% of cases.

By comparison, the human physicians achieved diagnostic accuracy rates of between 50% and 55%.

Researchers found the AI’s advantage was especially apparent in triage situations requiring rapid decision-making with minimal available information.

When additional clinical detail was provided, the AI’s diagnostic accuracy increased further to 82%. Human experts achieved accuracy rates between 70% and 79% under those circumstances, although researchers noted the difference was not statistically significant in that setting.

AI also performed better in treatment planning

The study also evaluated how AI performed in longer-term clinical planning tasks.

In this experiment, the AI system and a group of 46 doctors were asked to review five detailed clinical case studies and develop treatment strategies. These included decisions relating to:

• Antibiotic regimens
• Ongoing management plans
• End-of-life care processes

The AI significantly outperformed the doctors.

Researchers reported that the AI achieved a score of 89%, compared with 34% among physicians using conventional resources such as search engines.

AI detected a diagnosis human doctors missed

One example highlighted in the study involved a patient with a pulmonary embolism and worsening symptoms.

Human doctors believed the patient’s anticoagulant treatment was failing. However, the AI system identified something clinicians had overlooked – the patient had a history of lupus, which may have been responsible for inflammation in the lungs.

The AI’s interpretation was ultimately confirmed as correct.

Researchers say AI will reshape medicine – not replace doctors

Despite the strong performance shown by AI systems, researchers stressed that the technology is not ready to replace physicians.

The study only assessed AI systems using text-based patient information. It did not evaluate the AI’s ability to interpret non-verbal clinical signals that doctors routinely use during patient assessment, such as:

• Visible distress
• Facial appearance
• Behaviour
• Physical examination findings

As a result, researchers said the AI functioned more like a clinician reviewing paperwork and offering a second opinion.

“I don’t think our findings mean that AI replaces doctors,” said Arjun Manrai, one of the lead authors of the study who heads an AI lab at Harvard Medical School. “I think it does mean that we’re witnessing a really profound change in technology that will reshape medicine.”

Dr Adam Rodman, another lead author and physician at Beth Israel Deaconess Medical Center, described AI LLMs as among “the most impactful technologies in decades”.

He suggested healthcare may move towards what he described as a “triadic care model”.

“Over the next decade,” Rodman said, AI would not replace physicians but instead work alongside them in a new model involving “the doctor, the patient, and an artificial intelligence system”.

AI use in healthcare is already growing

The findings come amid rapidly increasing AI adoption within healthcare systems.

According to research published last month, nearly one in five physicians in the United States are already using AI to assist with diagnosis.

In the United Kingdom, a recent Royal College of Physicians survey found:

• 16% of doctors use AI daily
• A further 15% use AI weekly
• Clinical decision-making is among the most common applications

However, concerns around safety and accountability remain significant.

UK doctors surveyed identified AI errors and legal liability as among their biggest worries.

“There is not a formal framework right now for accountability,” said Rodman.

He also stressed the continuing importance of human clinicians in patient care.

“Patients ultimately want humans to guide them through life or death decisions [and] to guide them through challenging treatment decisions,” he said.

Experts warn against over-reliance on AI

Independent experts said the study highlights the rapidly improving capabilities of AI systems in medicine, but also demonstrates the need for caution.

Prof Ewen Harrison, co-director of the University of Edinburgh’s Centre for Medical Informatics, said the findings suggest AI systems are beginning to evolve beyond theoretical testing environments.

“These systems are no longer just passing medical exams or solving artificial test cases,” he said. “They are starting to look like useful second-opinion tools for clinicians, particularly when it is important to consider a wider range of possible diagnoses and avoid missing something important.”

However, Dr Wei Xing from the University of Sheffield warned that the study also raised concerns about how clinicians interact with AI recommendations.

He suggested some doctors may unconsciously defer to AI-generated answers instead of independently evaluating clinical information themselves.

“This tendency could grow more significant as AI becomes more routinely used in clinical settings,” he said.

Dr Xing also noted that the study provided limited information about where the AI may perform less effectively, including whether diagnostic accuracy differed among certain patient populations such as:

• Older adults
• Non-English speakers
• People with more complex communication needs

He cautioned against interpreting the findings as evidence that publicly available AI tools are ready for independent medical use.

“It does not demonstrate that AI is safe for routine clinical use, nor that the public should turn to freely available AI tools as a substitute for medical advice,” he said.

Source: The Guardian