AI can ‘diagnose emergency cases more accurately than doctors’

The study, published in the journal Science last week, by a team of physicians and computer scientists at Harvard Medical School and Beth Israel Deaconess Medical Center.

In a real-world experiment with 76 patients admitted to the emergency department, the research team compared the diagnostic results of two internal medicine doctors directly examining them with the AI ​​models o1 and GPT-4o of OpenAI. The results are evaluated by an independent team of doctors and it is completely unknown what results are from humans and from machines.

The o1 model demonstrates superiority at the initial emergency triage stage, when information about the patient is limited but requires the most accurate and urgent decisions. The data showed that o1 was accurate or nearly accurate in 67% of cases when triaging. Meanwhile, the rate of two doctors participating in the trial was 55% and 50%.

The research team emphasized that AI has access to the same amount of information from electronic medical records as doctors without any prioritization of data processing. “We tested the AI ​​model against most of the existing benchmarks, and it eclipsed both previous models and physician reference metrics,” said Arjun Manrai, head of the AI ​​lab at Harvard Medical School.

Despite the promising results, scientists insist the study does not mean AI is ready to replace humans in life-and-death decisions in the emergency room. Dr. Adam Rodman, co-author, warns that there is currently “no formal legal framework for accountability” for AI diagnostics. He emphasized that patients still want people to directly guide and accompany them through difficult treatment decisions.

Besides, according to TechCrunchthe study also encountered some critical opinions from experts. According to Kristen Panthagani, an emergency physician, the study compared AI to internists, not actual emergency medicine specialists. “The primary goal of an emergency physician when admitting a patient is not to see what the diagnosis is, but to determine whether the patient is in a life-threatening condition,” she said.

Additionally, research is currently limited to the processing of textual information. AI’s ability to think on non-text data such as live X-ray and ultrasound images is still a barrier compared to humans.