AI shows strong performance on clinical reasoning tasks, but don’t expect it to replace doctors anytime soon

This is indicated by a study published by Science, although its authors warn that these results do not mean that artificial intelligence (AI) systems are prepared to practice medicine on their own, nor that doctors can be excluded from the diagnosis process.

The research, led by Harvard Medical School (USA) and using real data from emergency services with the language model, evaluated whether AI could, like doctors, review a messy medical history and use that information to determine the diagnosis and the steps to follow.

Overall, the results show that LLMs “currently offer notable performance in differential diagnosis, clinical diagnostic reasoning, and treatment reasoning, outperforming both previous generations of models and clinicians themselves in multiple domains,” the study indicates.

These same performance improvements are seen “when offering second opinions in real and unstructured medical cases in the emergency departmentwhere doctors must act quickly with limited and often incomplete information.”

The authors point out, among the limitations of the study, that their focus was limited to text-based reasoning, while clinical practice relies heavily on visual and auditory cuesareas in which current AI continues to have less capacity.

The study, led by Harvard’s Peter Brodeur, evaluated the diagnostic and treatment planning capabilities of a large-scale advanced language model—OpenAI’s o1 series—by comparing its performance to hundreds of previous clinician and AI systems on a wide range of clinical reasoning tasks.

These included both standardized clinical cases and a real-world study with randomly selected emergency room patients.

The advantage of LMM was most evident in the initial triage of patients in the ED, where physicians must make rapid decisions with minimal information.

Although both humans and AI improved as more clinical data became available, the model demonstrated strength under conditions of uncertainty, effectively using even fragmented and unstructured data from medical records, the journal summarizes.

LLMs, according to the authors, are rapidly approaching human-level clinical reasoning, and in some areas surpassing it.

Which does not mean that AI systems are prepared to practice medicine autonomously. “A model could get the primary diagnosis right, but also suggest unnecessary tests that could endanger the patient,” Brodeur said, in a Harvard statement.

An opinion article related to the study and signed by experts from Flinders University (Australia) indicates that AI must be carefully evaluated and regulated before its widespread adoption in the healthcare field, since rapid advances do not automatically translate into safe use for patients.

Researchers recognize that recent advances in AI offer real opportunities to support clinicians, especially in high-volume, high-pressure care settings.

But, they also underscore that real-world healthcare involves much more than text-based reasoning or test performance. Clinical practice depends on physical examination, listening to patients, understanding the medical and social context, and taking responsibility for the results.

Looking ahead, Flinders researchers argue that enthusiasm for medical AI must be accompanied by strong governance and clearer evaluation criteria.