Study warns of risks of using AI chatbots to seek medical advice

According to a study published this Monday in Nature Medicine, there is still a huge gap between what language models (LLM) promise and their actual usefulness for patients seeking information about their symptoms.

LOOK: SpaceX will prioritize the construction of a city on the Moon over Mars, according to Musk

The study, led by the Oxford Internet Institute and the Nuffield Department of Primary Care Health Sciences at the University of Oxford, concludes that those who use AI to decide on the severity of a condition did not make better decisions than those who relied on traditional methods (such as searching the internet or their own judgment).

Recently, LLMs have been proposed by various healthcare providers globally as potential tools to conduct preliminary health assessments and manage your conditions before seeing a doctor.

To test this ability of AI, the study authors evaluated whether LLMs could help citizens accurately identify medical conditions, such as a common cold, anemia or gallstones, and decide whether to go to the family doctor or the hospital.

The study revealed that those who used AI did not make better decisions than those who searched for opinions on the internet or relied on their own judgment.

The team did a randomized trial with almost 1,300 participants who were asked to identify possible health conditions and recommend actions to follow.

The scenarios, detailed by doctors, ranged from a young man with a severe headache after a night out to a new mother who constantly felt exhausted and out of breath.

LOOK: Like feeling a screwdriver on your face: life with “the most painful disease known in the world”

One group used an LLM to assist their decision making, while a control group used traditional information sources, such as consulting the Internet.

The results revealed a significant gap between the theoretical performance of AI and its practical use.

After manually reviewing the interactions between humans and LLMs, the team discovered significant failures in two-way communication: participants often gave little or incomplete information to the model, but also that LLMs generated misleading or erroneous information with recommendations that mixed good and bad advice.

The study concludes that current LLMs are not ready to be implemented in direct patient care.

“Designing robust tests for language models is key to understanding how we can take advantage of this new technology,” says Andrew Bean, lead author and doctoral researcher at the Oxford Internet Institute.

“In this study, we show that interaction with humans is challenging even for the best models. We hope that this work will contribute to the development of safer and more useful AI systems.”

In view of the results of the study, the authors warn that like clinical trials for new drugs, AI systems should be tested in the real world before being implemented.

“These findings highlight the difficulty of building AI systems that can truly support people in sensitive and high-risk areas such as health,” says Rebecca Payne, a GP and leader of the study.

“Despite all the hype, AI is simply not ready to take on the role of the doctor. Patients should be aware that consulting a language model about their symptoms can be dangerous, as they may give erroneous diagnoses and fail to recognize when urgent help is needed.”