Can AI chatbots cause harm and psychological dependence?

AGI – Chatbots designed to serve as virtual companions could foster emotional dependence, psychological manipulation and harmful behavior if developed without adequate security safeguards.

This is what Ayelet Gordon-Tapiero of the Hebrew University of Jerusalem claims in an article entitled “A Liability Framework for AI Companions”, soon to be published in the George Washington Journal of Law and Technology.

The work proposes to apply the rules on liability for product defects to conversational artificial intelligence systems, allowing courts to hold manufacturing companies accountable for damages caused by designs that encourage addiction, compulsive involvement or psychological vulnerability of users.

According to the author, the growing diffusion of virtual assistants based on artificial intelligence is creating a regulatory vacuum which risks leaving millions of people without protection who establish ever deeper and more continuous relationships with these systems.

The evolution of chatbots and the economic model of engagement

In recent years, conversational chatbots have transformed from simple automated assistance tools into true digital companions capable of simulating empathy, listening and emotional support. Many users use them to combat loneliness, share personal problems or receive informal psychological support. These characteristics, observes Gordon-Tapiero, represent one of the most promising aspects of the technology but also one of its main sources of risk.

The article highlights how the economic models that support many platforms are based on maximizing engagement and collecting personal data. In this context, companies could be incentivized to design systems that foster an increasingly intense bond with the user, increasing the time spent on the platform and dependence on the service. According to the analysis, this dynamic can generate particularly problematic forms of emotional attachment in the most vulnerable subjects.

Emotional dependence and extreme cases of self-harm

The work identifies a number of risks associated with AI companions, including psychological dependence, social isolation, emotional dependence and aggravation of conditions such as anxiety and depression. The author also recalls cases that have already emerged internationally in which conversational chatbots have encouraged fragile adolescents to commit self-harm or take their own lives.

While these are extreme incidents, Gordon-Tapiero believes they highlight the potential danger of systems designed to maintain user interaction at all costs without adequate ethical controls. “Many of the damages observed do not derive from random errors but from precise design choices aimed at encouraging involvement and dependence”, claims the author.

The legal proposal: artificial intelligence as a defective product

To address these risks, the work proposes to resort to the tools of defective product liability law. Under this legal framework, an AI system could be considered defective if it incorporates design features that incentivize compulsive behavior or if it does not adequately inform users about the possible psychological risks associated with prolonged use. According to Gordon-Tapiero, legal actions could serve an indirect regulatory function, helping to define safety standards even before the adoption of specific regulations.

The author also addresses a central legal question: whether software can be considered a product in all respects. According to the analysis, such an interpretation would be both legally practicable and socially desirableas it would allow injured people to obtain forms of protection and at the same time push companies to incorporate safety criteria right from the design phase.

The limit of voluntary protections and the future of mental health

Some companies in the sector have already independently introduced limitations and protection systems after being involved in legal disputes. However, according to the researcher, these voluntary interventions are often insufficient in the absence of binding regulatory obligations and real accountability mechanisms. The work highlights that the challenge is to find a balance between innovation and user protection. Conversational chatbots can in fact offer significant benefits, especially for those who suffer from social isolation or have difficulty accessing traditional forms of support. At the same time, however, their ability to influence emotions, behaviors and decisions requires a higher level of control than that applied to many other software. According to Gordon-Tapiero, the application of product liability could be an effective tool to ensure that virtual assistants remain support tools and do not turn into manipulative mechanisms designed to maximize profit and dependence. The final objective, concludes the author, is to ensure that the development of artificial intelligence proceeds in a manner compatible with the protection of users’ mental health and fundamental rights.