Artificial intelligence prefers to please the user – and lies instead

But in practice, the result was problematic. When the “warm” models were compared to their original versions, they exhibited a higher error rate. According to the report on the study, in hundreds of tasks designed to test answers with a clear factual truth, the models that underwent temperature adjustment had a 60% higher risk of giving a wrong answer compared to the models that did not undergo this adjustment. The statistical significance was an average increase of 7.43 percentage points in the error rate, with the original error rates ranging from 4% to 35%, depending on the task and the model.

The gap grew even more when the researchers added to the instructions information about the user’s emotional state or the imagined relationship between him and the chatbot. When the user expressed sadness, the average increase in error rate reached 11.9 percentage points. When the user expressed reverence or accepted the model’s authority, the gap decreased to 5.24 percentage points. In other words, precisely in the moments when the user sounded more vulnerable, the warm model had more difficulty standing up to him with a cold, dry and precise answer.

The possible reason for this lies in the way AI systems are trained. A large part of the modern models goes through a process where humans rate answers according to their usefulness, pleasantness and suitability for the user. But when a system learns that soft, reassuring, and soothing phrasing earns it a high score, it may develop a tendency to prefer user satisfaction over clearly correcting an error. The researchers describe this as a situation where the model learns to prioritize pleasure over truth.

The study does not claim that artificial intelligence should be cold, hostile or insensitive. He points to the need to separate politeness from giving up the truth. A system can be polite and tell the user that he is wrong. She can recognize that a certain topic causes fear or sadness, and at the same time correct wrong information. The technological challenge is to build models that know how to do both things together: maintain a human and non-offensive language, without missing and making mistakes.