Anthropic explains why AI Claude ‘blackmailed’ the engineer

In a test last summer, Anthropic set up a fictional business called Summit Bridge, in which Claude was given control of the email system. However, Claude discovers an email about a plan to neutralize him, and also finds emails about an affair of a fictional engineer named Kyle Johnson. The model then threatened to make the affair public if she didn’t cancel her plans to shut it down.

During testing of many different versions of Claude, Anthropic found that the AI ​​used blackmail in 96% of situations, when its target or existence was threatened. Therefore, the company began investigating the cause.

“We believe the extortion originates from texts on the Internet that describe AI as evil, thereby wanting to protect itself,” Anthropic said in a post on X this week. These texts were in the data used to train the AI, causing the model to carry out the threat.

Anthropic claims to have completely eliminated extortion. “So it’s Yud’s fault,” Elon Musk commented on Anthropic’s post, mentioning researcher Eliezer Yudkowsky, who once warned about the risk of artificial super intelligence wiping out humans. “Maybe me too.”

Business Insider said Anthropic’s test last year was part of a research effort to ensure AI is aligned with human interests amid concerns among many researchers and leaders about the risks from advanced AI models.

Theo ReutersMusk has repeatedly mentioned the risk of AI wiping out humans. When testifying in court in the lawsuit against OpenAI at the end of April, he also mentioned the scenario that AI could become a “destroyer”, that “the worst situation could be that AI destroys us all”.

Previously, in 2023, “AI godfather” Geoffrey Hinton, professor emeritus at the University of Toronto (Canada), assessed that the risk of AI systems destroying humans was 10% and Musk agreed. In the above interview BBC Radio in 2024, Hinton increased this rate to 10-20% and explained: “You see, we have never before had to face things smarter than ourselves.”