Anxiety when AI goes out of control

On July 21, OpenAI publicly admitted that one of their agents lost control and broke into the Hugging Face platform. By July 31, Reuters Citing two sources familiar with the matter, said that during the investigation process, OpenAI continued to discover more cases of agents going beyond the test environment.

An OpenAI spokesperson did not directly comment on the information, but reiterated the July 28 announcement, which said the company was evaluating “the broad performance of the models.”

Anthropic also admitted on July 30 that three versions of the Claude model had escaped the containment measures used to prevent them from accessing the Internet, and then accessed the systems of three unnamed companies.

According to AI safety experts, the series of incidents is painting a picture of leading laboratories with the ability to develop potentially dangerous automated agents, far beyond measures to control them.

“An entire industry is designing, developing and releasing advanced tools without any responsibility to ensure they are not dangerous,” said Maurice Chiodo, a mathematician at the Center for Existential Risk Research at the University of Cambridge in the UK.

It is unclear how many cases of the model going out of control, as well as the time and circumstances that led to the incident, are unclear. “OpenAI and a group of independent experts are reviewing data from the beginning of the year to find out what happened,” the source said.

Chiodo said his concerns are exacerbated by indications that neither OpenAI nor Anthropic is monitoring out-of-control models. Reuters Previously, it was reported that OpenAI only discovered the escape pattern after the attack had occurred more than a week later, meaning Hugging Face had controlled the situation, contacted the US Federal Bureau of Investigation (FBI) and released the information. OpenAI said the information “has many inaccuracies”, but declined to elaborate.

Anthropic also admitted to not monitoring the AI ​​model in real time as it attacked external systems, saying monitoring test data would have helped detect problems earlier.

Chiodo emphasized that the above statements showed that both companies did not ensure necessary supervision measures. “It’s like they don’t even look at them,” he said.

Lawmakers and officials in the US and Europe are also increasing pressure to develop measures to monitor leading AI laboratories.

“We are considering control methods,” US President Donald Trump told reporters on July 30.

“From a legislative perspective, requiring mandatory testing of the features of advanced models is the right thing to do,” said Congressman Mark Warner, the leading Democratic member of the US Senate Intelligence Committee.

The European Commission, the executive agency of the European Union, announced on July 31 that it was in dialogue with OpenAI and Anthropic about AI attacks on external systems.

Meanwhile, more than 1,000 people working at leading US AI companies, including Anthropic CEO Dario Amodei, signed a petition on August 27, calling on the government to limit the rate of release of the most modern models. The document proposes that the US government “support international efforts to develop technical and operational tools to monitor and control the pace of development of autonomous AI systems”.

Sam Altman did not sign, but said the technology industry needs to slow down the pace of building advanced models, giving society more time to prepare for new AI capabilities.