What is “model collapse” and how hallucinations threaten the future of AI | ChatGPT | Gemini | DeepSeek | Claude

As recalled in the article published in ‘Physical Review Letters’, the term “model collapse”, first coined in 2024, refers to a scenario in which an AI model trained with AI-produced data stops providing accurate results and instead produces inaccurate “gibberish” due to the poor quality of its training data.

LOOK: Google identifies the first zero-day vulnerability developed with the help of an AI

Some have warned that high-quality text data for training systems like Large Scale Language Models (LLMs) will run out later this year, so data produced by the models themselves has taken on a larger role in training, bringing the threat of model crash.

By analyzing a set of simple but powerful statistical models, called Exponential Families, the team of researchers from King’s College London, in collaboration with the Norwegian University of Science and Technology (Norway) and the Abdus Salam International Center for Theoretical Physics (Italy), discovered that it was enough to integrate a single piece of data from the outside world into their training to avoid this problem in all cases.

Although much simpler than generalized linear models (LLM), exponential family models are among the most powerful for data modeling. The team hopes that by looking at closed-loop learning in such a simple yet powerful environment, they can establish principles to prevent model collapse in most common LLMs.

Professor Yasser Roudi, Professor of Disordered Systems in the Department of Mathematics at King’s College London, explains: “Previous work on model collapse has mainly focused on large, complex LLM models, where it is not clear how these models work or whether the results are repeatable, leading to unexplained hallucinations, where it cannot be explained why an AI has generated an erroneous response.

By focusing on a simple model, we can determine why adding a single piece of data prevents them from generating information that is statistically meaningless. Starting from this foundation, we can establish principles that will be fundamental for the development of AI in the future. As more complex models are implemented in areas that affect our daily lives, from ChatGPT to self-driving cars, and synthetic data becomes more prominent in AI training, computer scientists will have the tools necessary to prevent this potentially disastrous scenario.

The published study exposes how standard training of exponential families (called maximum likelihood) in a closed-loop scenario, where a model is trained only with the data it produces, will always lead to model collapse.

However, the study shows that introducing a single piece of external data into the closed loop, or incorporating a prior belief during training (for example, based on previously acquired knowledge), prevents the model from collapsing. Surprisingly, this effect of a single external data holds even when the amount of data generated by the machine is infinitely larger.

The authors also provide evidence that a similar phenomenon is observed in another class of models, restricted Boltzmann machines, suggesting that their results are probably not limited only to exponential families. In the future, the group hopes to test these first principles with larger, more complex models, such as neural networks, to validate them.