Large Vietnamese language model with 120 billion parameters

On June 4, Viettel Artificial Intelligence and Data Service Center (Viettel AI) said that VT-Super-120B-A12B was “directly trained, fine-tuned and optimized for Vietnamese by a team of Vietnamese engineers”.

The model is built on Nvidia Nemotron 3 Super open architecture, with a scale of 120 billion parameters. According to the Center’s representative, thanks to the long context processing ability of Nvidia Nemotron architecture, VT-Super-120B-A12B can maintain a smooth flow of information between many documents, processes and complex conversations in the same task.

In large language models, “parameters” refer to the values ​​that the model learns during the training process to identify patterns in the data, thereby making predictions or responses. The higher the parameters, the larger the scale of the model, capable of representing complex relationships, but at the same time requiring more data, computing power and operating costs. Vietnamese models usually range from a few billion to several tens of billions of parameters, while the most advanced models of OpenAI and Google can have trillions of parameters.

However, a large number of parameters does not mean that the quality is always better, because the effectiveness also depends on the training data, model architecture and optimization method. In the VLMU capacity assessment ranking, a refined version based on VT-Super-120B-A12B currently has an average score of 85.47, ranking third among models in Vietnam, with the strongest in the STEM ranking with more than 89 points.

Viettel AI said that through tests, its model performed “in the leading group” in terms of accuracy compared to models of the same scale. “This is the result of training on local and optimal data for domestic operational problems,” said the Center’s representative.

This model is assessed to have enhanced Vietnamese language processing capacity without degrading the original model’s English performance, while also limiting the phenomenon of forgetting old knowledge, a common problem during the training and fine-tuning process. According to the developer, this is important for specific problems in Vietnam, where many regulations and processes may change depending on the type of record, or the specific implementation context.

Viettel AI also said it is building an LLM training process based on local data sources such as administrative data, business operations, real conversations and specialized document systems in Vietnam. This process includes many stages, such as continued pre-training to expand Vietnamese language capacity and knowledge on a large-scale data warehouse; Supervised fine-tuning to improve reasoning ability, followed by reinforcement learning to improve response accuracy and business processing capabilities.

“Mastering the LLM in Vietnamese is a core step to form sovereign AI solutions capable of truly accompanying Vietnamese organizations and businesses,” said Mr. Nguyen Manh Quy, Director of Viettel AI.

Ms. Shilpa Kolhatkar, Director of Nvidia’s AI Nations, assessed that the development of a large Vietnamese language model will contribute to universalizing access to artificial intelligence on a large scale in Vietnam, while transforming local languages ​​and data into practical application values ​​for government agencies and businesses.

Viettel AI said it is developing an AI Agent platform for Vietnamese people with the ability to perform a series of tasks in the same workspace. In particular, AI Legal Assistant is one of the first applications, expected to possess the ability to support document analysis, compare regulations, synthesize data and propose solutions according to the user’s specific problem, with improved quality compared to previous versions.