‘Humans must retain ultimate control over autonomous AI’

Mr. Kumar, Vice President of Asia Pacific region of ManageEngine, a division specializing in providing information technology management and monitoring software solutions under Zoho Corporation (India). He shared with VnExpress about the wave of autonomous AI as well as data security barriers for Vietnamese businesses.

– What is driving the wave of transition from “AI assistant” to “autonomous AI” – capable of making decisions and performing tasks in businesses?

– The core driver is data overload and real-time operational pressure. The traditional assistant model requires intervention, approval and command activation from humans, unintentionally creating a “bottleneck” when scaling. Businesses now need to optimize workflow performance. Repetitive tasks, from network monitoring to security vulnerability patching, need to be handled immediately without human delays.

The maturity of large language models (LLM) combined with AI agents capable of in-depth context understanding is the launching pad for autonomous AI to go beyond the limits of just providing suggestions, allowing the system to confidently make decisions and independently execute actions within pre-established safety boundaries. Gartner predicts that by 2028, 33% of enterprise software will have integrated autonomy features, a sharp increase from less than 1% in 2024.

– How does autonomous AI change the way businesses manage IT systems and handle complex technical problems compared to traditional methods?

– Autonomous AI will transform the IT management model from reactive monitoring to self-healing and proactive operations. Systems can perceive complex environments, make decisions, and act in real time with minimal or no human intervention at all.

As a result, the agent operates 24/7 without human approval, and makes decisions more effectively than a human operator at the same speed and scale. Even when a serious incident occurs, autonomous AI possesses comprehensive data collation capabilities, automatically analyzes the root cause, and executes remediation procedures in just seconds, thereby reducing system downtime.

– How do you assess the readiness of Vietnamese businesses to allow AI to automatically handle security-related tasks?

– Vietnamese businesses understand the theory well, but are very cautious when putting it into practice. The majority have an open mind and are willing to allow AI to automate security tasks of monitoring, classification and isolation at a basic level such as automatically aggregating and analyzing log data to detect unusual behavior, or automatically classifying the severity of network security alerts.

However, in core operations, where AI decisions can disrupt business operations, readiness levels are significantly reduced. This caution of Vietnamese businesses is not a weakness. On the contrary, it is a healthy risk management mindset that helps build a safe AI system with clear boundaries.

– When working with customers in Vietnam in particular and Southeast Asia in general, what do you find is the biggest fear of businesses when deploying AI?

– There are three major challenges that are holding back the journey of AI deployment. Loss of data control and security risks are top concerns, with concerns that putting internal data, customer information or business secrets into AI models could accidentally leak to the outside world, or violate strict privacy regulations.

The next fear involves deviations and “hallucinations”. In management and operations, one wrong decision can bring down the entire system or cause huge financial losses. Business owners are currently hesitant to delegate power to AI unless there is a clear authentication mechanism to understand its decisions.

Shortage of highly qualified human resources and hidden costs are also big problems for businesses. They are concerned that the current team of engineers is not capable of mastering, customizing and maintaining complex AI systems. Pressure on infrastructure costs, software licensing fees and fear of dependence on suppliers also makes them consider carefully.

The Vietnamese market is stepping from the awareness and testing stage to practical AI application. Domestic businesses have the advantage of open thinking, quick access to technology and are promoted by the National Digital Transformation Program. Many large and medium-sized enterprises have passed the basic AI tool testing stage. Amazon’s 2025 report shows that more than 18% of Vietnamese businesses have applied AI, an increase of 39% compared to the previous year, although 74% still mainly use AI at a basic level.

– Many Vietnamese businesses want to deploy AI quickly, but are afraid of losing control of the system. Where is the line between intelligent automation and giving too much power to AI?

– Need to maintain a “barrier” mechanism to help humans supervise. AI can perform common operational tasks on its own, but with strategic decisions or major impacts on the system, AI should only make recommendations, and the final decision should be made by engineers or operators.

In the digital race, speed is the advantage. But if an AI system operates at breakneck speed without oversight, it can magnify a small configuration error into a widespread system crash in seconds.

The most important factor for Vietnamese businesses is to build trust and ensure safety when applying AI. It is difficult for businesses to effectively utilize technology if employees are still concerned and lack trust in AI. You should start with low-risk internal tasks and gradually expand the scope of AI autonomy. At the same time, prioritize AI solutions that can explain the decision-making process, help managers understand the reasons behind each AI action and gradually build trust in the organization.

– How to avoid the scenario “AI makes a mistake but no one detects it”?

– Rapid deployment of AI systems across business functions poses a dual challenge for leaders: innovation coupled with risk management.

Businesses need to have settings to detect anomalies, allowing engineers to verify AI thinking processes immediately; set a threshold, mandating the activation of a human approval mechanism; cross-monitoring, which requires an independent monitoring layer to control AI behavior, all tasks must be automatically recorded in unmodifiable system logs.

In the AI ​​boom, due to the fear of being left behind (FOMO), many businesses rush to integrate without careful preparation, leading to unfortunate mistakes when deploying, the most common being training AI on “junk” data sources. As a result, AI learns outdated operating habits or makes erroneous decisions due to noisy input data.

Many businesses still make the mistake of “installing it and leaving it alone”. If AI is left to operate on its own without a systematic maintenance process, performance assessment, and periodic retraining, the AI ​​model will lose quality over time and make incorrect operating decisions.

Another common mistake is to underestimate the role of humans, thinking that just investing in AI tools can replace human resources or assigning all the work to machines. If the team is not trained on how to coordinate, monitor and work with AI, they can easily fall into the mentality of either being too dependent or lacking in trust in technology. Both reduce the effectiveness of AI deployment.

– With the problem of operational illusion, what red line should be set that AI is not allowed to cross?

– AI is not an invulnerable tool. Models are still prone to overstating, “making things up” or making mistakes, especially in completely new scenarios outside their training data set.

Enterprises need to establish a multi-layer monitoring mechanism, independent auditing and cross-validation. Never allow an AI agent to report on its own performance. Always compare the actions taken with the system log to detect early signs of erroneous AI reasoning before serious consequences result.

To keep AI agents under control, businesses need to set specific boundaries, such as absolutely prohibiting arbitrarily changing the network architecture and original security policy; Prohibit automatic bill payments or resource purchases exceeding pre-configured budget limits; Block privileges to manually delete backup data or move sensitive information. More importantly, there’s always an emergency shutdown button that instantly deprives the AI ​​of power, returning the system to a safe manual administration state if something goes wrong.

– When AI develops so quickly, is the legal framework running slower than technology?

– Certainly yes, the nature of law always requires time to observe, evaluate impact and build consensus. AI is evolving faster than the law is improving. Although Southeast Asia has made many efforts on safe AI and personal data protection, the region still needs more time to build a legal framework to clearly define responsibility when AI makes wrong decisions.

Whether there is a law or not, businesses need to proactively establish a clear governance mechanism, consider data sovereignty as a mandatory requirement, optimize technology infrastructure, improve team capacity and accompany partners with a long-term vision.