“I accidentally encountered a security issue while performance testing shifters,” engineer Andres Freund wrote on the Mastodon social network. That chance discovery prevented one of the longest and most sophisticated operations to gain illegitimate access to millions of devices around the world from progressing.
The message led to a link where Freund explained how he had encountered “a bunch of strange symptoms” when updating a program. He drew her attention to the fact that it used more of his processor’s capacity and, above all, it took half a second longer to access. That half-second raised his suspicion and allowed him to discover the dark work of a state intelligence agency for more than two years.
“It is very unlikely that it was the work of amateurs. There were no immediate rewards,” says Lukasz Olejnik, independent cybersecurity researcher and consultant and author of the book Philosophy of Cibersecurity [sin traducción al español]. “The time spent on this deception operation, the sophistication of the backdoor system and its code, all point towards an organization or agency that can afford such a project. It is simply much more likely that it was done by paying salaries.”
The attack was a so-called supply chain attack, which affects the software that supports the most well-known and common programs. In this case, the target was a compression tool used in Linux, a free and open source operating system. That tool is used in millions of machines. The goal of the attack is similar to creating a back door with a special key, that only they had, to access any building in the world that had that entrance.
This system is maintained thanks to volunteer developers who spend hours maintaining and updating different programs. This was called XZ Utils. A little over two years ago the attacker began collaborating with the programmer who was in charge of updating this software. The person in charge of updating and responding by email to requests for tweaks from other developers was overwhelmed. Part of the attack consisted of pure social engineering: convincing him to leave part of his tasks to someone behind an account of someone who called himself Jia Tan.
If the attacker gained the trust of the person in charge of that code, he could, over time, place his malicious code. If it had not been detected, this software would have been deployed on millions of servers and given privileged access. It is unclear whether the intent was to use it to break into one or more specific machines or a more massive attack.
Although the code and method require extraordinary computer skills, control of these programs often depends on stressed and troubled developers. In a message thread, the manager admits not getting to everything: “I haven’t lost interest, but my ability to get involved has been quite limited, mostly due to long-term mental health issues, but also for a few other reasons. . I’ve recently collaborated off-list with Jia Tan on XZ Utils and perhaps he’ll have a bigger role in the future, we’ll see. It is also important to keep in mind that this is an unpaid hobby project,” writes the manager, whose only new explanations have been that for the moment he will not respond to journalists “because first I need to understand the situation thoroughly enough.”
“There are a lot of people burned out in software, both open source and commercial. In this case it can be useful, but not a decisive factor,” says Olejnik. “It is compelling proof that even niche or obscure, semi-orphan software can be a matter of national and international security. It is a hidden cost of the software. On the other hand, no one can blame the maintainer of XZ, there is not a wide choice of developers for this type of software,” he adds.
It is likely that other fake accounts pressured the manager to hand over his work to Jia Tan sooner. The case reveals both a success and a hole in the community that maintains much of the code of our entire digital infrastructure. The hole is that finding the weak link is relatively easy. The success is that the code is available and accessible so that someone like Andres Freund can detect the trap and become famous.
Freund himself believes that this time they were lucky: “It’s not that I think I didn’t do anything new. I did it. What I mean is that we had an irrational amount of luck and we can’t just rely on something like that from now on,” he wrote on Mastodon. The attack is special due to the combination of factors, but the free source software blocks on which the Internet is based have been attacked on other occasions, also by alleged intelligence agencies. It is likely in fact that there are other similar cases underway or proposed. With closed source there have also been extremely famous cases.
The xz backdoor was initially caught by a software engineer at Microsoft. He noticed 500ms lag and thought something was suspicious.
This is the Silver Back Gorilla of nerds. The internet final boss. pic.twitter.com/6IyJQ2tpMm
— vx-underground (@vxunderground) March 30, 2024
A famous X account (formerly Twitter) dedicated to malicious code has made a viral meme thanking Freund. “The xz backdoor was caught by a Microsoft software engineer. He noticed a latency of 500 milliseconds and thought something was strange. This guy is the silverback gorilla of geeks. The fucking master of the internet.”
This other meme makes even more sense, showing how, in this case, the world’s essential software was “suspiciously maintained by an actor paid by a state during office hours.” The original drawing on which this meme is based is the work of cartoonist Randall Munroe and in the legend he says something similar to what happens in reality: “A project that a random person from Nebraska has been maintaining since 2003 without anyone thanking him.” .