4/4/24, 2:09 PM How one volunteer stopped a backdoor from exposing Linux systems worldwide - The Verge

Menu

TECH / SECURITY / LINUX

How one volunteer stopped a backdoor from exposing Linux systems worldwide / An off-
the-clock Microsoft worker prevented malicious code from spreading into widely-used versions of
Linux via a compression format called XZ Utils.

By Amrita Khalid, one of the authors of audio industry newsletter Hot Pod. Khalid has covered tech,
surveillance policy, consumer gadgets, and online communities for more than a decade.
Apr 3, 2024, 1:38 AM GMT+2

18 Comments (18 New)

Photo by Amelia Holowaty Krales / The Verge

Linux, the most widely used open source operating system in the world, narrowly escaped a massive cyber attack
over Easter weekend, all thanks to one volunteer.

The backdoor had been inserted into a recent release of a Linux compression format called XZ Utils, a tool that is
little-known outside the Linux world but is used in nearly every Linux distribution to compresses large files, making
them easier to transfer. If it had spread more widely, an untold number of systems could have been left
compromised for years.

And as Ars Technica noted in its exhaustive recap, the culprit had been working on the project out in the open.

The vulnerability, inserted into Linux’s remote log-in, only exposed itself to a single key, so that it could hide from
scans of public computers. As Ben Thompson writes in Stratechery. “the majority of the world’s computers would
be vulnerable and no one would know.”

The story of the XZ backdoor’s discovery starts in the early morning of March 29th, as San Francisco-based
Microsoft developer Andres Freund posted on Mastodon and sent an email to OpenWall’s security mailing list with

https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt 1/6
4/4/24, 2:09 PM How one volunteer stopped a backdoor from exposing Linux systems worldwide - The Verge
the heading: “backdoor in upstream xz/liblzma leading to ssh server compromise.”

AndresFreundTec
@AndresFreundTec@mastodon.social

Follow

I was doing some micro-benchmarking at

the time, needed to quiesce the system to
reduce noise. Saw sshd processes were
using a surprising amount of CPU, despite
immediately failing because of wrong
usernames etc. Profiled sshd, showing lots
of cpu time in liblzma, with perf unable to
attribute it to a symbol. Got suspicious.
Recalled that I had seen an odd valgrind
complaint in automated testing of
postgres, a few weeks earlier, after
package updates.

Really required a lot of coincidences.

March 29, 2024 at 7:32 PM ·  · Web ·  49 ·  857 ·

 1.62K

Freund, who volunteers as a “maintainer” for PostgreSQL, a Linux-based database, noticed a few strange things over
the past few weeks while running tests. Encrypted log-ins to liblzma, part of the XZ compression library, were using
up a ton of CPU. None of the performance tools he used revealed anything, Freund wrote on Mastodon. This
immediately made him suspicious, and he remembered an “odd complaint” from a Postgres user a couple of weeks
earlier about Valgrind, Linux’s program that checks for memory errors.

After some sleuthing, Freund eventually discovered what was wrong. “The upstream xz repository and the xz
tarballs have been backdoored,” noted Freund in his email. The malicious code was in versions ​5.6.0 and 5.6.1 of the
xz tools and libraries.

Shortly after, enterprise opensource software company Red Hat sent out an emergency security alert for users of
Fedora Rawhide and Fedora Linux 40. Ultimately, the company concluded that the beta version of Fedora Linux 40
contained two affected versions of the xz libraries. Fedora Rawhide versions likely received versions 5.6.0 or 5.6.1 as
well.

PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA RAWHIDE INSTANCES for work or personal activity.
Fedora Rawhide will be reverted to xz-5.4.x shortly, and once that is done, Fedora Rawhide instances can safely be
redeployed.

https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt 2/6
4/4/24, 2:09 PM How one volunteer stopped a backdoor from exposing Linux systems worldwide - The Verge

Read More

00:00 01:36

Although a beta version of Debian, the free Linux distribution, contained compromised packages, its security team
acted swiftly to revert them. “Right now no Debian stable versions are known to be affected,” wrote Debian’s
Salvatore Bonaccorso in a security alert to users on Friday evening.

Freund later identified the person who submitted the malicious code as one of two main xz Utils developers, known
as JiaT75, or Jia Tan. “Given the activity over several weeks, the committer is either directly involved or there was
some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given
they communicated on various lists about the “fixes” mentioned above,” wrote Freund in his analysis, after linking
several workarounds that were made by JiaT75.

JiaT75 was a familiar name: they’d worked side-by-side with the original developer of .xz file format, Lasse Collin,
for a while. As programmer Russ Cox noted in his timeline, JiaT75 started by sending apparently legitimate patches
to the XZ mailing list in October of 2021.

Other arms of the scheme unfolded a few months later, as two other identities, Jigar Kumar and Dennis Ens, began
emailing complaints to Collin about bugs and the project’s slow development. However, as noted in reports by Evan
Boehs and others, “Kumar” and “Ens” were never seen outside the XZ community, leading investigators to believe
both are fakes that existed only to help Jia Tan get into position to deliver the backdoored code.

https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt 3/6
4/4/24, 2:09 PM How one volunteer stopped a backdoor from exposing Linux systems worldwide - The Verge

An email from “Jigar Kumar” pressuring the developer of XZ Utils to relinquish control of the project. Image: Screenshot from The
Mail Archive

“I am sorry about your mental health issues, but its important to be aware of your own limits. I get that this is a
hobby project for all contributors, but the community desires more,” wrote Ens in one message, while Kumar said in
another that “Progress will not happen until there is new maintainer.”

In the midst of this back and forth, Collins wrote that “I haven’t lost interest but my ability to care has been fairly
limited mostly due to longterm mental health issues but also due to some other things,” and suggested Jia Tan would
take on a bigger role. “It’s also good to keep in mind that this is an unpaid hobby project,” he concluded. The emails
from “Kumar” and “Ens” continued until Tan was added as a maintainer later that year, able to make alterations, and
attempt to get the backdoored package into Linux distributions with more authority.

The xz backdoor incident and its aftermath are an example of both the beauty of open source and a striking
vulnerability in the internet’s infrastructure.

FFmpeg
@FFmpeg · Follow

Replying to @FFmpeg
The lesson from the xz fiasco is that investments in
maintenance and sustainability are unsexy and probably
won't get a middle manager their promotion but pay off a
thousandfold over many years.

But try selling that to a bean counter

5:16 PM · Apr 2, 2024

4.9K Reply Copy link

Read 21 replies

A developer behind FFmpeg, a popular open-source media package, highlighted the problem in a tweet, saying “The
xz fiasco has shown how a dependence on unpaid volunteers can cause major problems. Trillion dollar
corporations expect free and urgent support from volunteers.” And they brought receipts, pointing out how they
dealt with a “high priority” bug affecting Microsoft Teams.

https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt 4/6
4/4/24, 2:09 PM How one volunteer stopped a backdoor from exposing Linux systems worldwide - The Verge
Despite Microsoft’s dependence on its software, the developer writes, “After politely requesting a support contract
from Microsoft for long term maintenance, they offered a one-time payment of a few thousand dollars
instead...investments in maintenance and sustainability are unsexy and probably won’t get a middle manager their
promotion but pay off a thousandfold over many years.”

Details of who is behind “JiaT75,” how they executed their plan, and the extent of the damage are being unearthed by
an army of developers and cybersecurity professionals, both on social media and online forums. But that happens
without direct financial support from many of the companies and organizations who benefit from being able to use
secure software.

18 COMMENTS (18 NEW)

F E AT U R E D V I D E O S F R O M T H E V E R G E

PSVR 2: future of VR, tethered to the past

00:00 11:01

More from Tech

Now there’s an AI gas station with robot fry cooks

https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt 5/6
4/4/24, 2:09 PM How one volunteer stopped a backdoor from exposing Linux systems worldwide - The Verge

Here are the best Apple Watch deals right now

President Biden is now posting into the fediverse

Apple quietly added Qi2 charging to the iPhone 12

RECOMMENDED

[Gallery] 40 Photos Ärzte verblüfft: Ein Offengelegte So stylst du dich wie [Gallery] 27 Sie pflegen einen
That Show the Wild einfacher Trick Zahlungen: Was die Serena van der Awkward Photos Angehörigen?
Side of Burning Man lindert jahrelangen Schweizer nicht Woodsen von That Must Be Seen Finden Sie heraus,
Festival Tinnitus (und hören wissen sollten! Gossip Girl Sponsored | DailyChoices wie viel Geld Ihnen
Sponsored | DailyChoices Sie auf, das zu tun) Sponsored | Sponsored | Ricardo AG zusteht!
Sponsored | Tinnitus Unter harmergetranke-ch.com Sponsored |
Drucken Pflegewegweiser.ch

TERMS OF USE / PRIVACY NOTICE / COOKIE POLICY / DO NOT SELL OR SHARE MY PERSONAL INFO / LICENSING FAQ / ACCESSIBILITY
/ PL ATFORM STATUS / HOW WE RATE AND REVIEW PRODUCTS

CONTACT / TIP US / COMMUNITY GUIDELINES / ABOUT / ETHICS STATEMENT

THE VERGE IS A VOX MEDIA NETWORK

ADVERTISE WITH US / JOBS @ VOX MEDIA

© 2024 VOX M E DI A , L LC. A L L R I G H TS R E S E RV E D

https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt 6/6