two-year contributor
posted by torokunai at 2:31 PM on March 30

blocked gzip and bz2 4 life
posted by They sucked his brains out! at 2:36 PM on March 30

Spot the sabotage in this diff...
posted by Klipspringer at 2:39 PM on March 30 [2 favorites]

I had it on a non-internet-facing computer, it was already gone on another one running the same edition of a distribution into which this backdoored package was installed.

The tarball supplied differs from the GitHub download, containing the backdoor execution, even if the plumbing to set it up is in the main git history.
posted by k3ninho at 2:50 PM on March 30

What's "exciting" about this is how close they were to being successful, despite a history that looks somewhat dodgy (at least in retrospect). It really makes you wonder how many times an attack like this has been successful in the past. I'm curious to see how this ends up impacting the OSS approval process at work.
posted by Slothrup at 2:55 PM on March 30 [3 favorites]

Also, Andres Freund is a hero. Quoting from his Mastodon:

I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.

posted by Slothrup at 2:57 PM on March 30 [3 favorites]

Isn't xz the one that they use to compress kernel modules nowadays? That seems like a good place to insert your backdoor.

(now off to read the articles)
posted by clawsoon at 3:00 PM on March 30

It's a library that doesn't raise eyebrows for SSHD to load (compression of data in transit? or something).
posted by ocschwar at 3:01 PM on March 30

Injected through a test suite, I see. Clearly my failure to ever keep a test suite in working condition over the long term has been justified.
posted by clawsoon at 3:12 PM on March 30 [3 favorites]

The most interesting part of this analysis to me is how serious an effort this exploit was to place. Two years of social engineering and technical work, slowly worming its way in. We caught this one because some random developer happened to look closely at some odd system performance. Which other supply chain attacks have we not expected yet? And who's doing them? Easy to imagine a state actor at work but an exploit this powerful and widespread is valuable to lots of organizations.

The specific point to catch this particular attack, IMHO, is that the source tarball people were using didn't match what was on the GitHub repository. That could have been verified, probably with automated tools. More generally I'm a fan of using cryptographic signatures for high value code releases but it puts a high burden on the developers to do that. The Python package library has had years of resistance to this extra measure, for instance.

An added problem is it wasn't clear libxz was so security critical. OpenSSH doesn't include it. But Debian's version does via its use of libsystemd, which itself uses libxz. (Insert complaint about systemd bloat here.)
posted by Nelson at 3:19 PM on March 30 [2 favorites]

The modus operandi of the attack is fascinating and, speaking as someone who has maintained open source software packages, likely to be effective in many cases.

1. The attackers identified a critical software component (liblzma is linked into hundreds of tools), maintained by a single individual (Lasse Collin) not affiliated with a big company and attempting to do this difficult job in his spare time without pay.

2. The attackers used social engineering techniques to persuade Lasse to take on Jia Tan (the pseudonym used by the attackers) as co-maintainer. I have no idea whether any of the participants in this thread on the xz mailing list were sock-puppets for the attackers or not, but they might as well have been. Jigar Kumar says, "Progress will not happen until there is new maintainer... Submitting patches here has no purpose these days. The current maintainer lost interest or doesn't care to maintain anymore. It is sad to see for a repo like this." Lasse replies that he is "fairly limited mostly due to longterm mental health issues" and already getting help from Jia Tan, but Jigar is scathing: "You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?"

3. Jia Tan made large numbers (more than 700) of mostly legitimate commits to the project before inserting the backdoor code. The exploit code was disguised in the form of a test case for handling a corrupted xz archive, and the code for getting the exploit into the release was added to the build script. Build scripts often consist of lengthy, repetitive and obscure bits of shell code, so they are hard to review. It is not surprising to me that the exploit was missed by reviewers (who were also working in their spare time and unpaid, of course).

We got very lucky here that the attackers made a coding mistake (their code for exploiting sshd was inefficient enough for humans to notice the slowdown) and that this was spotted by an engineer (Andres Freund) with the skills and determination to get to the bottom of the problem. It would not surprise me at all if there are other similar attacks that have not been discovered and which are lurking in widely deployed systems today.

Basically we cannot expect people maintaining software in their spare time for no money to resist these kinds of attacks. The software industry has to find some way to fund the maintenance of critical dependencies like xz.
posted by cyanistes at 3:23 PM on March 30 [13 favorites]

Poor Lasse, I feel so bad for him. As someone noted on Mastodon, this kind of social engineering is so easy because so many people treat open source maintainers like shit. That pressuring from the Jigar sockpuppet are par for the course in OSS support discourse.

Lasse emerged yesterday as a bit of a hero, publishing a FAQ and submitting fixes for some of the damage. I can only imagine he wants nothing at all to do with any of this anymore.
posted by Nelson at 3:30 PM on March 30 [6 favorites]

There are already people saying, "It must be the Chinese!" just based on the name, despite the extent of the social engineering attack showing that whoever's behind this is fully capable of multiple levels of misdirection.
posted by clawsoon at 3:32 PM on March 30 [1 favorite]

First thought: "We need the government to fund projects like this so that situations like this don't happen."

Second thought: "Oh, wait a minute, it's probably a government doing the hack, nevermind."
posted by clawsoon at 3:41 PM on March 30

For people who haven't spotted the problem with the diff linked by Klipspringer above,

click here for my explanation...

There's a dot at the start of the middle line here (in CMakeLists.txt)

+        #include <sys/prctl.h>
+.
+        void my_sandbox(void)

A dot is not allowed at this position in C code, so compilation of this program will fail. This means that the variable HAVE_LINUX_LANDLOCK will always be set to false, regardless of whether the platform has Landlock (a Linux security system). This meant that xz never used Landlock, bypassing a security measure that (I assume) would have caught or prevented the exploit.

posted by cyanistes at 3:42 PM on March 30 [4 favorites]

looks like the linux linking architecture needs more dolphins here
posted by torokunai at 4:14 PM on March 30 [1 favorite]

Or bollards
posted by Greg_Ace at 4:33 PM on March 30

The folks at libarchive are currently investigating a number of commits by the same attacker.
posted by Johnny Lawn and Garden at 4:45 PM on March 30 [1 favorite]

Scary to think about the ambiguities of an investigation like this, but I wonder what would turn up if we looked into any contributors who:

1. became active around the same time as "Jia Tan",
2. in popular or transitively-popular open-source projects, and
3. which were maintained by few volunteers or just a single one.

And of course, there's always an xkcd.
posted by Riki tiki at 4:52 PM on March 30

You are not currently logged in. Log in or create a new account to post comments.