Privilege Escalation 01: Half a Second From Disaster
https://codykretsinger.com/episode?show=privilege-escalation&ep=1

Cold open:
For almost three years, this guy was the best thing that ever happened to the project. He was reliable, did the boring work that nobody else wanted, fixed the bugs, answered emails, and took a load off a maintainer who was drowning. If you were that maintainer, he was a gift. Then in February 2024, that gift shipped a backdoor into software that runs underneath a huge chunk of the internet. The kind of access where you log into machines anywhere on the planet like you own them. And we didn't catch it because we were good. We caught it because one engineer at Microsoft was chasing a totally unrelated slowdown.

He noticed his logins were running about half a second, yes, half a second slow, and got annoyed enough to really dig in. Half a second. That's The whole time, the whole margin. That's the difference between a normal day of the week and the worst thing that could have ever happened to us.

Narrator:
This is Privilege Escalation. Part of the Threat Aware Podcast Network. Incidents get reported. They rarely get explained. Each episode takes one attack apart from the inside, delivered with a perspective of someone who has spent time on both sides of an attack. How they got in, and how you stopped them. Here's your host, Cody Kretsinger.

I'm Cody Kretsinger, and this is Privilege Escalation. You've already read the XZ write-up and So did I about 40 times. We're not gonna do that today. And instead, today I want to talk about what this actually was, which is a con. And how these things really work, because I used to run them. XZUTils is a compression tool. In fact, it's boring as hell. And it ships basically in every version of Linux. So it's sitting underneath. Servers, the cloud, probably a big chunk of your infrastructure that you're running right this second. For years, one guy maintained it, one volunteer completely unpaid.

His last name is Colin, that'll make sense more here in this story. Everybody wants to make this particular incident, if you want to call it that, about the code, the SSH hook, the injection buried inside the build itself. And we'll get there eventually, because frankly it's clever. But I don't want you to walk away from this thinking that code was a problem. Because that's the wrong take. This was a people problem from beginning to end. So here's the timeline. A new contributor shows up. Their handle doesn't matter, but they start sending in patches. It's small stuff, actually, frankly, really good stuff.

Nothing weird because nothing weird is actually happening yet. It's just a helpful stranger building a track record month after month, year after year. I want you to feel how boring that is, because there's no attack here, and there won't be for quite some time. And then eventually there's some pressure that starts building. The pressure really starts and it doesn't come from this new contributor. That's one of the tells, actually. Other accounts show up on this mailing list, this repo's mailing list, and one of them starts really getting on Collins case. Frankly, in public. The project's too slow.

You need help. Maybe this other contributor should have more control. And here's the thing, folks. The maintainer, he is honest. He's open. He tells people. He tells the whole mailing list that he's struggling. He does so in a very public way. He mentions his mental health, says he's got no time, that he's unpaid, he's just one guy. The attackers know now exactly where to push. Not because the maintainer screwed up. He didn't. He was honest. He was buried. And who on earth thinks that a mailing list is a hunting ground for threat actors? So now you have a burned out maintainer.

A crowd telling him to hand off the keys, and one incredibly patient, incredibly helpful guy just so happens to be standing right there. It works. The gift to this project now has commit access. He becomes a co-maintainer. He gets trust. Now, years in, the payload goes in. It's buried deep in the build. Hidden in files dressed up as test data, shaped so a normal code review will just slide right past it. And on the machines that matter, it wires straight into SSH. The right key, and you're in. Everywhere it's installed. It was just it was days from hitting everybody's stable release when it got caught by the lag.

The half a second. Lag purely by accident from a guy who wasn't even looking for it. So I've run long cons. Not this one, not this one, but this kind, if you want to call it that. And when I hear people call the XEU Tills a sophisticated malware, I know they're looking at the wrong thing. The tradecraft here isn't the code itself. It's the patience. And patience is one thing nobody really can defend against because defending against it is a slow. Boring, expensive thing to do. And everything this attacker did, everything this contributor did, I recognize.

They built a legend, real history, real contributions. Because when you do this right, most of what you do is actually helpful. The help is the disguise itself. It's the thing that you are wearing. You don't sneak past the guard. You spend two years becoming the guy who brings the guard his coffee in the morning. They maintain that trust relationship. They kept their own hands clean. On this pressure part as well, if you can recall that. That's the elegant portion of all of this, because the candidate stays nice, they stay reasonable, while other voices, whether they're actual real voices or s you know the same person, are doing the pushing.

You never want to be the one actually making the tension in this particular case. You want to be the calm fix to a problem that Everybody else is making so loud. And this is the part that hurts the most. They went after the tired guy. Not the strongest link, the most exhausted, worn out, fed up. The individual that just needed help. And he's gonna be the one that's going to feel the relief handing it off, not the suspicion, right? Because Most of the time the way in isn't somebody's weakness. It's their exhaustion or it's the way that they're not thinking about a particular thing.

Their guilt about not doing enough is enough to get them to do the thing that they're not supposed to do. The fact that they just want the help and they hand it off to somebody else. These are all great qualities, all of That's why it makes it feel so gross from the inside. So when people tell me this was advanced, nah. It was just patient. And it was human. And somehow that just makes this all worse. Because we have tools for advanced, but we don't have anything for patient. So what do you do about this?

Like how do you go about defending anything like this? Because nothing you can buy would have caught it. Full stop. Nothing on the market caught this in the wild. A human caught it by accident because his SSH felt slow. So you have to start somewhere uncomfortable, right? You have to start with the question almost nobody can answer, which is what am I running that's held up by one person? And you've all probably seen this comic. It's a comic where all the modern infrastructure is drawn as this gigantic tower. It's balanced on this tiny block, and it's got the text that some random person in Nebraska has been thanklessly maintaining this since 2003.

And frankly, like let's be honest, like everybody laughs, but here it the correlation is huge because it's not a joke. It's literally an asset inventory on what's going on. And in this particular case, XZ was that block. So what can you actually do? Do about this? Well, you gotta do some homework, the stuff that nobody likes doing. You've got to map the stuff you depend on down to actual humans and count how many are one bad week from just hanging it up, from just vacating their spot. That exact vacancy this guy filled by volunteering. You can see that risk once you actually bother to look for it.

And almost No one looks for it. Everything else after that is just really unglamorous work. Let's be super honest here. It's the standard stuff. It's watch the build and the release process, just not the source, make sure that there's reproducible builds, that there's provenance, and I mean this. We have to stop hanging our whole business on just one unpaid person that we've never even thanked before. We got lucky, embarrassing lucky. A half a second of lag and one engineer who wasn't even on the clock for this. That's the entire reason this near miss was instead of a normal day.

And within a week, my whole feed is vendors explaining how their product totally detects XE utilities. And let me get this straight. The magic box, the ones that the vendors are talking about, the ones that stops the most. Sophisticated supply chain attack ever seen, sat there quiet through this whole thing, right up until a human got annoyed that his SSH was slow, and then somehow learned to detect it? And detect it afterwards? Once we handed it the answer? That's not detection. That's a kid who copies your homework and asks why you're impressed. But that vendor noise, whatever.

That's Not what gets me. What gets me is the real lesson that was sitting right there after the catastrophic wreckage. And it's the least technical thing in the world. The entire planet is balanced on people we don't pay, we don't thank, and that we don't help. The bad guys here didn't beat that. They just used it. His way in was offering to help a guy that we all left drowning. The gap we made that was the exploit. And what'd we do about it? Well, we patched the one package, then we wrote these magnificent think pieces, we moved on.

We didn't fund anybody, we didn't change one thing that made this all possible. We just got a good story out of it. So my honest take It'll happen again. It happens again. It's almost certainly happening right now. Some other boring, critical little project, some other exhausted person, and some patient, friendly stranger, three commits deep into earning their trust. And when it pops, we'll call it sophisticated again. It'll be another supply chain thing. The vendors will Do their dance, and we'll get lucky again. Until we don't. So if you do one thing after this, don't make it buying something.

This is not about purchasing a product to fix a problem. Go figure out what your world actually rests on, what it sits on. Take the software you'd be dead without and trace it down to the human beings that are maintaining it. Find your guy in Nebraska and then do something that isn't nothing. Fund them. Mirror the project so you're not one deleted repo from a disaster. Contribute to the project. Put somebody on watching the project. Cut the dependency if you can't do anything else. Because the lesson of XZ isn't that the bad guys are clever.

It's that we all built this on a trust relationship and then never once invested in the people we were trusting. The attacker knew exactly what that was worth, and we're still the ones acting surprised.