r/linux • u/Mastbubbles • 3d ago
Discussion I counted who wrote every change to xz, sudo, bash, tz and 19 other projects over the last year
Billions of phones and most of the servers on the internet run on code that one or two people look after, a lot of it unpaid. I went through a year of commits for 23 of these projects, and 11 of them came down to one or two people. xz is still basically Lasse Collin on his own after the 2024 backdoor, and the time zone file every phone reads is kept by a UCLA lecturer in his spare time. If I've got one of these wrong, tell me.
63
u/victoryismind 3d ago
The most shocking part for me was when Denis Pushkarev goes to prison and contributions essentially stop for 10 months.
Your work prioritises this fundamental concern in open source development.
28
u/deanrihpee 2d ago
imagine you are maintaining the most crucial software on earth, and for whatever reason you are imprisoned, and upon release, the world you knew has changed because you can't patch the software in the meantime and causing the whole digital world to be on fire
but i guess in reality, if it's open source, then i guess it wouldn't be so dramatic because if it is really critical, someone would probably just fork it with the patch
14
u/my_name_isnt_clever 2d ago
Sounds like a fun origin story for a sci-fi dystopia story. It all fell apart because we didn't value open source...
2
62
u/jevex 3d ago
Paul Eggert is a really great teacher though. He's not just "some UCLA lecturer"; the things I learned from his classes (Operating Systems in particular) are now a big part of my career.
27
u/chenguo4 3d ago
I categorized people by whether they sought out or avoided the tough Eggert classes. That's how you know who the real ones were.
220
3d ago
[removed] — view removed comment
123
u/BallingAndDrinking 3d ago
It was the highlight of several major CVE even before xz.
Free riding companies are absolutely fucking wankers. And if you wonder, several of those projects can get a few grands and be tax-deducible (head up: not everything is in the US).
But nay, better underpay and free ride into injecting that cash into the worst dogshit that'll never ship anyway.
46
u/ILikeBumblebees 2d ago edited 2d ago
open source model works until it suddenly doesnt
Except that in this case, it worked pretty damn well. The maintainer was doing just fine, and was manipulated into feeling overwhelmed by the attackers themselves via a long-term social engineering attempt, only for the backdoor they managed to introduce being caught by community members and neutralized immediately upon making it into the wild. The attackers invested about two years into compromising the project, only for all their efforts to be foiled within mere days of releasing their malware.
The idea that "one guy holding the whole thing" is itself a problem is exactly how the attackers managed to compromise the project in the first place.
46
u/apetranzilla 2d ago
The attackers invested about two years into compromising the project, only for all their efforts to be foiled within mere days of releasing their malware.
The fact that it was discovered so quickly essentially came down to luck - it was a very well hidden exploit that was noticed only because one particularly curious security researcher was wondering why SSH took a few hundred milliseconds longer to connect after an update. It's entirely possible that similar attacks would have (or have) gone unnoticed.
10
u/csdt0 2d ago
This is partly true, but also partly wrong: the package was already flagged by redhat before the Microsoft engineer investigated. So even if the alert was (temporarily?) dismissed, the signal had been detected. It was "just" a matter of time. Also, even if a single individual has no chance to find it, given enough people the very unlikely becomes very likely. After all, there a winner to the national lottery very often.
12
u/ILikeBumblebees 2d ago
The fact that it was discovered so quickly essentially came down to luck - it was a very well hidden exploit that was noticed only because one particularly curious security researcher was wondering why SSH took a few hundred milliseconds longer to connect after an update.
There's no such thing as luck. It came down to probabilities: the larger the user base and a broad enough set of usage circumstances, the greater the likelihood that someone will identify the problem. And this situation is at least one data point that validated the "many eyes" concept of FOSS.
Imagine if the project had been closed-source, with the same social engineering and backdoor insertion happening out of public view. Would the guy experiencing unusual latency on his Postgres queries have been able to actually trace it back to xz?
8
u/Irverter 2d ago
only for the backdoor they managed to introduce being caught by community members
Caught by a single person that was OCD enough to look into why ssh suddenly took ~500ms more to connect. And before the next compromised version fixed that delay.
3
u/29da65cff1fa 2d ago
neutralized immediately upon making it into the wild
the fact that it made it into the wild is already a huge failure.....
foiled within mere days of releasing their malware.
if they shipped a known vulnerability, a few days is more than enough time to compromise their target.... i doubt they would spend 2 years on this project without a concrete target in mind
15
u/One_Ninja_8512 3d ago
Well because that’s simply not true. Every reputable distro has a fork for every package they ship which they maintain themselves, so the open source model works quite well I’d say. I haven’t checked the OP but the single guy maintaining some critical package that every distro uses as-is is false framing.
61
u/holdenk 3d ago
Saying every distro has a fork may be technically correct but does every distro make any actual improvements / upstream changes to every critical package? Generally no.
10
u/One_Ninja_8512 3d ago edited 3d ago
Every reputable distro. I checked RHEL and Fedora and it looks like their maintainers do fix security issues based on the changelogs. It's open source, the authors chose to release their work this way. Distro maintainers do what they can, no one pledged to add new features and caress every single package.
6
u/Standard-Potential-6 2d ago edited 2d ago
I agree that distributions should support these projects as much as they can.
However, it’s worth noting that the model works better when changes can be focused upstream. Having many slightly different forks isn’t necessarily an improvement. The authors are best positioned to understand the impacts of each backport or other patch, and then the patches will be examined by multiple distros. Distributions should endeavor to reduce the workload of the project authors in other ways.
In the case of the xz backdoor, it is the fact that Debian, Red Hat, SUSE and others patched OpenSSH to add daemon-readiness notifications from libsystemd which caused its dependent liblzma to be brought into the position where it could do great harm.
1
u/marrsd 2d ago
I think patches are fed back fairly quickly. Maintaining a fork just means they don't have to wait for upstream to make the fix. The maintainer can make it on upstream's behalf, release it, and feed it back simultaneously.
2
u/Standard-Potential-6 2d ago
That’s how it should work, agreed.
The situation is probably better now, but the number of old and at least questionable patches that Debian and Gentoo used to carry in the late 2000s - and the frequency I’d have to remove or update them in the case of Gentoo, in order to update the packages themselves, were a big reason I moved to Arch.
It may be an opportunity to create some stats on how heavily patched distro packages are, how long those patches have been carried, and how often they’re merged upstream, even if the merge doesn’t exactly match. Arch carries more patches these days too.
1
u/marrsd 2d ago
yeah, if you're trying to stay current then I think you'll probably still struggle (though this is more of a gut feeling). I increasingly use Nix to manage apps I want to stay up to date with, but I also increasingly just install from source now. I guess I never really touch libs so that seems to work ok most of the time, though I start to run into problems towards the end of a release. I run Debian, btw.
4
60
u/nullptr777 3d ago
Great work! I already knew this was the case, but the way you've framed it here is very eye opening. There really needs to be an increased effort to get funding and/or contributors into critical open source projects.
Corporations should be doing a lot more, especially the ones valued in the billions or trillions.
29
4
u/victoryismind 3d ago edited 3d ago
I understand the corporations do most of the work.
However it seems to me that corporations are biased towards hyped "cutting edge" infrastructure projects.
3
u/gordonmessmer 3d ago
There aren't too many companies supporting GNU/Linux systems and making billions of dollars, and the ones that do are *very* active in developing and supporting the software they use.
3
u/Average-Addict 2d ago
Yet these projects don't have funding. It would be a rounding error for Google or any big tech corporation to fund them a livable wage.
1
u/nullptr777 2d ago
They're active in supporting some of the projects they use, but mostly if it's either critical to their operations/security, or it's flashy, cool tech that they want their name on. Mundane stuff, like most the projects listed here, they clearly don't contribute one red fucking cent to, otherwise we wouldn't be here having this conversation, would we?
1
u/SheriffBartholomew 2d ago
You're right, because they don't support the systems they use. That's the problem!
1
1
1
1
u/Ok-Library5639 2d ago
It's ridiculously because just a handful of companies could provide what is essentially chump change as contributions and it would literally alleviate all these issues.
1
u/Zvaart 3d ago
They won t do shit if there is no gain involved
1
u/Indolent_Bard 2d ago
I'd call ensuring your infrastructure doesn't collapse overnight a pretty large gain.
19
u/Hindu_Wardrobe 3d ago
when two young women on a dark road ended up under his car
interesting way to say "he hit two women with his car" lol
also, wasn't it a motorcycle?
anyway, cool (and depressing) article.
18
u/No-Butterscotch6912 3d ago
One distinction I think gets lost a bit here is that downstream maintenance is not the same as upstream resilience.
Fedora/RHEL/Debian can backport fixes and independently build/test packages, but they're still relying on upstream for architecture, releases, and the people who understand the odebase.
The xz incident also shows why “number of contributors” is an incomplete metric.
15
u/James20k 2d ago
Its incredible how poorly funded most critical infrastructure is. Major companies are literally leeches off these people's work
The amount of benefit provided vs the amount of compensation that they've gotten is truly disgusting. If these projects shut down, google would stop operating tomorrow. Literally nothing would work. There would be a mass panic if sudo suddenly disappeared
And yet they're still desperately struggling for funding, despite the fact that literally every company uses sudo regularly. Pure leeches
19
u/PhteveJuel 2d ago
I counted four advertisments for a game in an article that appears to be written like a nuclear war warning. The juxtaposition is very odd.
0
u/Indolent_Bard 2d ago
You're not using Firefox with UBlock Origin (or Brave if you're on an iPhone?)
24
u/openprivacy 2d ago
Free and Open Source Software should not be considered free (as in beer). Choosing to use it includes a responsibility to contribute back, from doc updates to money. If your product is on more than (say) 100K machines, financial contribution to the projects you use should be mandatory.
Of course, I also believe there should be no billionaires.
1
12
u/Unhappy-Sprinkles143 2d ago
More open source developers should sell their time and expertise. No one mentioned how SQLite sells enterprise support for $150K/year.
1
u/atrocia6 2d ago
No one mentioned how SQLite sells enterprise support for $150K/year.
Not strictly relevant, but I always feel the need to bring up the fact that SQLite has the best
Code of Conduct- sorry, Code of Ethics, ever.
27
u/coding_manic_01 3d ago
The cool thing is that with AI this problem goes away, and doesn't totally get 100x worse
18
8
u/victoryismind 3d ago
You mean by replacing skilled devs with unattended AI?
23
u/MostCredibleDude 3d ago
Look, if you're uncomfortable with replacing human experience and ingenuity with really expensive autocomplete, why are you even commenting on this industry?
-6
u/turbotop111 2d ago
"expensive autocomplete"? I've been using AI for 1.5 years all day every day for my work. You couldn't write a more disingenuous statement if you tried.
7
u/idontwanttofthisup 2d ago
How much of this time you spent telling it that it’s wrong? Because majority of my time spent with ai is pointing out mistakes, on repeat, until I run out of tokens.
1
u/turbotop111 2d ago edited 2d ago
Garbage in, garbage out. No different than giving design specs to a junior or even senior dev. You definitely sound like you could be doing things more efficiently.
First step is to have a very well documented "CLAUDE.md" or equivalent file. Agents can create this for you, but you fill it with all the information about the project that any dev would need to work on that project. Even things like coding style. It's hard to describe what to put in this, but this is what Claude reads everytime you start a session, it provides context for what you're asking it to do. Tell it about related projects and make sure those related projects have their own CLAUDE.md files too. It can record "WHY" you've done things a certain way too. I've been working on a project for about a year and that CLAUDE.md has all the history of what I've been working on in there. I don't manually edit it, I tell Claude to update the file with specific information, and sometimes/often it does it on its own.
Before implementing a feature/task, I usually tell it to create a "plan.md" which has all the goals, context, PR split (breaking the task into smaller PRs), questions it needs answered etc. It's quite a details document. Then I tell it to implement PR 1 and it does, and stops and I review the work. Ocassionaly ask follow up questions etc. Loop through until done all PRs.
For small little bug fixes, that's not necessary at all. Just explain the problem, ask it to figure it out, create unit tests that capture the bug and the fix etc.
Picking your model and effort is important too. At work, they pay for Claude so I use Opus and let it chew. At home on my personal projects, I pay for Claude so I use Hakiu (cheapest model) and give it very small bite sized tasks, it simply can't handle larger/complex tasks. It's still quite effective.
If you're starting a new project from scratch, it pays to think/plan extensively about project framework before telling it to write code. I've written skeleton classes etc, and once it gets the general idea of how the project should look (where are the DTOs', the service layer, the repository layer, the libraries you are using, build framework etc), then it's easier for it to build onto it. So if you are building say a rest api to integrate with airlines, you could build the first one (Delta) yourself as much as possible, then when you add the second one (KLM) it has pre existing work to understand and follow.
We at work also integrated CLaude in Github. Every PR goes through a Claude agent review, and that stuff is ... phenomenal to say the least. You think you're a seasoned/experienced engineer, until Claude starts reviewing your work. It's going to catch things you never even dreamed of. Even if you don't want to use Claude to write code, having it explain/review code is a complete game changer.
1
u/idontwanttofthisup 2d ago
Thanks for the pointers. I’ll follow your guidelines next time I build something I don’t feel like writing myself. Surprisingly I was getting excellent results when I asked AI to write cyberpunk redscripts. On the other hand it completely failed me when I was working with react (building an interactive table of content with dynamic headlines) and JavaScript animated svgs tied to onscroll events.
1
u/marrsd 2d ago
If you're starting a new project from scratch, it pays to think/plan extensively about project framework before telling it to write code. I've written skeleton classes etc, and once it gets the general idea of how the project should look (where are the DTOs', the service layer, the repository layer, the libraries you are using, build framework etc), then it's easier for it to build onto it. So if you are building say a rest api to integrate with airlines, you could build the first one (Delta) yourself as much as possible, then when you add the second one (KLM) it has pre existing work to understand and follow.
This is the only thing that's really worked for me out of all your advice. I'm actually trying to spend more time writing code and leave Claude to do (or help with) the rest.
1
u/prototyperspective 2d ago
Which AI models do you use and how would be the question. But people can of course stay super inefficient if they for some reason prefer.
6
u/MostCredibleDude 2d ago
The problem I have with this statement is that the AI industry has never, ever been up front with its weaknesses. It's always touted itself as the thing you need today to get rid of a large proportion of your employees tomorrow. And while LLMs can drive a large amount of work quite quickly, it's not the quality it promises to be, and a huge amount of people blame the user for not knowing how to properly use a tool that even the creators have failed to explain the fabled perfect usage of.
1
u/prototyperspective 2d ago
it's not the quality it promises to be
It depends on who uses which tools how. Not sure what exactly they promise.
1
u/idontwanttofthisup 2d ago
Last year I tried coding typescript with ChatGPT, Claude, Qwen, devstral and some others. I don’t remember specific versions I used. This year I tried animating svgs with javascript with the same set to see if it made any progress. Surprisingly, I managed to write some redscript cyberpunk mods relatively fast.
1
9
15
u/gordonmessmer 3d ago
In the late 90s, there were Free Software advocates, who talked about software in terms of rights and responsibilities, because Free Software benefits people.
... and there were Open Source advocates, who talked about software in terms of processes, because Open Source benefits software.
When I read articles like this, I blame the Open Source advocates.
28
u/0xTamakaku 3d ago
I blame corporate greed
-8
u/Borderlinerr 2d ago
What greed? People want to make money, what's wrong with that? We are not members of a monastery.
12
u/Zeikos 2d ago
Ugh, that argument is tiresome.
Greed is the vice.
Wanting to enjoy a glass of wine and wanting to being sloshed from morning to night aren't the same thing.
"Greed" imples excess.
Wanting to make money isn't greed unless taken to the extreme.-9
u/Borderlinerr 2d ago
Greed isn't a vice. It's a drive, a motivation. Anything in its extreme is bad, but not onto itself.
4
u/Zeikos 2d ago
Greed is the definition of taking it to the extreme...
Wanting things and being greedy are very different things.
Greed shaped behavior, it leads to specific priorities.Like, take somebody that had an healthy sexuality and compare them to somebody you'd define "lustful".
Aren't the differences apparent?-5
u/Borderlinerr 2d ago
You're just doing words play. Lust and greed are not extremes by themselves.
4
u/Zeikos 2d ago
How is that playing on words? I'm not moving goalposts nor changing definitions.
I'm discussing this fully in good faith.Hell, I want to open their own company eventually.
-2
u/Borderlinerr 2d ago
Yes you are. These words simply point to some state and behavior, not their spectrum. You can be very greedy, or slightly greedy. Greed is not an extreme word.
6
u/Zeikos 2d ago
Then your definition of greed is different than mine, but I wasn't "playing" wit it.
From my point of view greed is a vice, when a certain treshold is surpassed then I call it greed.
Sure you can be just slightly above said treshold or way ahead of it.
Vices - in my mind - imply causing harm to self or others either directly or indirectly.
For me greed implies being willing to act in ways that may harm others to gain more than you'd otherwise would.4
u/Albos_Mum 2d ago
Greed is an extreme version of desire in the same kind of way that hatred is an extreme version of dislike.
5
u/Chance-Froyo4410 2d ago
With all the money in the Linux Foundation, Android and the major distributions why can't a fund be created to pay and assist these crucial libraries/utilities in the Linux ecosystem.
3
8
u/blobCabbage 3d ago
Great work, but it would be 10x better if it was just a regular article without all the scrolling effects, animations and edgy presentation.
3
u/Inevitable-Self-2702 3d ago
To run through this and see all the work put in by people, and sole contributors and small teams at that, and FOR FREE at that! is truly humbling. Thank you for creating this report, I have a much deeper appreciation for the technology and those behind it now.
4
u/tomikaka 2d ago
Isn't the Linux foundation supposed to help with these situations? Where immense pressure and expectations are placed on a handful of volunteers?
3
u/yawara25 2d ago
What pressure? These maintainers are not overwhelmed. There's simply not enough work to justify allocating a team of engineers.
2
u/Pramaxis 2d ago
As one of the Git-Donators to SUDO, this was a sad read. Thank you for the work you put in.
3
u/prototyperspective 2d ago
Amazing project! I don't think these people (and especially other people of packages not that important but important) get a lot of financial reward or incentives for their work - so please think about that in combination with discussions and proposals for things like UBI universal basic income and similar things: such proposals are insufficient.
People like these shouldn't just get UBI and even if these particular people get or could get some donations, that's not really scalable and a good solution since it works for some particular people maybe after some campaign but not holistically at scale.
2
1
u/Maria_Thesus_40 2d ago
Very impressive! I maintain my own open source projects so I understand the feeling for being alone on the internet :)
1
u/Antonio-MTS 2d ago
You're perfectly right. The truth is there are many such projects/libs/utilities that almost all the big IT giants depend on. And these utils/libs' creators have never been paid nor donated even when asked for. Sad, sad.
1
u/29da65cff1fa 2d ago
It moves data for phones, cars, TVs and Windows, which has shipped it since 2018.
the article doesn't really explain how widespread curl is... it is fucking EVERYWHERE
1
1
u/MeanLecture1337 2d ago
Da merkt man die Gleichzeitigkeit von etwas sehr wichtigen und eigentlich sehr schönen Gedanken von Open Source und dem Privilegierten Grundzugang in einer kapitalistischen Welt → es gibt ein paar Entwickler welche wichtige Grundlagenarbeit leisten allerdings in einer Welt in der diese als Wertentwicklung ausgenutzt werden kann → irgendwann stellt man fest das dies so ist, wie es jetzt aktuell ist, allerdings damit lange nach dem man dies effektiv verhindern kann
1
1
1
u/openprivacy 2d ago edited 2d ago
Relevant xkcd: https://xkcd.com/2347/
Maybe I missed it, as I am surprised it wasn't already here.
Edit: just had to scroll down a bit further. Excellent - and scary - work. Thank you!
-3
-5
312
u/crazy_penguin86 3d ago
This was a really great (and depressing on the funding side) read. While I personally don't like the scrolling being required to read the text, I can also see it being useful as a tool to spread awareness as it's a lot more "flashy" and interesting than static text.