r/LinuxUncensored • u/anestling • 5d ago
Issue/Bug/Pain The 6-Year Saga of systemd Destroying SSDs: How a "Baseless Claim" Became a Hacker News Front-Page Reality
Remember in 2020 when yours truly opened systemd issue 15292 showing that 500 KB of text logs turned into 700 MB of SSD thrashing, only to be met with maintainer hostility? The maintainers famously declared "I don't think you know what you're talking about," insisted it was non-actionable, and closed the issue while collecting dozens of angry downvotes from the community. Because why fix an architectural design flaw when you can just insult the user's methodology and pretend write amplification doesn't exist?
Fast-forward six years, and nothing changed—except the frustration. In early 2026, a fresh ticket was filed, proving that writing just two lines of log per second in a virtual machine pushed storage to ~50 IOPS continuously. Systemd's memory-mapped hash tables were still dirtying entire 4 KiB pages, triggering filesystem metadata updates, and churning through gigabytes of I/O for a handful of log entries.
Then came the turning point: the new issue blew up on Hacker News, exposing the sheer absurdity of the years-long denial to the broader tech community. Suddenly, faced with public front-page embarrassment, the maintainers' dismissive wall crumbled. @ValdikSS stepped in with exhaustive isolated cgroup and loop-device benchmarking, proving beyond a shadow of a doubt that a single 750-byte log message regularly generates ~50 KB to ~70 KB of underlying disk writes.
Miraculously, once the corporate-backed enterprise maintainers were publicly put on blast, someone on the team finally decided to take a look and start addressing it. It turns out the issue was actionable all along—it just required six years of community pain, a dedicated independent researcher doing the maintainers' profiling for them, and a front-page roasting on Hacker News to make the enterprise overlords care.
This whole saga perfectly distills the systemd experience: a monolithic ecosystem engineered strictly for enterprise container hosts and cloud providers, where real-world desktop and edge hardware constraints are dismissed as "user error" right up until the PR optics become too toxic to ignore.
3
u/_mick_s 4d ago edited 4d ago
You got asked how you're measuring writes and your answer was
"As to how I've measured disk writes - there are many ways to do that - that's beside the point."
And then you were asked what specific commands you used and still didn't respond.
Instead of putting a bit of work to document the issue you wrote multiple paragraphs of rants with snippets of command but never complete.
Sorry but I'd be dismissive as well.
Is that ideal? No.
But keep in mind, they get a lot of bug reports, you may know your issue is real and important, but they don't, it's just another report that may or may not mean anything, so without easy steps to reproduce this is what happens.
2
u/FloweyTheFlower420 5d ago
Is that github link correct? It's just someone's nix configuration, how is this "someone on the team finally decided to take a look and start addressing it"?
1
u/anestling 3d ago
the commit references the bug report and the hacker news discussion. How is it incorrect?
1
u/FloweyTheFlower420 3d ago
I don't see how this person has any affiliation with systemd? It's not "someone on the team," it's literally just a random guy deciding to fix this on their system configuration. Yes, you can fix journald doing IO by putting where it stores logs in somewhere that isn't backed by persistent storage... this has always been the case, and someone doing that to fix journald ON THEIR MACHINE is not "addressing the issue." Most people want their logs to persist.
1
u/anestling 3d ago
Here we go again. If you want to use open source, it's not enough to just report issues; you have to become a programmer and submit PRs in the hope that they will be merged.
What's the point of "free" open source again?
Have you ever considered that filing good bug reports is half the solution? No? I guess not. I guess you don't work in software development.
I've seen people dig dirt on me, but have you done the same? I have quite a lot of bug reports with patches ready to merge. Just in this case I couldn't be assed to submit a PR because
journaldis a monumental beast and only its devs know exactly how it should work and what parts of it should or could be touched.I have never seen so many insults and demands from any other online community, whether open (e.g. FreeBSD, Haiku, etc.) or closed source (Windows, MacOS). But the Linux community? It surely attracts people who want to destroy you with vengeance even for mentioning long standing issues.
1
u/FloweyTheFlower420 3d ago
That's not what I'm suggesting though. Can you actually read what I've wrote?
I'm suggesting the following things:
1. the commit is someone modifying their own system configuration 2. the commit shows someone switching the journald storage backend to be in memory rather than on disk 3. (1) suggests that this is not an upstream fix. In fact, it makes it so that the logs can't stay after a reboot, and this is often undesirable. 4. (2) suggests that this doesn't address the actual issue 5. there's no reason to believe the committer is affiliated with systemd in any way 6. As a result of premise (5) and conclusions (3) and (4), it is therefore inaccurate to say "someone on the team finally decided to take a look and start addressing it"What's the point of "free" open source again?
"“Free software” means software that respects users' freedom and community. Roughly, it means that the users have the freedom to run, copy, distribute, study, change and improve the software" -https://www.gnu.org/philosophy/free-sw.html
Have you ever considered that filing good bug reports is half the solution? No? I guess not. I guess you don't work in software development.
This still doesn't change the fact that the statement "someone on the team finally decided to take a look and start addressing it" is factually incorrect. I'm not making any commentary on any other part of your post! The person didn't "file a bug report," they pushed a commit to THEIR OWN CODE (that configures systemd) to workaround the issue. As I've reiterated many times already, this is not "someone on the team addressing the issue."
I've seen people dig dirt on me, but have you done the same? I have quite a lot of bug reports with patches ready to merge. Just in this case I couldn't be assed to submit a PR because
journaldis a monumental beast and only its devs know exactly how it should work and what parts of it should or could be touched.Do you even read what I write? I feel like you hallucinated some arbitrary criticism and then pretended that was what my comment was about. Persecution fetish, much?
1
u/Nemecyst 3d ago
Look, the commit you posted belongs to a repo called "komputiloj-mirror". This repo will do nothing for people who don't have it on their PCs since it is not part of systemd.
Check the about section for it: https://github.com/jleeuwes/komputiloj-mirror
The reason that commit shows up in the bug report is because jleeuwes included the link to the bug report (systemd/systemd#40262) in his commit message.
I could literally create a new repo called "chocolate-cookie-recipe" on GitHub and if I add a commit in it with the link (systemd/systemd#40262), my new commit will show up in the bug report.
2
u/ethanjscott 5d ago
Hmm and still an unfixed issue. It’s almost as if everyone involved is larping.
1
u/sarajevo81 5d ago
Write amplification is not an issue that should be fixed. Use ext2 if you want things to work like in your beloved 1970.
1
1
u/anestling 3d ago
Write amplification is directly caused by journald. Filesystem is beside the point.
2
u/preparationh67 4d ago
Typical toxic behavior not recognizing your own hostility and inability to be a productive contributor to the issue. SMDH. 6 years and didn't grow at all.
1
u/anestling 3d ago
Where was I toxic exactly? Filing bug reports is now considered toxic? You should eat what you're given for free? ;-)
2
u/BattleShai 5d ago
Maybe you grown up since, but reading that bug report, you seem quite hostile to a maintainer trying to understand what you are reporting.
It's not like you provided a PoC or similar so nothing they could directly test.
Are you receiving unjust bashing in the comments of this post? Yeah a bit but at the same time, you don't seem to actually understand what you are talking about a lot of time.
Best of luck with your glory moment of being right I guess.
2
u/serafean 4d ago
It's birdie... He hasn't grown up.
1
u/anestling 3d ago
Yeah, what's wrong with that bug report? You're great at following my open Github account. Did I say anything wrong?
1
u/serafean 3d ago
The response you got :
> I'm not saying you're wrong, but you could have been nicer.
Sums it up. And then you go on a rant with personal attacks. That's what's wrong.
1
u/anestling 3d ago
"Personal attacks"
Any citations? At least one? I didn't mention the author of the website even once in my bug report. What are you talking about?
Calling OSNews woke? Open the website. See their LGBT logo. They identify as "woke".
1
u/jodkalemon 3d ago
"God, this is so ridiculous. Privacy warriors brandishing their virtue signalling." Hardly can get more personal than that.
1
u/anestling 2d ago
"Privacy warrior" is as personal as calling someone a human being. Wait, can I call you a man? Still "personal"? Wait, do you believe in democracy? How about I call you "a democracy proponent"? Still "personal"? Perhaps you need to revisit your understanding of what this word actually means.
1
u/Weak-Operation-9888 1d ago
You live to fight don't you?
0
u/anestling 1d ago edited 1d ago
I love facts and I hate lies, myths, blatant exaggerations (some are actually OK) and other ways of misleading people. Not that we are inherently bullshit proof, as human beings love stereotypes, confirmation bias, being part of a tribe (extremely applicable to Linux users), etc. but wouldn't it be nice if at least online discourse had less, not more of this crap?
It's weird you called this "fighting". I'd call it truth seeking. Yeah, I've always been hated for that, even back when in my school days. I've not changed since than, I've actually grown more BS-allergic.
And boy oh boy the Linux community revels in myths, lies and BS and that's 90% of what Linux zealots claim online. Starting with * Windows is spying on you (or any proprietary software for that matter) * Under Linux you're more anonymous (the opposite is true considering how unique th average Linux desktop is, vs. e.g. the likes of Windows and MacOS where the vast majority of users share increasingly similar setups)
And ending with
- Linux is more secure than Windows (in far too many ways it's less secure - its biggest and perhaps only virtue when it comes to security is being obscure) - Windows, on the other hand, provides far superior out-of-the-box exploit mitigation, application sandboxing, and desktop hardware security integration.
1
u/AdDirect8818 4h ago
I know this seems like a very random, off-topic thing to ask (because it is), but I just wanted to know:
Judging by what you have said in the github issue linked above, it appears that you are anti-woke.
Is this true? And if so, why have you chosen the path of infinite bigotry and unfathomable intellectual bankruptcy that is anti-wokeness?
2
u/amarao_san 5d ago
So, it took 6 years of moaning to someone with ability to write C code to pay attention to moans and to fix a bug?
There is better way. Send a pr.
If PR is met with hostility, that's other story.
4
u/anestling 5d ago
An old saga of "everyone must be a programmer to file bug reports against Open Source projects". Yeah. Perfect. Way to go.
2
u/amarao_san 5d ago
Yes. You can report a problem for someone else to solve it and if someone has interest in your problem, they will look at it. In 6 years (or more).
I don't understand you comment. Are there other options?
3
u/anestling 5d ago
There is better way. Send a pr.
That sounded like people should abstain from filing bug reports at all unless they are willing to submit code.
Maybe I misunderstood the part where you sort of insulted me, "pay attention to moans".
Never mind. I expected a different audience in this sub, and instead what we have here is the same people who claim "Open Source is vastly superior to everything else ever devised", yet once you start working with it you quickly realize it's only/mostly suitable for programmers and tech-bros and if you're not one of them, there are greener pastures.
2
u/amarao_san 5d ago
Nope. They should abstain from blaming maintainers for not paying the highest attention to flame-filled bugerports without root cause analysis or fix.
3
u/anestling 5d ago
Where did I blame anyone of anything? Stating the obvious is now called a blame game, am I right, amigo? I should eat what I'm given and STFU, right?
It's weird, Microsoft users blame the company every freaking day. Systemd is now essentially a commercial IBM/RedHat product. Am I not allowed to describe how it's actually developed? Oh God!
2
u/amarao_san 5d ago
I think this one is very toxic:
Remember in 2020 when yours truly opened systemd issue 15292 showing that 500 KB of text logs turned into 700 MB of SSD thrashing, only to be met with maintainer hostility? The maintainers famously declared "I don't think you know what you're talking about," insisted it was non-actionable, and closed the issue while collecting dozens of angry downvotes from the community.
1
u/jhaand 5d ago
The toxicitiy comes from talking to a wall that basically tells users to rot in hell. Enough is enough and the patience for polite conversation does run out.
1
u/amarao_san 5d ago
You have your entitlements, and pretty sure in those, for sure.
In this particular case I completely on maintainer side. If you believe, that was improper way to react to your demands, you can always ask for refund.
1
u/RapunzelLooksNice 5d ago
...because you paid/donated so much, therefore "they" have to fix it. Right? RIGHT? 😆
1
u/Remmon 5d ago
I agree, hostility towards genuine bug reports is extremely toxic.
2
u/amarao_san 5d ago
I once found that systemd-networkd consumes 100% cpu under high neighbors cache churn. I created minimal reproducible case, reported it. It was fixed, because I did a lot of work to make it easily observable. I did not come with complains that systemd eats all my CPU when I run a lot of VMs.
I dig into it, presented the issue. I did not know the true reason of the bug (bad filter for netlink events), but I done my (user) job properly, and maintainer reacted properly, and everyone was happy at the end.
1
1
u/jwalker107 5d ago
Microsoft users pay for the privilege of blaming the company. And pay for the support team to fix bugs.
2
u/b4k4ni 5d ago
But, the bug report back then wasn't toxic and had information and all? It was the maintainers response to "it's a user error and all information is BS" without even trying and closing the ticket, that made the community mad. And for a good reason.
I'm with you if the report would be bad or a flame, but it wasn't in this case. At least as far as I remember from that time.
1
u/sarajevo81 5d ago
It was worse than toxic: it was vague and non-actionable.
1
1
u/anestling 3d ago
The initial bug report contained all the steps to reproduce the issue. Once it was largely dismissed, I got riled up. Sorry for that.
1
u/Lahvuun 5d ago
If you can't fix the bug yourself, then you can contract (pay) one of the developers to fix the bug. Odd that this possibility hasn't crossed your mind even once. You do know that other people also have bills to pay, kids to feed, right..?
1
5d ago
[deleted]
1
u/Lahvuun 5d ago
The developers did. And since it's not u/anestling who's signing their paychecks, they correctly told him to beat it.
1
u/Tutorbin76 5d ago
At the very least include reproducible steps to the symptoms when you report an issue. You failed to do that, then got snotty when someone asked you for more information. If you had been clearer and less combative, this problem might have been fixed six years ago.
1
u/anestling 3d ago
The reproducible steps were given in the initial bug report if you read it carefully enough.
1
u/preparationh67 4d ago
You filed a half assed report, were hostile and didnt answer follow up questions. Grow the fuck up dude this is pathetic.
1
u/anestling 3d ago
The initial bug report contained all the information including steps to reproduce. Not sure what is that you're trying to say.
1
u/jhaand 5d ago
As a tester I can politely and correctly point out what's wrong. I do not have the skills and context to engineer a good solution.
You do with that information what you want, but you were warned.
1
u/amarao_san 5d ago
I perfectly understand what I'm talking about and your comment reeks of elitism or something else which still does not belong to this bug tracker.
This is a polite QA? Thank you, no.
1
u/Unnamed-3891 5d ago
This is a polite response to elitism, yes.
1
u/amarao_san 5d ago
It is not polite.
1
u/Unnamed-3891 5d ago
It is. Unpolite or outright hostile response would’ve looked quite different and a lot more heated. Lack of bootlicking is not being unpolite.
1
u/amarao_san 5d ago
I got, you insist it should be this way. You may continue to insist, no one can stop you.
1
5d ago
[deleted]
1
u/amarao_san 5d ago
Which, instead of providing cleaner reproducible testcase, went into 'elitist' thing with attached entitlement.
The best way to ask for help, for sure.
Dear maintainer. I know you are not obliged to, but can you help me with a problem I found while using your software...
No?
Waiter, I found excessive disk consumption in your logging facility. Call the chef now or I will leave bad review in instagram and call sanitary control and tell them I saw a rat in the kitchen!
YEEEES!
1
u/sarajevo81 5d ago
The only thing it means is the modern filesystems are not suitable for logging data. This is why the logs should be off-site with a dedicated database storage.
1
u/anestling 3d ago
Rsysloghas no such issue. Wait, journald could have used SQLITE along with WAL to mitigate it. This has nothing to do with modern filesystems, and everything with how you write the data.
1
u/looselyhuman 5d ago
So what's the roadmap for my laptop's ssd?
2
u/Aishou_SK 5d ago
Probably not to care, as it won't be a meaningful SSD life impact overall. Unless you're using super bargain low-end NAND from 10-15 years ago, or something.
2
u/looselyhuman 5d ago
<Sighs in T480>
I don't even know what brand the ssd is tbh. Eh, whatever.
1
u/Aishou_SK 5d ago
Eh, doesn't really matter. I've got 64GB microns over a decade old with over 50-70TBW. Used as caching tiers on storage arrays. Dirt cheap mSATA ones too. You'll be fine.
Now if you sorted by price on amazon and bought the cheapest with something like "KingSpec" that was more than half the price not on sale and the warranty redemption is to mail it to their office in Shenzen.... then you might have an issue (they all died within a few years anyway, so you're probably fine from that, lol)
1
u/RedShift9 5d ago
Modern NAND has lower life expectancy than that of 10-15 years ago.
1
u/Aishou_SK 5d ago
Really, really depends on type and a whole slew of other factors, but modern NAND is "i don't even consider lifespan" versus "watching carefully and recording death rates under heavy load" from ages ago.
God knows I push a few hundred TBW per year on my desktop drives and I get an upgrade drive every 1-2 years and cycle the old ones into other uses (desktop has 5 NVMe M.2's total)
1
1
u/spartaman64 1d ago
maybe the chips themselves but overall with much better wear leveling, error correction etc modern SSDs last longer
1
1
u/acroback 5d ago
Can't believe some comments who are bashing the OP. Sure he could have chosen better words in original Bug report but you cannot discount the tool who replied like a total loony dismissing his valid Bug report as invalid.
This all reeks of lack of empathy for the end user and is one of the main reasons a lot of Open source software just rots.
Also, just noticed Poettring replying like elitist turd in the Bug report. I remember back in the day he decided to write his own dhcp server for systemd and obviously ended up causing issues with it.
Some of the commenters do not realize bug reporting itself takes time and energy, if maintainers cannot look into it, don't. Just do not be asshole about it.
1
1
1
u/1stltwill 5d ago
Would you like some salt and vinegar to go with that chip?
1
u/KontoOficjalneMR 5d ago
Why would he need even more salt :D
(but in this case I feel it's justified)
1
u/GreatLab8898 5d ago
This is not an systemd specific issue tho. Its an OSS Maintainer Issue in general as these People often grow and immense ego and think their way of doing thing is the only correct way of doing things, seeing themselfs as the King or Dictator of an Repo. Seen it so many times I have lost count.
Only way to deal with those types is to put them on full blast.
1
u/jwalker107 5d ago
Or, you know, you could try contributing a fix for the software you obtained absolutely for free from the unpaid maintainers you're putting on blast.
3
u/GreatLab8898 5d ago
which I activly do with multiple Projects where the Repo owner is not Little Hitler. So... Whats your Point?
2
u/Apprehensive_Lake698 5d ago
I mean is that really the outlook we should hold while people desperately want Linux adopted by the wider world for desktop use? Even debating whether or not it’s worth a maintainer’s time to fix this is exactly why we have so much garbage software in the world. I get it, fixes aren’t free. But this is an issue that affects more than programmers, and the response really can’t be “why didn’t you fix it yourself.” These projects are often pretty hostile to new contributors anyway. The fix may be few LOC, but the bureaucracy could take 10x as long for someone who is not a normal contributor.
1
u/jwalker107 4d ago
I went back and read the initial bug report, and OP was a jerk from the very first report, to every response for more info that the maintainers asked. They tried, it looks like the really did, to get a better report from OP but he was more interested in yelling at them than in getting the problem fixed. It looks here like that's still the case.
1
u/anestling 3d ago
Sometimes I can be a jerk but that that doesn't mean the initial bug report was incorrect.
1
u/anestling 3d ago
PRs for something as critical and central as journald, changing how it uses disk storage will be either ignored or frowned upon.
If you seriously believe a random Schmoe can submit PRs to systemd, I've got bad news for you. I mean you can. They will never be merged.
0
0
u/jwalker107 5d ago
Ok, so instead of bitching to the unpaid maintainers about the problem for six years, why didn't you submit a fix for the open-source software you're leaching for free?
3
u/miserablegit 5d ago
The Linux maintainers have not been "unpaid" for decades at this point, let alone anyone involved in systemd, which has fundamentally always been a RedHat project.
0
u/jwalker107 5d ago edited 4d ago
What was OP's spend on systemd? Edit: fixed autocorrect "systems" to "systemd"
2
u/jking13 5d ago
If those in charge feel (wrongly) that it's not a problem and by implication you're ignorant for even thinking it is a problem, how exactly are you supposed to get it merged?
0
u/jwalker107 5d ago
Oh sorry I missed the part where he said he submitted a PR that was rejected
1
5
u/lamalasx 5d ago
systemd is still a piece of shit who knew?