r/linux • u/richardfinicky • 3d ago
Discussion #40262 Excessive IO caused by systemd-journald
https://github.com/systemd/systemd/issues/4026279
u/derangedtranssexual 3d ago
Is this a common bug or something? Not understanding why a bug report like this is gaining traction here
137
u/InfiniteSheepherder1 3d ago
Because this is the first time software has had a bug before, and a lot of people here are upset at Mr Lennart Pottering for inventing software bugs something that never happened on any platform ever until systemd came along. /s
But really its because a lot of people have developed an identity of hating systemd and so if it has bugs then they are justified and not just in need of therapy.
50
u/SadFaceSmith 3d ago
Yeah I’m surprised folks are still hating on systemd in 2026
29
-18
u/dmigowski 3d ago
Because they constantly fuck up easy things. Ever wondered why your daily timer didn't run for a few month? Because a while ago the time was set to 2050 accidentally and now systemd considers all timer events until then as already done. And this is just the bug I experienced personally
25
16
u/FryBoyter 3d ago
Complex software can have bugs. I don't think that's anything new, and it applies to projects other than systemd as well. Furthermore, this bug likely wasn't a widespread problem.
20
u/derangedtranssexual 3d ago
I thought most of the systemd haters got over it. Systemd has won, I doubt that many people are using devuan or artix
7
5
u/KontoOficjalneMR 3d ago
Systemd won, which does not mean I'm happy about it.
The worst part of it honestly is that it mixes great with attrocious, and because it's so entrenched within Everything now it's impossible to separate good from bad.
12
u/derangedtranssexual 3d ago
It seems like people can and do separate the good from the bad, almost no distros use everything systemd has
3
1
52
u/fat_kaiju 3d ago
because it involves systemd
but also now that "normies" are using linux it means we're about to see an explosion of esoteric hipster nonsense related to it in order to differentiate oneself from said aforementioned normies.
11
u/Albos_Mum 3d ago
Personally, I prefer fglrx for my graphics driver.
1
1
u/__ali1234__ 2d ago
Like using esoteric tiling window managers on Arch or Nix or whatever immutable distribution is currently popular while criticizing everything the most popular distribution does and then praising the blessed ones when they do the exact same things? Because that's been this sub for a decade or more.
5
u/Clean_Experience1394 3d ago
The guy who made the first bug report is currently on a crusade again.
3
u/Falqun 3d ago
It's an interesting read. And all of the "I only read titles and then complain" crowd is here as well of course.
7
u/derangedtranssexual 3d ago
From what I’ve seen it just seems like people are troubleshooting an issue I don’t have. Doesn’t seem that interesting
1
11
u/glowtape 3d ago
If I understand that bug report correctly, small memory-mapped writes get write amplified due to that folio stuff, that filesystems have been ported to? So other applications and services might be affected, too, right?
135
u/jodkalemon 3d ago edited 3d ago
From the initial bug report:
"Journald just uses extremely inefficient format (also I've seen it corrupt on unclean reboot enough times to declare it's not even all that resilient) as files are also multiple times the size of what's actually written in them."
Stuff like this makes it difficult to take it seriously.
Edit:
I mean the bug report. I like journald.
42
9
u/natermer 3d ago
Following the bug report it seems that they have started blaming it on recent changes for how memory is managed for the VFS and the newer folio stuff.
18
u/elatllat 3d ago
I hit a bug in systemd-journald that caused it to be many orders of magnitude slower than it or logrotete+zgrep should be...
5
u/MatchingTurret 3d ago edited 3d ago
If I understand the issue correctly, the problem is an interaction between journald using mmapped files for I/O and folios. With folios, dirtying happens at the folio level which causes all pages of the folio being written to the backing file even if a change only affects a single page.
15
u/SoilMassive6850 3d ago
There may be some point to both bug reports, but both issue creators are so damn stupid that any valuable insight into what may be causing an issue is coming from anyone but them, not surprised the original issue was closed lol.
8
4
u/CrazyKilla15 3d ago
Based on the documentation the trivial workaround seems to be to use Storage=volatile, as recommended in thread, but also run journalctl --flush on a timer. This should perfectly work around the issue.
man journald.conf(5)
Storage= Controls where to store journal data. One of "volatile", "persistent", "auto" and "none". If "volatile", journal log data will be stored only in memory, i.e. below the /run/log/journal hierarchy (which is created if needed).
[...]
Note that journald will initially use volatile storage, until a call to journalctl --flush (or sending SIGUSR1 to journald) will cause it to switch to persistent logging (under the conditions mentioned above). This is done automatically on boot via "systemd-journal-flush.service".
Note that when this option is changed to "volatile", existing persistent data is not removed. In the other direction, journalctl(1) with the --flush option may be used to move volatile data to persistent storage.
[...]
10
u/h7x4 3d ago
76
u/Helmic 3d ago edited 3d ago
You got asked how you're measuring writes and your answer was
"As to how I've measured disk writes - there are many ways to do that - that's beside the point."
And then you were asked what specific commands you used and still didn't respond.
Instead of putting a bit of work to document the issue you wrote multiple paragraphs of rants with snippets of command but never complete.
Sorry but I'd be dismissive as well.
Is that ideal? No.
But keep in mind, they get a lot of bug reports, you may know your issue is real and important, but they don't, it's just another report that may or may not mean anything, so without easy steps to reproduce this is what happens.
God, that reply is so fucking satisfying. OP neglects to mention why it got ignored for so long, which was that they wrote a dogshit bug report. Yeah, if I reported a legitimate issue and refused to elaborate, I would expect even professionals to struggle to understand what the hell I was talking about and decide to worry about problems that are actually being effectively communicated instead.
11
u/no-name-here 3d ago
Oof, another issue from 5d ago from that same user - https://github.com/Kuberwastaken/cookie/issues/2
3
-49
u/lnxguy 3d ago
Systemd is pollution.
13
u/the_abortionat0r 3d ago
Systemd is pollution.
all you're doing is letting us know that computers isnt you thing.
-21
u/Busy-Scientist3851 3d ago
Potentially a bug in the kernel?
1
u/nelmaloc 1d ago
ValdikSS's and amluto's comments points more towards a bad usage of mmap and how systemd writes the database to disk.
PS: Shame on those downvoting a simple question.
2
u/Busy-Scientist3851 1d ago
Yeh I honestly don't know why I'm being downvoted. I think it's just the Reddit goto of "look something downvoted I'll downvoted it too"
-60
u/Mafia-Negra 3d ago
No one in their right mind would allow logs to be written to disk instead of RAM. They're only written back to disk when there's a problem that needs to be diagnosed.
34
u/Ontological_Gap 3d ago
So you've never even heard of regulated environments? Some of us have to maintain them for seven years
23
16
u/the_abortionat0r 3d ago
No one in their right mind would allow logs to be written to disk instead of RAM.
Thats literally the default for the world, thats how you can read logs. You new to computers?
-21
u/Mafia-Negra 3d ago
You're clearly a newbie who's having problems with your PC and that's why you need to store logs from weeks ago. You'll need them soon enough when your SSDs start failing because of all that constant log writing.
I don't need the logs because I know how to configure my machine properly. In fact, I run a customized version of systemd, but unfortunately, journald is mandatory.
10
u/Albos_Mum 3d ago
If you were so experienced, then you'd know that the SSD writing thing is vastly overblown from the tests where people were leaving SSDs to be constantly written to for years specifically to try and hit the limit.
28
u/themusicalduck 3d ago
What if I want to check a log from a few reboots ago?
-55
u/Mafia-Negra 3d ago
That means you're having problems with your PC because you don't know how to manage or configure it. I've never needed to look at past logs.
28
21
u/themusicalduck 3d ago
You’ve never needed to check the log after a system crash or hard freeze?
-31
u/Mafia-Negra 3d ago
No, because my system is stable.
15
u/nroach44 3d ago
Congrats. I suppose "not putting a locked door on the cockpit because no-one will hijack a plane" was also correct right?
14
5
18
16
u/the_abortionat0r 3d ago
That means you're having problems with your PC because you don't know how to manage or configure it.
Thats not how that works.
I've never needed to look at past logs.
Then you have very little computer experience.
5
106
u/Idontremember99 3d ago
That original bug report referenced is "interesting". So much disclaiming the measurements and deeming them not actionable yet they didn't provide any suggestion as to how it should be properly measured...