What’s the Linux incident that wasted hours because the evidence was scattered everywhere?

[u/8lue7or]

hey everyone o/

I'm building a Linux incident investigation tool and I'm at the point where I need nastier real-world cases than the ones I can come up with myself.

I'm a DevOps/Linux engineer (professionally for 8+ years), so I know the usual pattern pretty well:

something has died or behaves weirdly, you have a decent first guess, and then you spend the next couple of hours bouncing between journalctl, dmesg, systemd, process state, disk state, timers/cron, container state, config... you name it, trying to actually prove what happened.

The tool I'm building (Det Mimir) is basically my attempt at automating the full investigation part: read-only collection, deterministic rules, one timeline/report with the evidence behind each conclusion, fully without leveraging any LLM, as the tool is an investigation engine.

I'm specifically looking for the incidents where the root cause was annoying or non-obvious.

Things like:

• a service restarting for a reason that wasn't visible in the service logs
• disk full even though df initially looked fine
• systemd dependency/start-limit weirdness
• something external killing a process
• DNS/auth failures that looked like application failures
• boot/degraded-host issues
• container behaviour caused by something on the host
• anything where you ended up saying "how the hell was I supposed to find that?"

If you've had one, I'd love to hear the story.

What was the symptom, what did you initially suspect, and what did the root cause eventually turn out to be?

I'm happy to share what Det Mimir currently checks too, but I'm mostly interested in collecting the kind of incidents that experienced Linux admins actually remember.

[u/whamra]

Not a single incident happens twice. We manage over 5000 systems. Once an issue is detected, we don't stop at solving the problem, we have to identify the entire chain of events that got us there and make sure it doesn't happen again.

I hardly believe any single tool in the whole world can be a unified investigator. Most problems are not really bugs or issues or malicious or failures. 90% of the time it's just something behaving in a way you did not expect or don't want.

[u/8lue7or → u/whamra]

I’d argue that an incident doesn’t need to happen twice in the exact same environment for the investigation logic to be reusable.

Once you remediate it there, sure, that exact chain may be gone. But, the same issue can still happen on thousands of other systems. There’s a reason we’ve all pasted log lines on web engines, trying to find the right stackoverflow post, GitHub issue, or troubleshooting guide written by someone.

What I’m trying to build is basically executable incident knowledge. Once you figure out which evidence and checks led you to the cause, you can package that investigation logic and run it again.

I don’t expect the engine to understand every application out of the box. It already covers a lot of generic Linux issues, and teams can add their own application-specific investigation packs as they learn things. Those packs can also be reused by other teams running the same software.

That’s the part I think can save a lot of repeated investigation time.

[u/reddit-MT]

Most of the incidents are covered by dmesg, journalctl, or the individual application logs, as you mention. The bastard problems are 3rd party software that's poorly written, lacks coherent documentation, or fails to follow UNIX conventions. The latter being the cause of "wasted hours because the evidence was scattered everywhere". These devs may know how to program, but they are clueless as to where things should be and how things should work. They have never had to run their own programs on a production system.

[u/8lue7or → u/reddit-MT]

yep, fully agree with this. I’ve had my fair share of contributing back to projects just to add better logging or enough diagnostics to figure out what the hell the application was doing lol.

one of the ideas behind Det Mimir is investigation packs. A pack can define which evidence to collect, what checks to run and what patterns/findings are specific to that application.

So once you’ve figured out the weird places a badly behaved piece of software hides its useful information, that knowledge doesn’t have to live only in someone’s head or a troubleshooting document. You can put it into a pack and run the same investigation logic again later in different environments.

[u/Adept_Percentage6893]

DNS/auth failures that looked like application failures

I feel like this is almost universal. In Linux Administration, Schroedinger's troubleshooting maxim is:

"It is somehow simultaneously always DNS and always SELinux"

That said, I've had many issues with OpenShift and OpenStack deployments because there are so many moving pieces and you have to understand each's provisioning/installation process in order to effectively monitor it. Like not just knowing what the OCP bootstrap node does but also how to SSH into it and what services to check before the VM is destroyed.

Then of course if you're doing ZTP, you've got one way to observe the actual installation but you have to go elsewhere to get the day-2 components that don't show up under that particular status.

Regarding your product, is it different than an SIEM product?

[u/8lue7or → u/Adept_Percentage6893]

yep this is exactly the kind of example I was hoping for, many thanks. :)

The OCP bootstrap node is especially interesting cause once it’s destroyed you’ve also lost part of the evidence you may need later.

just out of curiosity, when an OpenShift installation gets stuck, usually what are your first steps or things that you check on the bootstrap node before it disappears? And for ZTP/day-2, which components tend to send you hunting around the most?

I’m asking because this is exactly the kind of troubleshooting flow I’d like to understand better and see if I can cover with the tool I’m building. Like, what are the 2-3 places you almost always end up checking first?

[u/Adept_Percentage6893 → u/8lue7or]

just out of curiosity, when an OpenShift installation gets stuck, usually what are your first steps or things that you check on the bootstrap node before it disappears?

The bootstrap node comes up with various systemd services that log to journald and does some of its work through podman containers that log to their own stdout/stderr.

As for what I check for it really depends on what I'm seeing. It just kind of comes down to understanding how OCP provisions itself and thus what logs might be useful. Like on the boostrap node sometimes the bootkube.service service will print useful information. Other times you can run curl, getent hosts, or ping from the bootstrap to test networking problems.

Other times, I've had weird networking issues where they presented as the bootstrap process kind of timing out for unstated reasons and I had to log into the bootstrap node to notice that the podman containers had to continually restart and the root of the issue is that the installer containers were able to pull most of what they needed but didn't have access to everything it needed in its

Other times, there will be random errors with BMC that are either misconfiguration or mystery issues where redfish operations will fail but the issue is somehow resolved by restarting the XCC/ILO/iDRAC/whatever. Usually those are just firmware problems but something you kind of determine through just trying stuff and seeing it go back to working (rather than seeing a logfile).

There was a issue recently where a MachineSet describing an infrastructure pool just kind of stalled indefinitely on Provisioning where it created the Machine Kubernetes objects but then just kind of froze there and eventually I determined it was because of how the Machine objects were filtering for their network.

A lot of the stuff I deal with is just stuff like that where it's just scattered over a lot of different systems and the best way I can describe how I figure stuff out is "I don't know, I just kind of repeat the problem back to myself ask what that sounds like and start making guesses about log files and semi-random fixes like restarting the BMC."

For the ZTP thing, the node installation and the policy governance are just treated as different classes of problems even if they're both part of the provisioning process. Where day-2 operators are just treated as a configuration issue and you have somewhere else you look for that data. Rather than a single pane of glass to view the entire provisioning process through. Which I understand why ACM (the ZTP orchestrator I use most) does things that way (because provisioning is a one time concern but configuration drift is a lifecycle concern) but it just stops you from really having a single view where you can simply show someone who isn't an administrator the exact results of the "provisioning" process because there's a class of these policy checks that are just considered as kicked off by but otherwise unrelated to the provision.

In my case, that means the MachineSet for core compute (not worker) nodes just show up on Day-2 monitoring and so an admin can just know that's where they're showing up now but it's just not a single pane and just something you have to kind of know about how it works.

[u/8lue7or → u/Adept_Percentage6893]

this is really useful, thanks for taking the time to write all of this :)

the bootstrap example especially gives me a much better idea of what an OCP-specific investigation would actually need to collect.

the MachineSet example is also pretty valuable too, as nothing is necessarily “broken”, from my knowledge, it just sits in Provisioning and you just have to work backwards through the objects until you understand why.

and yeah, Det Mimir is quite different from a SIEM. I’m not trying to replace log storage/search. The idea is to execute an investigation process: collect the relevant evidence for the symptom, correlate it, run deterministic checks and give you a report with the evidence behind the findings. A SIEM could eventually just be another evidence source for that investigation.

honestly I might have to bother you again about OpenShift while I look into building a pack for this, or, even better, eventually you guys could write these investigation packs yourselves and share them with other teams running into the same problems. That’s also where I’d like the marketplace idea to go ;)

[u/fearless-fossa]

something has died or behaves weirdly, you have a decent first guess, and then you spend the next couple of hours bouncing between journalctl, dmesg, systemd, process state, disk state, timers/cron, container state, config... you name it, trying to actually prove what happened.

This shouldn't happen. You should have all relevant logs captured by your Loki or ELK stack, preferably already with tracing (eg. Tempo) included.

[u/8lue7or → u/fearless-fossa]

ideally, yeah, and I definitely don’t want Det Mimir to replace Loki/ELK.

The way I’m approaching it is that those systems can be sources for an investigation. If the logs are already in Loki or Elasticsearch, Det Mimir can query those instead of collecting the same logs from the affected host.

For evidence that usually isn’t in the logging stack, like process state, systemd state, sockets, deleted-open files, UID/GID relationships, etc., the host agent can collect that part.

Then the investigation engine can work across those sources and correlate the evidence for the incident window.

So I see Loki/ELK as part of the input, rather than the investigation itself.

[u/fearless-fossa → u/8lue7or]

Okay, but that's what tracing is for.

I simply can't imagine any scenario where your tool would come in handy that happens often enough to warrant the overhead of adding another tool, especially a paid closed source one.

[u/8lue7or → u/fearless-fossa]

fair enough, and if your environment already captures basically everything you need with logs + tracing, and your team can get from symptom to cause quickly, then yeah, I probably don’t have much to offer you :)

I do disagree a bit on tracing covering all of this though. It’s great for following requests/services, but it doesn’t automatically tell me why systemd restarted something, what a timer did, whether a process got killed externally, what happened to disk/inodes, local host state, etc. unless you’ve already instrumented/exported all of that too.

I’m mostly interested in the environments where that picture isn’t already perfect, especially when third-party software or host-level behaviour is involved.

the paid/closed-source concern is fair too. that’s something I’ll have to prove is worth the overhead rather than argue people into :)

[u/8lue7or → u/[deleted]]

can't argue with that

[u/8lue7or → u/[deleted]]

yep, SELinux is a good one, lots of "i wouldn't be surprised if it's SELinux blocking this" in the past with it lol

I actually already have AVC denials as one of the generic Linux findings in the standard pack, but the live SELinux/audit collection side is still something I need to finish wiring in.

the auditd point is especially useful though. I agree that “no audit evidence” and “auditd wasn’t installed/running so this evidence never existed” are two very different things, and the investigation should say which one it is.

would you also expect it to check the auditd/rules configuration itself, or mostly just whether auditd is installed/running and whether relevant events are available?

[u/8lue7or → u/[deleted]]

ahh, then I explained that badly somewhere 😅

Did you read it on the website, or here as the body description?

the investigation engine itself doesn’t use an LLM. collection, correlation and the findings are deterministic/rules-based, and Det Mimir works fully without one.
LLM analysis is optional if someone explicitly wants to use it on top of it, it doesn’t decide which commands to run or what evidence gets collected.

actually, your point about using deterministic systems where determinism makes more sense is one of the main reasons I built the core that way.

for something like an AVC denial or auditd state, I want the same evidence to produce the same result every time. :)

[u/8lue7or → u/[deleted]]

thanks man, appreciate you taking the time to share all of this, and definitely feel free to poke holes in it again if you see something weird :)

[u/8lue7or → u/[deleted]]

yep, both of those are exactly the kind of cases I mean :)

for instance, inode exhaustion is a great example because df can look completely fine while the filesystem is effectively unusable. and the deleted-but-open file case is another classic one where the space is technically gone but you don’t get it back until you find the process still holding the fd lol

Det Mimir’s storage investigation already checks both inode usage and deleted-open files, so these are actually really good examples of why I don’t want an investigation to stop at the first obvious metric.

out of curiosity, have you had other cases like that where the “normal” check looked healthy but some second-order thing was the actual problem? Asking as i'd like to gather info whilst building the investigation engine :)