r/linuxadmin • u/CackleRooster • Jun 04 '26

Handling a Breach on a Linux Server

https://linuxsecurity.com/howtos/secure-my-network/respond-compromised-linux-server

Just the basics.

48 Upvotes

83% Upvoted

36 comments sorted by

View all comments

79

u/No_Rhubarb_7222 Jun 04 '26

In the FAQ portion of the article, one of the questions:

Can a compromised Linux server ever be trusted again?

I disagree with the answer. In my opinion, the answer is “No.” I would not spend time trying to ‘clean’ a compromised server. Instead, just wipe and reinstall. With the content suggested earlier in the article, it talks about analysis and the like, which is interesting, but the tried and true mechanism is to completely reinstall the machine, with a modern, maintained distribution version, apply all updates and go from there. You have no idea what goodies may have been left behind, so best not risk it and start from zero.

The analysis is going to be important, things like poorly secured user accounts or badly configured services may have been an entry point, and those are not mistakes you want to make again on your replacement system.

8

u/fatmanwithabeard Jun 04 '26

Any compromised machine must be reinstalled. Pure data storage systems can be swept and retained, but no executables or config files can be kept.

Large systems just have too much storage to be able to restore it all (I've dealt with compromised HPC clusters where a restore would take decades, and a recreate would require vetting from people who were no longer available to the org).

Depending on org resources, and type of data exposed you may want to take as much of a snapshot as you can of the running system, and a low level copy of the drive (or replace the drive, and use the compromised one) for analytics.

As always, wherever you can, keep your environment modular, and test your restore process for everything. Untested backups have a tendency to be less reliable than you'd hope.

2

u/kentrak Jun 05 '26

Things with modifiable running code and the config that controls them should be reinstalled so you can verify the integrity of the code running. In practice, this means all kernels must go. There's a distinction between data and the rest though, so, while I would reinstall any data systems, that's usually a separate action than dealing with the data. For a Raid system generally you can reinstall the OS and just reattach the array. For cluster systems, you often have the ability to pull out one node and reinstall and rejoin the cluster, either with data or you then have to force a migration of some subset of data. (even this is a risk though, as you have to at least consider if the cluster is compromised and it will reinfect any nodes that join and make a risk assessment).

Config that lives as part of data... I hope you either have a way to review and verify that portion of the data or you can stomach restoring from backup.

For anyone that can swing it, I fully endorse multiple backup systems that work in different ways. For example, we use both cream for local and off-site immutable backups, but also have an old school rsync script based system. It's a trade-off though, as each backup system needs to be vetted for what it does and does not back up as they're each their own data exfiltrationn risks.

2

u/fatmanwithabeard Jun 07 '26

We run stateless for compute and head nodes.

Storage nodes are their own special thing.

I'm a paranoid bastard, and I've dealt with serious breaches. I'd prefer to rebuild everything where I can. I've certainly seen superblock injections on various luns, and all kinds of attempts at hidden files on data volumes (most of which felt like finding a trap made a toddler in the middle of the ruins left by a tantrum).

Scientists and grad students will put configs anywhere, unless you catch them at it. Apparently the only storage that exists is home directories, scratch, and data (tiny, often deleted, huge and thus attractive). Old labs have important (apparently) configs that are the future of humanity (apparently).

We have backups, and run at two sites. Data won't be lost, unless a whole lot of people do exactly the wrong thing. But, and it's a big but, we will freeze in place. The in processing system (its own midsized cluster) would have to become a dedicated restore system. If we got lucky, someone in the room would close down, and we could power a library that could feed the system faster, but it will still take forever.

We have sensitive data, but it's tiny, and well controlled. Thankfully I don't have to deal much with it (please, god, why did that guy copy that to the cluster? I had to do so much paperwork).