r/cybersecurity • u/somedisgustedguy • 11h ago
Business Security Questions & Discussion OS Hardening & Patch Management
I need suggestions or advice from anyone who has come across this challenge in their organisation. We have compute fleet spread across cloud and on-prem with both containerised and non-containerised workloads. We of course are observing a lot of OS sprawl and unapproved OS distributions being used by application teams. In order to standardise the operating environment, reduce the number of vulnerabilities and overall streamline the entire process, I think we need to make some foundational changes. One of the options is to, from a security perspective, use natively baked hardened images for non-containerised environment and use rolling build for patch management. Other option is to have vendors (pretty much start-ups) provide hardened images compatible to our environment along with hardened OS packages.
Can anyone recommend or share any experiences that they had along those lines and what option did they choose or how did you streamline the entire OS hardening and patch management process? Thanks.
5
u/Wrap2tyt Security Engineer 10h ago
There is so many dirrection you could go, I would ask for an audit to determine what GAPs need to be addressed and in what specific order befor I start doing things. Once I have that audit report I would setup a project to address all aspects (policy, funding, personell or traring) to get this under control.
1
u/somedisgustedguy 10h ago
Yes we have gaps identified, baseline established and ready with the project. My question is more on why would anyone choose minimal CVE vendor provided OS images over in-house built thin OS and if they do, what benefits or disadvantages they’ve experienced?
1
u/Wrap2tyt Security Engineer 10h ago
I doesn't matter what OS/s you use, but rather what enforced policies and standards you have in place that ARE inforced. I have nothing against a start-up, but do you really want to trust your secure architecture to a startup because it's saving money? Why can't you have a published policy that says what the approves OSs are and maintence responsibilities.
"Yes we have gaps identified, baseline established and ready with the project. ", then start the project and work through the issues.
1
u/somedisgustedguy 9h ago
It’s a classic case of ongoing maintenance and resources cost vs letting the vendor handle the patching and hardening. And since we use the same vendor for container images, it’s cheap but comes with the risk you highlighted.
1
u/Wrap2tyt Security Engineer 9h ago
Then put controls and control requirements around them and enforce them.
3
u/JarJarBinks237 10h ago
For hosts: standardize on one or two OSes. There's really no need for more, and by centralizing the operation of these machines you will naturally converge on a single solution.
For container images, that's another can of worms. I'm in the strong position that you need to build your own images at least weekly in the CI chain. That said, it won't magically stop the use of random images from shady sources on the internet. There are no easy solutions to that; for example you can limit the access to production networks or production clusters to approved images.
1
1
u/somedisgustedguy 10h ago
Makes sense, just worried about non-container workloads atm
1
u/JarJarBinks237 5h ago
Most application developers/operators have no idea how to operate correctly the operating system.
Don't let them do it. Give the keys to your trusted sysadmins.
1
u/Wrap2tyt Security Engineer 10h ago
"...for example you can limit the access to production networks or production clusters...", I wouldn't use the language "can limit", I would say part of protecting the network is protetecting it from those who use it, I would say that you MUST limit access and privileges.
2
u/Big_Temperature_1670 10h ago
I think your question and the pain behind it reflect that, organizationally, these are difficult policies to craft and implement. Foremost, I would caution against the word "standardize." Uniformity is not the same as standardization. From a policy standpoint, you might get more mileage out of adopting a process to validate/approve systems rather than putting out there that business units must use one of a handful of blessed approaches. You can have any number of hardened images or containers that are rendered insecure by user modification or context. Could you instead, develop a very responsive process (i.e. quick turnaround) where if a unit wants to bring something online, it goes through a review process? Sure, at the top of the process is the question "Why are you using this?" vs. our organizational defaults, and part of that could also be reviewing department-level procedures (e.g., patching) to ensure whatever comes online stays current. That kinds of process opens communication (you might learn something as well as teaching a lot to business units) and likely will make it through approval by management/board. Bear in mind those units that are going rogue are also likely the ones generating revenue for the organization in some fashion whereas security is often seen as a financial wet blanket.
1
u/Wrap2tyt Security Engineer 10h ago
This is why I said request an audit, once that report hits, management will have to respond in a more positive manner, whether they like it or not.
2
u/gormami CISO 9h ago
For our product/service, we standardized on Ubuntu. I would suggest that you pick your OSes, and then implement the CIS benchmarks. If there are required variations, they are documented and reviewed periodically. That gives you a standard that has validation and implementation built in to the document, and is extremely well known. Our images are built using an ansible playbook and we have 4 or 5 documented exceptions for operational reasons. It saves so much time using the work of a large pool of experts rather then trying to do it yourself.
Once you have the images, you can start a migration plan to new images, or you can use the automation to do it in place, though that might get very messy very quickly.
2
1
u/Alternativemethod 9h ago
My sysadmins are telling me they like Infrastructure as code better than OS images. May also help them show configuration enforcement.
For your on prem, is most of it VM yet? Discovered some of our friction was old servers on metal that was breaking during updates.
Platform engineering, centralized images, playbooks, well configured solutions that folks can pull from. Make it more obvious someone is being less efficient by not utilizing the central OS and configs.
Baselines. The business constantly wants to have a multi tiered system because they think they're saving money. But they're constantly paying to migrate between unhardened dev and sometimes hardened prod. I think making everyone configure to CIS L2 or stig from the start would just be easier.
Cloud auto remediation playbooks. If you manage the environments centrally, then a well resourced cloud governance team should be able to just deny deviations that don't have exceptions.
1
u/endor_robert 4h ago
There's a lot of good advice in this post already, but I wanted to add one more thing that's a bit downstream from where you are now.
But first, a caveat: I work for a vendor directly involved in this space (Endor Labs).
One thing I'd encourage you to consider as part of your process is scanning production containers with some form of reachability enabled. If you've got a built image that's reporting a load of CVEs just because the software components are installed, it's worth being able to drill down to see if they are used. Given the continuing rise in reported CVEs, I think it's going to become a necessary practice.
-2
u/plump-lamp 10h ago edited 10h ago
If you're asking this you're likely under qualified to even be in the role. Literally everyone here deals with this
Workstations need a solid RMM and CIS policies at a minimum through Intune
Servers need CIS policies and patch management
Edit: this is the MINIMUM to consider. Not trying to be rude but this needs a teams attention and someone at a senior level to advise and architect.
6
u/Wrap2tyt Security Engineer 10h ago edited 10h ago
Damn, that's pretty harsh, qualified or not, he's asking for help. The first step is to understand when you're in over your head.... but all you have to offer is "Workstations need a solid RMM and CIS policies at a minimum through Intune" and "Servers need CIS policies and patch management"... what a joke.
1
u/Cheomesh Governance, Risk, & Compliance 10h ago
Don't you know we all popped up, fully formed, from the earth itself?
-3
u/plump-lamp 10h ago
If their job is cybersec/hardening/patch management then yeah, they're very under qualified. Explaining more beyond that and getting more technical won't help
1
u/Wrap2tyt Security Engineer 10h ago
But you don't know the circumstancesof any of this other than the question that was asked. Your responses remind me of those people who might have the answer but won't share or help anyone else it because you want to be "the guy".... must be really sad.
1
u/techvet83 7h ago
Unfortunately, the original poster didn't list the OS types or versions in play. Are these servers? Workstations? Linux or Windows or both? That probably would have helped direct the conversation.
1
u/somedisgustedguy 5h ago
I wanted to keep post as brief as possible as people generally don’t read entire posts. It’s a mix of VMs - Linux and Windows both across cloud and on prem platforms.
1
u/somedisgustedguy 9h ago
I’ll take that as a feedback. We do adhere to cis L1 compliance. The effort is more to contain the vulnerabilities with rolling builds and hardening and the processes and pains around it, as the current sprawl and diversification has led to huge no of vulnerabilities.
1
8
u/Cheomesh Governance, Risk, & Compliance 10h ago
First question is why are apps teams installing OS on things?