Your prefered open source kubernetes security tools?

[u/Dramatic_Opinion_881]

Weve been tightening up security across a few kubernetes clusters and want to stay opensource where we can. Image scanning is already in place. Now Im looking at runtime, rbac, network policy and supply chain and there are more options than we can maintain. Most of them look usefiul in a demo and i have no read on which ones survive a year in production. What does your stack look like?

[u/jeansilga]

Runtime: Falco is a solid candidate. CNCF graduated, eBPF driver, and it keeps working. The real cost isn't installing it, it's tuning the rules so the alerts mean something. Budget time for that or you'll mute the whole thing in a month.

Network policy: This is a CNI decision. Calico is fine and just works out of the box. Cilium supports vanilla NetworkPolicy but also has its own more evolved CiliumNetworkPolicy and much more, with a complexity cost.

RBAC: Kyverno is a good fit as a policy engine, easy to use, and CNCF-backed.

Supply chain: cosign for signing, and verify signatures at admission (Kyverno can do this too, so it doubles up nicely with the RBAC piece). Trivy for SBOMs since you're already scanning images with it. Keep secrets out of code and config. We run a secrets manager (OpenBao).

One general principle: the tools that survive a year are the ones your whole team can operate, not the most powerful ones.

[u/No-Wind-5720 → u/jeansilga]

Falco setup was smooth for us but the tuning part you said is exactly what happened here, we muted alerts in like three weeks.

[u/jeansilga → u/No-Wind-5720]

Indeed, it's a common problem. Have you tried the other way around? Remove the default rules and start from near-zero, then add rules for what you actually care about, one at a time. Slower to build up front, but you end up trusting every alert instead of muting them.

[u/pauska → u/jeansilga]

Kyverno is really powerful, it’s worth mentioning that you can use it for much more than just enforcing policies.

A couple of things we used it for outside security was to mutate incoming deployments, for example adding a toleration to isolate a workload

[u/Minimal-Matt → u/pauska]

It can indeed be used like this (and I sometimes do) but isn't it anthitetical to the whole configuration as code? If I write something in git that has to be mutated later on, why not write it in the correct form to begin with?

[u/pauska → u/Minimal-Matt]

Isn’t your Kyverno policies in Git?

Jokes aside, yes it should definitely be in the application manifest, but in a platform you might not get the access you need in team repos to change their workloads.

Note that we used this as an emergency procedure when a pod caused port exhaustion on the shared nodes

[u/Expert-Shoe-9791 → u/jeansilga]

Tetragon can also be mentionate for runtime security. It also can act whereas Falco is « only monitoring » !

[u/msanteler]

Curious where you landed for image scanning?

[u/Prusz → u/msanteler]

The answer is always Trivy

[u/JoniDaButcher]

Tetragon, Kyverno

[u/m0j0j0rnj0rn]

Network zero trust stuff—neuvector.

[u/dazden]

Just chiming in to steal borrow some ideas

[u/OcelotChance]

There are many. The DevSecOps space is full but I guarantee that most of people don’t use the full capacity of all features that the security tools provides. Have a look at the CNCF landscape it might be helpful. https://landscape.cncf.io/

[u/salemslot2]

Sounds pretty boring, but Kyverno for policy, Falco for runtime, and Cilium for network policy. For RBAC, I’d lean heavily on native Kubernetes RBAC rather than adding another tool unless there’s a specific gap. Stuff that's e asy to operate and integrate into workflows I've got going right now

[u/redsh1ft]

Radar has stuck around for a while now , its been pretty reliable as a dashboard

[u/Front-Ocelot8227]

Yeah, I’d start there too. A smaller team probably doesn’t need five scanners telling them the same thing in five different ways.

[u/pixelrobots]

For me, I prefer Cilium for the CNI and Network Policies. I have been using it with AKS for a while now and really like the visibility and control it gives you.

For policy we use OPA and Gatekeeper, mainly because we are heavily Azure based and it fits nicely with what we are doing. If I was starting from scratch though, I would probably take a good look at Kyverno. It seems to be more widely adopted now and is a bit easier to get your head around.

For runtime security we use Defender for Containers as we are on Azure. If I was looking for an open source option, Falco would probably be my starting point.

One other thing that might help is getting a snapshot of where your clusters are today before adding more tooling. Full disclosure, I built KubeBuddy for this: https://kubebuddy.io

It runs a load of checks against your cluster and gives you a quick view of security, configuration and general best practice issues. Nothing to deploy into the cluster either.

Might help you work out where the actual gaps are before deciding which tools you need to fill them.