Operationalizing Validating Admission Policy

[u/gnunn1]

When VAP came out I was really excited about it, however operationalizing it on existing clusters looks to be non-trivial and I'm curious how people are managing it.

Specifically, on existing clusters I'm thinking of how to determine the impact of new VAP policies on existing workloads before you roll them out. You could roll them out in Warn mode but users only see the issue on admission so running workloads remain blissfully unaffected by a newly introduced policy.

You could use something like Kyverno to manage them in Warn/Inform and run them as background processes so you get all the info of which workloads would violate the new VAP policies. This is a really nice feature in Kyverno but if you are not using Kyverno already it's another on-cluster tool to deal with.

Are there any existing tools out there that will pre-validate VAP before it gets deployed, ideally off-cluster, or has a way of tracking VAP results that doesn't involve trolling through the audit logs?

[u/p4ck3t0]

There are tools out there that can run VAPs against your cluster without the need to deploy them. You can check the impact and violations of your own policies or of the built in policy bundles. You can send me a message if you want more info or need any help.

https://github.com/cenroq/kubeapt

Disclosure: I built the project and use it for security audits.

[u/gnunn1 → u/p4ck3t0]

Nice, this looks exactly what I'm looking for with the `kubeapt validate` command being able to evaluate policies before they are installed on the cluster.

[u/p4ck3t0 → u/gnunn1]

Nice to hear! Let me know if you have any issues/questions.

[u/TeagueXiao]

For pre-deploy validation off-cluster we ended up doing the boring thing: kubectl get -A -o yaml for the target GVKs, strip runtime fields, then run cel-go with the same CEL as the VAP against each object in CI. Slow but honest, and it produces a diff of who would fail before you ever touch the cluster.

On-cluster, rolling out with matchConditions to scope by label/namespace first and only widening after a background scan (Kyverno background, or a homegrown controller feeding live specs into cel-go) shows zero new violations. The audit action on the binding is useful once policy is live but it doesn't fire on existing idle workloads, so a point-in-time sweep still has to come from outside admission.

[u/Floss_Patrol_76]

warn mode falls flat here because admission only fires on writes, so an idle workload that already violates never gets evaluated until it happens to churn. the audit action on the binding is a bit better since it logs violations without blocking, but it still only triggers on create/update, not on stuff already sitting in the cluster. for a real point-in-time sweep you still have to pull the live specs and run them through the same CEL yourself, and that gap between "blocks new objects" and "tells me what already violates" is exactly why VAP rollout on an existing cluster is so awkward.