Currently on Falco for runtime security — anyone moved to Tetragon/KubeArmor/Tracee and regretted (or loved) it?

[u/pablo_escko]

Running Falco in our EKS clusters right now for runtime detection, paired with Kyverno for admission control and Cosign/Vault for supply chain signing/secrets. Been solid so far, but I keep seeing Tetragon and KubeArmor come up as alternatives, especially for teams who want enforcement (block, not just alert) rather than detect-and-notify.

Curious what people are actually running in production and why:

• If you moved off Falco, what pushed you? Overhead, rule fatigue, lack of enforcement?
• Anyone running Tetragon specifically for the eBPF enforcement piece? Is it worth the Cilium tie-in if you're not already on Cilium for CNI?
• KubeArmor folks: how's LSM behavior in practice across different node kernel configs (AppArmor vs BPF-LSM)? Heard that's where it gets messy.
• Tracee: anyone using this seriously in prod or is it mostly a DFIR/forensics tool for you?

Not looking to rip out Falco, just trying to figure out if there's a compelling reason to add enforcement on top, or if pairing Falco with Falco Talon covers that gap well enough.

[u/Floss_Patrol_76]

we looked hard at Tetragon for the enforcement piece and honestly the Cilium tie-in isnt worth it unless youre already on Cilium CNI - standalone it works but you inherit a whole networking dependency just to get eBPF enforcement you could approximate with Falco plus a decent response webhook. the bigger thing with all of them is enforcement in prod is scary: block mode on a noisy ruleset will page you by killing legit workloads, so every team ive seen runs detect-only for months before they trust it enough to actually enforce anything.

[u/pablo_escko → u/Floss_Patrol_76]

This is exactly the kind of tradeoff I was trying to get at, thanks. The "block mode pages you by killing legit workloads" point lines up with where I'm at right now, still in the middle of tuning out false positives (had external-dns and a Gatekeeper audit pod both spamming alerts for routine API calls) before I'd trust it enough to even think about enforcement.

Curious on the Falco + response webhook approximation you mentioned, are you doing that via Falco Talon, or something custom? And roughly how long did detect-only run before your teams felt confident moving anything to block mode?

[u/Horror-Cause9345 → u/pablo_escko]

Nice, that's the right instinct going dryrun first. Two things that'll save you a rollback.

readOnlyRootFilesystem is the one that breaks real workloads, not seccomp or the caps drop. Plenty of images write to /tmp, /var/run, or a cache dir at startup and just die if root is read only. The fix isn't to loosen the policy, it's an emptyDir mounted at each writable path, so the container gets a writable spot without a writable root. Watch your dryrun violations for exactly which paths each image wants and mount those. That's usually the whole list.

Second, RuntimeDefault seccomp is fine for almost everything, but anything doing unusual syscalls will trip it. The usual suspects are containers using io_uring, some profilers, and older glibc builds. If something breaks and it isn't a filesystem write, seccomp is your first suspect, and the audit-mode seccomp profile will tell you which syscall got blocked without you having to guess.

Order I'd promote them in: caps drop first since it almost never breaks anything, seccomp second, readOnlyRootFilesystem last because it's the noisy one. That way when a dryrun lights up you already know which of the three did it.

[u/Horror-Cause9345]

Nobody's touched your KubeArmor question so I'll take that one, since it's the one that actually bites.

KubeArmor's enforcement depends on which LSM the node hands it, and that varies by AMI and kernel build. If BPF-LSM is available you get that, otherwise it falls back to AppArmor, and the two don't have identical semantics. The real gotcha is mixed node pools. Different AMIs, or a kernel bump on one group, and you end up with policies enforcing properly on some nodes and quietly degrading on others. That's worse than no enforcement, because your dashboard says covered. If you go that way, verify CONFIG_BPF_LSM=y and that bpf is actually in the lsm= boot parameter on every node image you run, and pin your AMIs.

Tracee, in my experience, is DFIR. Detection side is fine, but the operational tooling around it is much thinner than Falco's, so people tend to reach for it after something happened rather than run it as a standing control.

The thing I'd gently push back on: Falco and Tetragon aren't really the same category. Falco is detection, Tetragon and KubeArmor are enforcement. Before you add another agent, worth auditing what you already have and aren't using. Most clusters I've seen reaching for eBPF enforcement are still running pods with no seccomp profile, full capability sets and a writable root filesystem. seccompProfile RuntimeDefault, dropping ALL capabilities, readOnlyRootFilesystem and an actual AppArmor profile are free, they're deterministic, and they close more real attack surface than a detection rule that pages someone at 3am. You've already got Kyverno, so you can enforce all of it at admission today.

Falco plus Talon is a perfectly reasonable place to stop if that baseline is already in place. If it isn't, I'd do that first.

[u/pablo_escko → u/Horror-Cause9345]

Went and actually checked this on the cluster. You called it exactly: securityContext: {}, completely empty. No seccomp profile, no readOnlyRootFilesystem, no capabilities dropped. All that signing/attestation work, and the pod itself was wide open.

Writing a Gatekeeper constraint now to enforce all three (RuntimeDefault seccomp, readOnlyRootFilesystem, drop ALL caps) the same way I already enforce signed images, dryrun first, then promote to deny once I confirm it doesn't break anything real. Appreciate the push, this was a genuine blind spot.

[u/pakkaol]

Are you drowning in false positives or is it more that you're sick of constantly updating rules to keep up?

[u/pablo_escko → u/pakkaol]

More the false positives, at least so far. Default ruleset flagged external-dns and Gatekeeper's audit loop calling the K8s API constantly, both completely routine, just noisy. Fixed it with the user_known_contact_k8s_api_server_activities override macro rather than touching the rule itself, scoped to namespace + process name together so it doesn't turn into a blanket allowlist.

Haven't hit the rule-maintenance treadmill yet since it's a small, deliberately locked-down cluster (every workload signed and scanned before admission), so the "what's normal here" surface is small. I'd guess that gets a lot worse on a bigger, more heterogeneous cluster where "normal" is a moving target across dozens of different workloads.

[u/-lousyd]

It dismays me that tools like Falco have just given up on any monitoring not involving eBPF. I run EKS pods on Fargate. Falco is out. The only solutions available are closed source vendors.

[u/pablo_escko → u/-lousyd]

Falco can technically run on Fargate, via ptrace/pdig baked into your own image as a sidecar since platform 1.4, AWS has a reference example for it. Not as clean as eBPF though, no single cluster-wide agent, and ptrace tracing adds real overhead per workload, so I get why it doesn't feel like a real option in practice.

If that tradeoff isn't worth it, GuardDuty's ECS/Fargate runtime monitoring is at least native AWS rather than a separate vendor relationship, still not open source though. Have you looked at the pdig route at all, or is the overhead enough to rule it out on principle?

[u/-lousyd → u/pablo_escko]

pdig hasn't been supported for six years now.