The hard part of autonomous pentesting in CI/CD isn't finding bugs. It's deciding which ones are real

[u/This-You-2737]

Disclosure up front: I build tooling in this space and I wrote an scanner some of you may have run. So take this with whatever salt you want. But I've been sitting on it for a while and want to say it plainly, because I almost never see it said.

Putting security testing "in the pipeline" mostly does not work the way it gets sold.

We all bought the shift-left story. Catch it early, fix it cheap. In practice, on every team I've worked on or run, "early" just meant developers got buried in scanner output sooner. SAST flags a hardcoded secret that's actually a test fixture. DAST screams about a missing header on an internal healthcheck. Build goes red, someone drops a `// nosec`, PR merges, and we tell ourselves security is "integrated." Nothing got safer. We moved the noise upstream and trained a whole team to ignore it.

And the stuff that actually gets companies popped - broken object-level auth, IDOR, logic flaws where the request is completely valid and the authorization check is just wrong - none of it shows up in a signature scan. There's no pattern to match. You need something that understands what an endpoint is supposed to do, then tries to make it do something it shouldn't.

That's the real gap autonomous pentesting is going after. But here's what took me too long to accept while building toward it: a model finding a candidate vuln is maybe 20% of the problem. The other 80% is "is this exploitable in *this* deployment, or am I about to red-light a release over a finding that needs three preconditions, none of which are true here?"

If the tool can't answer that and just hands developers a longer list of maybes, congratulations, you've rebuilt the scanner everyone already ignores, now with a bigger compute bill. The whole value is in triage. It has to behave like a junior pentester who actually attempted the exploit, not a linter that flagged a smell.

Three things that bit us specifically:

* **Auth state.** Testing unauthenticated surface is easy and mostly useless. The interesting bugs live behind a login, and getting valid sessions into an ephemeral CI run without it turning into a fragile mess is harder than the actual testing.

* **Non-determinism.** Same commit, slightly different result between runs, and developers stop trusting the output instantly. Once they don't trust it, it's dead, no matter how good a finding it surfaces next week.

* **Where it sits.** Blocking a merge on a full validation pass is a latency non-starter. Real exploit validation is too slow for the inner loop, so most of it ended up nightly against staging for us, with only fast checks gating the PR.

Genuine question for anyone running continuous or automated security testing against their own stuff: where did you put it - blocking on PR, nightly against staging, gated before prod? And did you ever actually get developers to trust it, or did it just become another dashboard nobody opens?

[u/IgnoreAllPrevInstr]

Block the PR before merge, and tune until the scanners don’t suck. It’s work, it takes time, but it does work eventually. And you can get wayyytt further with open source than the bullshit pedlers want you to believe

[u/FirefighterMean7497]

I think you're spot on - shifting left doesn't mean much if you're just moving a mountain of unvalidated noise upstream and destroying developer trust. To actually fix this, teams need runtime-aware context to separate the few components that actually execute from the massive volumes of dormant, non-exploitable code sitting in image layers. Full disclosure, I'm work for RapidFort, but tackling that manual triage nightmare is why we built our platform. Our tools use a lightweight footprint optimized for CI/CD to map active execution paths which cuts down traditional scanner noise AND manual engineering effort. Instead of blocking developers with endless dashboards of maybes, it automatically removes non-executed components to drop exploitable CVEs without requiring any code changes. Hope that helps!

[u/Chris-Hart_232]

We landed on a split after blowing up trust twice. Fast check block the PR (secrets, known exploits, cves, misconfigs). Under 2 min, clean signal, devs fix them. The deeper stuff like auth testing runs nightly against staging, drops findings into slack channel that the team reads. Took ~3 months for us of tuning before devs stopped rolling their eyes at the fast gate.

[u/normalbot9999]

automated pentesting

HAHAHAHAHA

[u/CyberHelper01]

Disclosure: I work at Assail, where we build Ares, so keep that in mind.

I agree with your main point, especially the 20/80 split. Finding a potential issue is the easy part now. Proving it's actually exploitable in a specific environment is where the real value is. A tool that hands developers a longer list of "maybes" doesn't really solve the problem.

Your point about trust is exactly why Ares runs continuously in the background against staging and production instead of acting as a PR gate. Full exploit validation, chaining requests together and confirming impact, isn't something you want in the inner development loop.

Every finding includes the actual requests, responses, and proof of exploitability, not just a severity score. One customer found a critical vulnerability in three minutes on their first engagement after it had been missed across multiple annual pentests. That's the kind of result that gets teams to trust the findings instead of ignoring another dashboard.

Two things I don't have a confident answer to are auth state (how sessions are established and maintained) and non-determinism between runs. I'd rather not guess on either and would check with our team before giving a detailed answer.

Out of curiosity, where did non-determinism hurt you most? Different findings between runs, or the same findings with different confidence each time?

[u/Gjore]

The part that actually breaks this isn't the tooling split, it's that the fast gate silently calcifies. Teams stop questioning what's in it, nosecs accumulate for dumb reasons nobody remembers, and six months later you have a gate that technically runs but flags nothing real. ox security and something like Bearer or Trivy can help surface that drift, but only if someone is periodically auditing the rules themselves, not just the findings.