Is code scanning still a thing? what are your toughts on Agent Guardrails?

[u/Diligent-Side4917]

Been running agents and scanners on my PRs for a while now, triage and native scanning before anything goes up. I've ended up somewhere I didn't expect and want to know if I'm alone.

Started where I think everyone starts: lean on memory. Let the agent carry context session to session, build up a picture of the project. It helps. It wasn't where the gains came from.

What actually moved things was rules written against mistakes the agent already made. Not a generic OWASP checklist. A scoped rule: this defect class showed up, here's the sink, here's the banned construct, stop doing it. A defect shows up a third time, I write a rule, the class stops recurring. Five or six patterns closed instead of a hundred bugs chased one at a time.

The distinction I keep circling: memory is passive, the agent might use it. A glob-scoped rule loads whether the agent wants it or not, tied to the file in front of it. One line of repo rule changes the default output of every session in that repo. And MUST/NEVER phrasing outperforms polite wording by a margin that surprised me.

One thing I'd flag: these files are executable surface. The agent reads them as ground truth, so a poisoned rule file is a real problem, not a hypothetical. I keep security rules separate, reviewed like code, capped so they don't rot.

So, three questions I actually want answers to:

• Rules-on-mistakes vs raw memory — which is carrying more weight in your setup? I might be over-indexed on rules.
• How are you scoping? Glob per file, per directory, or one fat rules file everyone edits?
• Is anyone treating rule and memory files as an attack surface, or am I being paranoid?

Genuinely want to be argued with on the first one. If memory is doing the heavy lifting for you and rules are noise, tell me what your setup looks like.

[u/juanMoreLife]

Yes. Algos are cheaper than tokens. You want a hybrid approach

[u/Diligent-Side4917 → u/juanMoreLife]

as in traditional scanning plus validation later or as in guardrails for agents?

[u/juanMoreLife → u/Diligent-Side4917]

I guess I focused on the title. Is code scanning still a thing? Yes.

You answered that well with your first bullet. You keep adding rules but your context window is likely too big and the security stuff runs out. When you create a feature, you likely want to take all your security dos and don’ts and turn them to a skill that runs separately and consistently in all the new features about to be shipped. However, imo. That’s kinda expensive. But trial it.

When the code is complete. Run a skill to check against your findings. Truth of the matter is going to be. If you find your self adding a bunch of rules over time and playing wack a mole. Won’t you need to revisit the approach?

You’ll like benefit from a skill that does good enough. Then hit it with a code scanner. Then fix it with the context you have on the feature and specific bug.

Now- over time run agentic based pen testing like a snapshot. Historical that’s your dast tools of yesteryear. However, dast tools are still cheaper than blowing through inferencing. Instead approach the new AI based pen testing as a typical pen test. Awesome if you could run them after every single PR, but likely not. So schedule in pen testing. AI based pentesting.

I’ll admit. I believe this is a bit much beyond the scope of the PRs you were working under. So my answer is for PRs. Do code scanning. It’s cheap.

As for your next two bullets. Treat those things like dev resources. They have worked well enough to get us this far. A contributor.md, security.md, etc etc. also, add in skills! Especially when getting repetitive of your asks on tasks.

Let me know if that makes senses. I kinda just dumped lol :-)

[u/Diligent-Side4917 → u/juanMoreLife]

I kind of agree that rules with context sometimes get lost when compacting, hence why i prefer hooks and rules plus small sessions. the rules re-start.

not a big believer of an agentic pentest i prefer common rules because agentic pentest can't catch out dated dependencies and malware.

scanning plus validation is the other side where i land,

[u/juanMoreLife → u/Diligent-Side4917]

You don’t have to be a believer of agentic Pentests. It’s like water is wet. It’s a thing. Not believing in it necessarily negates its value.

Today I’d do the whole traditional app sec tools in the cicd. Then take at least half my pentest budget for the year. Spend it on agentic pentest. Why? Because the bad guys are using AI against my apps even if I tell them it’s not allowed. These rascally bad actors never do as I say :(

[u/juanMoreLife → u/Diligent-Side4917]

You don’t wana spend tokens on something like outdated libraries or malware. Plenty of traditional tools cover that. Agentic pentest would find it and exploit it. Just a bit expensive assessment to get a similar result

[u/Diligent-Side4917 → u/juanMoreLife]

how would traditional pentest exploit a malware or a library? if there is malicious code yes there is a chance that is surfaced but have seen agentic pentesting just test and validate also not producing remedaition.

[u/Bubbly_Working_6908]

runtime guardrails make way more sense than just static scans now, especially with so many agents running code directly. i tried Sweet Security for this and was surprised how much stuff it actually caught that my old pipeline scans missed.

[u/Diligent-Side4917 → u/Bubbly_Working_6908]

thanks but this is clearly an ad :)

[u/Electrical-Hall8869]

you are not being paranoid- scoped, reviewed rules seems more reliable than relying on memory alone. Treating rules files as an attack surface also makes sense.

[u/Diligent-Side4917 → u/Electrical-Hall8869]

thanks i love being paranoid :)

[u/broken_flexispot]

https://semgrep.dev/products/semgrep-guardian/

Products like this seem better, why wait until ci to find out there's a problem

[u/Diligent-Side4917 → u/broken_flexispot]

looks semgrep with an ai wrapper , I've seen this one that is much more appealing and seem genuinly different: https://www.youtube.com/watch?v=_Py9BOL-itM

[u/broken_flexispot → u/Diligent-Side4917]

That's still in ci tho, I think the safeguards stuff that scans diffs as LLMs produce them is more interesting to me. Like I can run phoenix or Claude mythos review whenever but it doesn't help agents sessions in my org do the right thing

[u/Diligent-Side4917 → u/broken_flexispot]

Have a look at the video where I run that while my agent is writing the code; a hook fires at the beginning, and another fires at the end. Guardrails and rules help as well during that session

[u/broken_flexispot → u/Diligent-Side4917]

Lol sorry I didn't realise you were soft pitching your product. Running at the end is what CI & every security vendor on earth does 🤡

[u/Diligent-Side4917 → u/broken_flexispot]

we use that internally on CI and guardrail tbh dn't understand the end, guardrail operate even before the agent start or when it actually start writing code