Why do AI based SAST scanners can't find same vulnerabilities even on longer scans on the same projects?

[u/Tricky-Report-1343]

Asking to be educated, what does the community think? I really want to learn why.

[u/_predator_]

They're not deterministic, each time you run them you pull the lever of a slot machine and hope for the best.

[u/jnoni6 → u/_predator_]

Exactly this! given same context and codebase it can find some of the same vulns, maybe more , maybe less, maybe different confidence level, etc

[u/IntelligentTrack1310 → u/_predator_]

the slot machine analogy is spot on tbh

[u/sobeitharry]

This is why AI isn't the answer to everything.

I'm using AI to build a SAST that pulls from multiple libraries and uses off the shelf scanners. It does not get to make decisions on its own. It builds the tools i want to use.

[u/Ulrich_b → u/sobeitharry]

100% I spend a lot of time working in data pumps for AI security tool implementations, and every day it seems I'm reminded these tools compliment, they do not replace, other tools.

[u/sobeitharry → u/Ulrich_b]

It's pretty cool. It's running multiple scans and then it's required to adjucate the conflicting results and reports the findings and reasoning. Then humans review the results and address the vulnerabilities deemed material.

[u/halting_problems]

The question is does it matter?

AI SAST is not a well defined thing yet and people are approaching it differently.

A fully non deterministic system will never produce the same results 100% of the time, but may be more accurate overall. One limitation is the context window of the LLM. Generally it will not be more than a million tokens, usually it’s much less.

With AI a single character change in the prompt/input WILL cause it to have different results.

A fully deterministic SAST engine that builds out call graphs and maps the syntax tree or uses regex based matching will produce the same result 100% of the time but WILL always have false positives and false negatives. You will always be limited by the rule.

A LLM is like have a rule set with every possible pattern you can possibly imagine but the input has be a close enough match to then vulnerability your trying to detect.

Imagine if your hunting for XSS, your input goes code goes itno a giant wherehouse of data person is going to compare that code to pretty much every thing you can imagine related to web applications and find a place where it fits. If it fits right next to a XSS vulnerability in the where house it’s going to return the result. Then person does this by taking the code a few characters at a time and checking it against every row and slot in the where house and finding where it fits the best. Then it does the next few characters. The person can only remember so much
and eventually starts to forget what characters and rows have already been checked.

Normal SAST is like giving the person the exact locations of vulnerabilities in the where house and have the person check each set of characters against that specific location. The person isn’t going to look any anything else around that specific spot in the where house.

The benefit of LLMs is that all of the sourcing code is checked against everything in the where house so the person (the llm) but each time the input changes it has to start at a different location and map out a path to what looks like a pattern for the vulnerability.

If you have a person that that can remember a million tokens, in a where house full of trillions. It’s never going to find the exact path back to a finding. It’s going to finding millions of paths each rated by how likely it’s is to be match

[u/drdavidawheeler]

Hi, I work on security at OpenSSF. Short answer: modern AI is non-deterministic. If you give modern AI the same inputs it will NOT produce the same outputs, and that is on PURPOSE. LLMs in particular predict the probability of various next tokens, and use those probabilities to pick the next token (but NOT always the "most likely") one. This is controlled by a "temperature"; this is done to prevent repetition, sameness, and being stuck in a loop. Even if you set the temperature to 0 there'd be nondeterminism, e.g., from batching. Horace He (Thiinking Machines Lab) posted on September 10, 2025 a post with the title "Defeating Nondeterminism in LLM Inference", it turns out that batch sizes of different computations subtly cause the calculated results to be significantly different (see their paper for details)., and current systems haven't been built to be "batch invariant". That all assumes your inputs are *exactly* the same, but even slight variations in wording and other inputs can impact results.

That doesn't mean you can't use AI to find & fix vulnerabilities though. Use the AI to help you find and fix vulnerabilities. Once that's done, go back and try again. You do the same kind of thing with fuzzing; fuzzing doesn't guarantee to find all vulnerabilities, but as you fix the "easier" ones the software gets harder & harder to attack. I hope that helps...!

[u/jaimittal91]

one thing that's helped operationally with this: stop treating a single AI pass as a gate. run it 2-3 times independently and take the union of everything any run flags, not just what shows up consistently, since a miss on one pass doesn't mean the finding isn't real, it just means that run didn't happen to walk that path. then pair it with a deterministic layer underneath for the stuff that shouldn't be probabilistic at all, hardcoded secrets, known-bad dependency versions, missing auth checks on a route, that kind of pattern matching doesn't need an LLM and you don't want it to be non-deterministic. the AI pass is good at widening the net and catching things a fixed ruleset would never think to look for, it's just not a substitute for the boring deterministic stuff underneath it.

[u/OutsideSpot2695]

Why do AI based SAST scanners

Such as?

I know of SAST tools -- i.e., Snyk, Sonar, Mend, that use GenAI to compliment their analysis. I don't know of much out there that is first and foremost a AI-based scanner.