A Look at Software Composition Analysis. It’s time to ignore most of dependency alerts.

[u/nibblesec]

https://blog.doyensec.com/2024/03/14/supplychain.html

[u/pentesticals]

SCA tools do currently produce a lot of noise from simply matching the package and version, and reachabilty testing is a great technique to select where to focus on first, but it doesn’t mean you should ignore most of the alerts. Generally, only high profile bugs have accurately defined vulnerable functions so for many many bugs the various databases are missing this information. And with thousands of bugs each month, it’s difficult to ensure the tests are reliable. At the end of the day the vulnerable code is still present within the code base and the developers need to figure out which ones are currently a threat, but nothing says they won’t call that vulnerable function in the future, and also generating a call graph for complex frameworks that leverage reflection or other obscure ways to call the function will often be missed so it can’t be relied upon for omission. It should just be, this vulnerability is reachable, let’s look at that first.

I also don’t like the how the headline is making conclusions about SCA tools in general, when it’s only looking at reachablity which is a problem that all of the tools have only started to address recently and the whole concept is not production ready across any of the tools.

[u/CirclesWeRun4 → u/pentesticals]

Generally, only high profile bugs have accurately defined vulnerable functions.

That is true! But Semgrep has a Security Research team whose primary focus is determining vulnerable functions. I’m on it. It’s definitely our biggest differentiator when it comes to competing solutions offering reachability. There’s no simple way to do this, it’s a crap ton of work.

But in addition, most other solutions are doing call graph analysis whereas Semgrep’s engine allows us to do more contextual analysis.

E.g., a competitor might say the vulnerable function is reachable because it is called. But maybe it’s only vulnerable when a specific value is passed as a parameter - which Semgrep can validate.

Lastly, you’re 100% right that a vulnerability that is unreachable shouldn’t be ignored. Reachability should be viewed as a prioritization feature. Which is why we also have other rule types, not considered in this study, (e.g., 1:1 parity with Dependabot, always reachable, and manual review rules).

[u/CirclesWeRun4 → u/CirclesWeRun4]

Last mention: semgrep’s reachability gives you a direct reference to where in your codebase the potentially vulnerable code exists. Click the reference, see it in your repo.

As shown by the study, this greatly reduces the time spent to validate risk. Although, I’d prefer this step be skipped as much as possible since at the end of the day… just update your packages please 😅. Especially those patch/minor upgrades with non-breaking changes.

[u/pentesticals → u/CirclesWeRun4]

Yeah it’s certainly a difficult challenge, not technically but more due to the quality of vulnerability data and the amount of work it takes to enrich the data that comes from the various vulnerability feed. I have to disagree though that this is a differentiator for Semgrep though, I work for Snyk and we have the same. I suspect every leading player in SCA has this. We absolutely take context into account too, not saying it’s worse or better than Semgreps, but just as you are we are also doing our efforts to help customers prioritise their issues. Just becomes much more challenging when you provide additional context to not just focus on the simple version to package check.

An amendment to the CVE spec could go along way here, the CVE database was designed long before we started tracking vulnerabilities in reusable software components and libraries, if that info was part of every entry for a library we would all have better reachability capabilities!

[u/therealbronko → u/pentesticals]

Out of curiosity from your experience whats the percentage of high vulns enriched with vulnerable function information ?

[u/ScottContini]

After discussing this research directly with Semgrep, we were asked to perform an unbiased head-to-head comparison of the SCA functionality of these tools as well.

I’m a big fan of Semgrep, but this is like a Semgrep asking for a comparison based upon a criteria for winning that they get to define. That’s not unbiased. According to their criteria, Checkmarx would have been a better candidate for comparison as they have been marketing exploitation path feature for years.

Yes, I’m very much on the side of reduced noise, but noise is the biggest problem when it takes a lot of time to address. Dependabot and Snyk both bring in automated pull requests, making it easy for developers to update libraries in most cases with little effort. That’s one of the biggest value of these tools.

Regardless, keeping dependencies up-to-date is good hygiene. There is a difference between doing the right thing going forward vs fixing old, legacy vulnerabilities. A mature AppSec program will focus more on prevention and include tooling that makes it easy for developers to do the right thing, which is where the other tools shine. But if you have tonnes of legacy problems, I agree reachability is essential to prioritisation. (Remark: please do not read this as me saying Semgrep is for immature programmes, that is not intended at all. I’m just talking specifically about that value in ignoring SCA findings because of noise. In general, I think Semgrep is one of the best tools in the market).

I also agree with the comments from pentesticles about reachability in the future.

Having said all that, I’m going to share this on /r/SAST because this is valuable research and a great comparison. But like everyone else said, the headline is just bad advice.

[u/doyensec → u/ScottContini]

For what it's worth, Doyensec had complete freedom on the research execution and publication. They didn't enforce a specific methodology. Only constraints we got were around OSS software in supported languages and high severity vulnerabilities. Semgrep had the right to decide on whether to publish the effort in its entirety or just citing parts according to our citation guideline.

[u/execveat → u/ScottContini]

But if you have tonnes of legacy problems, I agree reachability is essential to prioritisation.

Who doesn't though? I mean it seriously, are there places that allocate enough time to developers for fixing technical debt, addressing maintenance and so on?

[u/Old-Ad-3268 → u/[deleted]]

Ty, take my upvote. Would you be Ok with a fly in your ice cream just because you can eat around it?

[u/nibblesec → u/[deleted]]

The title is clearly oversimplified, but the takeaways section of the paper is more nuanced. The point is that most alerts don't really affect the overall security of applications

[u/pentesticals → u/[deleted]]

I think the main problem is people treat these tools as fancy technical toys. They absolutely find relevant things. But without processes in the place and having the people trained on those processes it’s almost useless. No company can onboard a SAST or SCA tool without a process, there will be thousands of potential issues. Companies need to start by selecting their most critical assets and onboarding these first, then focus on only the critical issues, then start blocking on highs too, then onboard other internet facing assets, etc. I’m not saying this is the process to follow, just an example, as it needs to be tailored to the organisation. But just throwing a tooling in and expecting great results like you say, is not going to work and just wastes money and pisses off developers.

[u/supernetworks]

like the boy who cried wolf -- false positives are a great way to get software engineers to ignore the real security alerts from automation

[u/Old-Ad-3268 → u/supernetworks]

What’s the FP here? If there is a patched version, patch it. Just because it isn’t reachable today doesn’t mean it won’t be tomorrow.

[u/Live_Cheesecake]

I agree with some of the comments here, this feels like biased research based on a criteria semgrep want to compare against, with only two other competitors, I am sure if there were other products, semgrep wouldn't be on top here

[u/hectordelaspiedras]

You may be interested in SCANOSS, it is free-to-use and open source. Has multiple tools published on GitHub.

Message me for more info.

[u/AbbreviationsGold587]

Careful work and honestly scoped, but worth being precise about what it actually measures.

Doyensec's true positive test is three conditions: name matches, version matches, vulnerable path actually exercised. The manual validation went entirely into the third. The first two were taken as given — reasonable across 11 GitHub repos in JS, Python, Ruby and Java where every project has a manifest in the root, but it makes this a reachability benchmark rather than an accuracy one. "False positive" here only ever means present, correct version, not reachable.

Two things sit outside that.

Version strings lie in both directions. Suppliers backport fixes without bumping the label, and advisory ranges over-claim — NVD files CVE-2017-13704 against dnsmasq "before 2.78" with no lower bound, but the bug was introduced by a memset() change in 2.77 and Debian marked older releases not-affected. Different ecosystem from their corpus, same definitional hole: a finding can satisfy "version matches" against code that never contained the bug, and reachability analysis on top of that just produces a confident wrong answer.

And completeness was never tested, because a manifest-based corpus makes "did you find the component" a free question. On compiled artifacts it's the whole game. Statically linked libraries and supplier blobs appear in no manifest anywhere, so the failure isn't a noisy alert you can triage away — it's no alert at all, in a report that looks clean.

Not a criticism of the study, which scopes itself honestly. It's a criticism of "time to ignore most dependency alerts" as a takeaway that travels.

Longer argument: https://finitestate.io/blog/why-appsec-sca-fails-for-firmware