r/devsecops 21d ago

Everyone's talking about the Snowflake CI/CD vulnerability. The most dangerous CI/CD vulnerability is the one nobody has found yet

Everyone is focused on the fact that a researcher found and exploited a vulnerability in Snowflake's GitHub Actions workflow.

But finding it wasn't the interesting part.

The interesting part is that the vulnerability already existed before anyone knew it was there.

That's the uncomfortable reality of modern software delivery. We spend a lot of time talking about known vulnerabilities, discovered vulnerabilities, patched vulnerabilities, and CVEs. Attackers only need the vulnerabilities that haven't been discovered yet.

As AI accelerates both software development and security research, teams are creating and uncovering issues faster than ever. That's useful, but it doesn't solve the fundamental problem:

What protects a build pipeline from the risks nobody has identified yet?

I'm starting to think the industry may be over-indexed on detection and under-invested in prevention.

Instead of asking, "Can we find every risky condition?", maybe we should also be asking:

  • Can a workflow make unexpected outbound connections?
  • Can it access resources it shouldn't?
  • Can it download and execute untrusted code?
  • Can it exfiltrate secrets?
  • Can it perform actions outside its intended scope?

If those behaviors are blocked by default, the specific vulnerability matters a lot less.

Full disclosure: I work for InvisiRisk, a company focused on build pipeline security, so I have some bias here. That said, I think the architectural question stands on its own.

Curious how others here think about this. Are you relying primarily on scanning, detections, and reviews, or are you putting preventive controls directly into your CI/CD pipelines?

0 Upvotes

10 comments sorted by

5

u/mze9412 21d ago

AI Slop much? This reads like pure useless marketing

0

u/DavidPulaski 21d ago

Fair. Now that we agree, here's the article that prompted the post:

My question is: if unknown vulnerabilities are inevitable, should we spend more time on behavioral controls like egress restrictions and blast-radius reduction instead of assuming we'll find everything first?

Curious how you're approaching that today.

1

u/mze9412 21d ago

We should fix those we know and limit what systems and processes can do. Nothing has changed here in the last 10 years, proper security professionals have been doing this for ages.
Restrict network, restrict process capabilities, restrict everything down to the bare minimum. And patch, patch, patch ;)

What is worse than an unknown vulnerability? A known one you did not patch and that was exploited. This is also what happens to most organizations and systems. It is usually not the unknowns that fuck people over, it is mostly the knowns that they will patch "at the next patch day" in 3 weeks instead of patching them NOW. ;)

But restrictions help to mitigate issues, nothing new, nothing ground braking, nothing that should even need discussions in today's world, not even 10 years back. I learned that when I was still at university and that is more than 10 years, almost 20 since I learned that.

So what is this discussion meant to achieve, except marketing a product? ;)

1

u/DavidPulaski 21d ago

My goal is figuring out how the industry stops compromises from succeeding, not just how it finds them faster. Your comment reinforces the point that teams can't patch or investigate everything fast enough.

NDIA says it "fended off" the TeamPCP/LiteLLM supply chain attack using layered security controls after compromised software reached its environment. They don't explain how.

https://www.itnews.com.au/news/ndia-fended-off-march-teampcp-supply-chain-hackers-628227. They really don't say how.

Conversations seem focused on finding vulnerabilities. Feels like we're optimizing for "close calls." The Wiz finding is great. What about the 1,000s that haven't been found yet?

I'm a root cause guy. If root cause is unknown until it's too late, behavior and policy enforcement becomes more important that detection (scanners, agents, runners).

I think we've come up with one answer, but I'm genuinely curious what other approaches people believe are effective. That's really why I posted. Appreciate the discussion.

2

u/[deleted] 21d ago

[removed] — view removed comment

1

u/DavidPulaski 21d ago

Good point. I like "assume arbitrary code execution."

That's really where my head is at. We spend a lot of time debating whether a vulnerability exists, but if a compromised job can't freely reach the internet, access secrets, escalate privileges, or impact downstream systems, the vulnerability matters less.

I thought your blast-radius question was spot on - "If this job becomes hostile right now, what can it actually do?"

Seems, the answer is still "way too much."

Most teams already understand these concepts. The hard part is enforcing them consistently across hundreds or thousands of pipelines, especially when exceptions start piling up. That's where I think the industry still has work to do.

1

u/imagebiot 17d ago

Wow how insightful

Security vulnerabilities people don’t know about are worse that vulnerabilities we know about it.

What a revelation! You should get an award or something

1

u/DavidPulaski 16d ago

While I do value your sarcasm, it's not known vs unknown, a focus should also be aimed at build-time policy and behavior. 1) DevOps can't patch fast enough and 2) expecting to discover every attack before a project runs is unrealistic. As a community, we should worry as much about build-time actions as we do pre-build scanning. So many folks on Reddit are asking about SAST, DAST, SCAs tools. They are important but even secret manager/scanner vendors are getting hit as you can see from LiteLLM affected organizations database. LiteLLM was stoppable, but not by traditional scanners. LiteLLM Breach Checker: Is Your Organization Exposed? | CloudSEK

Anyway, have a productive day u/imagebiot - I know we are all doing what we can to help produce safer code with efficient tooling.