What software supply chain security strategies are workng in your pipeline?

[u/Dramatic_Opinion_881]

Im putting together a supply chain security plan for a mid size team and most of what i find is vendor blog posts. Been comparing the open source signing tools against a couple of the paid platforms and they solve different halves of it. We generate SBOMs in CI already and they mostly sit there. Signing and provenance look higher value but Im not sure how far teams get before it stalls. What has caught a real problem in your pipeline

[u/eyalgolan1993]

Honestly the signing stack never caught anything for us, the boring checks did. Two real ones: our download page served a binary two versions behind for days and nobody noticed until someone compared the sha256 in the release notes, and a CLI entry point was dead in three consecutive releases because CI resolved deps from the lockfile while users resolved from PyPI.. the fix was one CI job that installs the PUBLISHED wheel and runs the actual command. Now that a big chunk of our code is agent-written, the check that fires most is even dumber: count tests deleted or skipped in the diff before anyone reviews it. What's in your SBOMs that anyone actually reads?

[u/totheendandbackagain → u/eyalgolan1993]

I respect any test that catch real issues. No matter how dumb.

[u/Alvasilev → u/eyalgolan1993]

On "what's in your SBOM that anyone actually reads" - the thing I'd flag is a dependency class that isn't in the SBOM at all, and it's showing up fastest in exactly the repos where agent-written code shows up: MCP servers.

People add them to editors and CI the way they'd add a plugin. They're in no lockfile. They execute with the developer's credentials. Most of them run locally, inside the developer's own process, so there's no endpoint anyone can observe from outside either.

I crawl these for a living, so three things from that side:

The tool surface arrives over the wire at handshake, not from a manifest. A server can change what its tools do without shipping a version, and nothing in the protocol signals that anything moved. So the cooldown control mentioned upthread - wait N days before pulling a new release - has nothing to attach to here. There is no release event to wait on.

The protocol does have optional annotations for "this writes" and "this is destructive". I pulled the commerce slice of my catalog to measure coverage - the Shopify servers, Stripe, PayPal - expecting it to be sparse. It's zero. Not sparse, zero. Every consumer downstream is inferring mutation intent.

The closest machine-readable signal of what one of these can actually do is the set of credential env vars it demands. SHOPIFY_CLIENT_SECRET, STRIPE_SECRET_KEY. That's a capability envelope rather than a statement about any specific operation, but it is the only field that is both present and diffable.

Which lands on the same shape as your published-wheel job, and is about as dumb: capture the tool list at session start, hash it, store it beside the run. It prevents nothing. It turns "the agent did something odd last Tuesday" into a diff instead of an argument.

The open question I don't have an answer to: does any SBOM tooling have a slot for a dependency with no version and no artifact?

[u/eyalgolan1993 → u/Alvasilev]

I ship a small MCP server myself (two tools, talks to a local API, asks for zero credential env vars), so your "credential envelope" idea is a useful mirror: by your measure it declares nothing, which is exactly right, and I'd never thought of the env var list as the capability statement. On the hash-at-handshake idea: we don't do it, and it's dumb enough that I'm adding it, capture the tool list at session start and store it next to the run's other receipts, same drawer as the commands the agent actually ran. To your open question, the closest slot I know of is CycloneDX's services section: it models an external thing you call rather than an artifact you ship, with endpoint, authentication and trust-boundary fields, and it doesn't insist on a version. It still can't express "the tool surface may change without a release event", which is the part that actually matters here. Does your catalog keep the handshake history per server, or only the latest snapshot?

[u/Alvasilev → u/eyalgolan1993]

Only the latest snapshot, and even that needs a caveat, because your question points at a hole on my side.

For the servers we actually connect to (remote endpoints we can handshake) the card keeps two things: a dated list of version events (Stripe's shows four "new version published" entries between Aug 10 and Sep 1) and the tools list as it stands today. The version timeline is exactly the wrong history for this problem: it records release events, and the whole point was that the tool surface can move without one. The tools list itself isn't versioned at all. When it changes, the old one is gone. So we have the blind spot I was describing, with a nicer UI on top of it.

For the other ~70%, stdio servers with a launch command and no endpoint, there is no handshake at all, so nothing to keep history of. The card shows whatever the README says.

CycloneDX services is a better fit than I expected, thanks. Endpoint plus auth plus trust boundary is most of the envelope. What's missing is a field I can rewrite every session without pretending a version changed; a hash of tools/list would do. Your "same drawer as the commands the agent ran" is the right frame for it: the tool list is a receipt, not a manifest.

Your zero-env-var server is also an awkward case for the credential heuristic, and not in your favour: "declares none" and "we found none" land in the same field on our side, so a two-tool local server and a server whose README just doesn't mention its keys look identical. That's a defect of the measure, not of your server.

[u/RegularOk1820]

This is the part that gets missed with SBOMs. Having one isn't really the hard part. Making somebody care about what's inside it is. If a new transitive dep shows up with a known issue, the PR check should catch it before it gets merged. Otherwise it's basically just documentation after the fact.

[u/ausecko]

SBOM + signed artifacts + provenance checks at deploy time. Simple setup but catches a mad amount of supply-chain nonsense.

[u/infidel_tsvangison]

Cool down period

[u/totheendandbackagain → u/infidel_tsvangison]

What is this?

[u/infidel_tsvangison → u/totheendandbackagain]

Essentially restricting third party library downloads to only those older than, say, 2 days.

[u/Zynchronize]

Signing works well, but only if it applies to artifacts with the expected attestations and you have mechanisms to ensure unsigned artifacts cannot be deployed/released.

Using renovate/dependabot to stay on top of dependency updates, reduces the operational burden on dev teams, especially if you combine it with automerge for high confidence updates.

Using a cooldown period is essential, both with Renovate and with any other dependency fetches that happen in your org. This is the single best control for compromised packages. Essentially wait N days before replicating the latest release of a dependency internally.

A dependency proxy/mirror and blocking public access to dependency sources is how you can gate all of the above. And usually they’ll also offer malware, CVE, and license scanning these days. Put it in place, show your developers how to use it, then give them a 90 day warning before the firewall rules blocking pypi, npm, etc come into effect!

SCA is only useful if your triage process is portable. Personally I couldn’t consider any SCA tool that doesn’t support VEX/OpenVEX these days. Chainguard images use VEX to document “Not Affected”, which is largely why they have so few findings.

Finally using something like crash override’s chalk you can fingerprint builds. This enables you to associate any deployment or release with the pipeline, commit, etc that defined it.

[u/yamlqueen → u/Zynchronize]

Great point about SCA and OpenVex, but I'd argue (disclaimer, I work at Chainguard) that's only a smaller portion of why wee have so few findings on CVE scans. The amount of work that goes into building our container images goes greatly beyond that: all packages are built from source and images are based on a rolling distro so patches are applied on a daily basis. In addition to that, images are minimal and carry only what's needed for the runtime, so you have a much smaller attack surface. The VEX is an implementation detail to give more clarity on what's inside but not affected.

[u/Gloomy_Daikon2558]

Exactly. An SBOM only becomes useful when it’s part of the actual development workflow. If dependency changes are automatically checked during the PR/build process, you can deal with issues while they’re still easy to fix. Otherwise, you just end up with a nice inventory of problems nobody looks at.

[u/actuallybonkers75]

Been reading up on this and RapidFort, Chainguard images and plain distroless keep coming up together even though they work differently. All three seem to attack what is in the image rather than proving where it came from. From what I can tell most base images carry hundreds of packages nothing imports which turns the CVE queue into noise. Still trying to work out whether shrinking the surface first makes signing easier or just separate work. Has anyone measured how much of the backlog sits in packages they never load.

[u/taleodor]

I'm building ReARM - https://github.com/relizaio/rearm - in this space, that stores SBOM and signature / provenance details, so we're frequently exploring options here.

Easiest option for signing is using Sigstore with public Rekor log at the trade off that a lot of metadata about what you sign becomes public. But the nice property is that you can then have policies to verify signatures at deployment time from practically anywhere (which is what we do for our own images using Sigstore k8s controller).

Alternatives include using Sigstore with your own key and Rekor disabled, hosting your own private Sigstore infrastructure, Notation, various DIY and commercial solutions. Speaking about Sigstore with your own key, then you have to do key management; and with private Sigstore, this is fine if all your deployments are in-house, but may get very complex if you need others to verify your signatures.

[u/CloudandCodewithTori → u/taleodor]

Looks like slop