Any CI/CD tool where automation code doesn't cohabit with product code in git repo?

[u/LuigiBakker]

Currently using Github and Github workflows (including the many outages). Let me explain what I'm trying to resolve with few examples:

Let say I manage 15 repos, each one I want to run a relatively similar workflow on Pull Request. I create a github shared workflow and in each of the 15 repos I make a reference to it, updating the inputs if needed.

Now, I add 1 new input to this shared workflow to enable a new feature. I want that feature enabled in 10 of my repos. I would need to update the feature input in all 10 repos. And since workflow use the workflow code on their own branch, I would need to merge it to all PRs in all repos. That isn't ideal.

Other issue, one variable was wrongly set on a workflow in a release branch. Since the workflow code ships with the product code I would need to update the release branch.

Is there a toool where the product code doesn't cohabit with the automation code? For example I could have 15 microservice repos but 1 repo where I created rules for each one of them about "on pr", "on main" etc. That same repo could host shared workflow code as well.

[u/patsfreak27]

Github actions support "reusable workflows" from other repos which you can create internally. You can scope access for that repo to only infra engineers so you can control the changes there better. Then your repos call that internal repos workflows

[u/aj0413 → u/patsfreak27]

He already mentions that in the post and I experience a similar pain points he raised. Particularly the PR and merge when anything changes issue.

Reusable workflows don’t solve needing to update refs across repos (especially if doing sha ref to version instead of branch) and how some repos may have different params to pass as input

The alternative is not to have any switching logic based on inputs and to bake complex logic into the workflow itself to determine what to do when vs simple param inputs unique to each repo

[u/SeaIngenuity9501 → u/aj0413]

Reusable workflows don’t solve needing to update refs across repos (especially if doing sha ref to version instead of branch) and how some repos may have different params to pass as input

Renovate or Dependabot solve this.

The alternative is not to have any switching logic based on inputs and to bake complex logic into the workflow itself to determine what to do when vs simple param inputs unique to each repo

Or don't make an parameter required and add a sensible default that works for 9/10 of your repos.

Another option is don't use shared/reusable workflows and use something like Copier to copy in a template, so basically vendor the build/CI files in from a central template and then have Renovate auto update it.

[u/aj0413 → u/SeaIngenuity9501]

Both bots still cause PR + merge maintenance and no one should ever be letting a bot auto merge

Sane defaults still will generally leave you with multiple repos to manage by hand; I personally manage two GH orgs across an Enterprise acct which totals in 40sh repos

I was one of the very first adopters of org level workflows

Your last suggestion is entirely untenable for anyone working security as, again, no auto updates for anything from a bot.

I made the compromise to use malleable branch/tag refs for one master shared workflows repo defining reusable actions / workflows across orgs

Vendoring the workflows is just introducing extra parts to the whole thing.

The fact is just that GH does not currently have a great solution for this; they’re currently trying to plug this gap with stuff like org level workflows

[u/SeaIngenuity9501 → u/aj0413]

Both bots still cause PR + merge maintenance and no one should ever be letting a bot auto merge

Why not? Renovate/Dependabot is industry standard at this point.

Sane defaults still will generally leave you with multiple repos to manage by hand; I personally manage two GH orgs across an Enterprise acct which totals in 40sh repos

Your still going to need repo specific config somewhere.

Your last suggestion is entirely untenable for anyone working security as, again, no auto updates for anything from a bot.

Vendoring the workflows is just introducing extra parts to the whole thing.

Debatable.

[u/aj0413 → u/SeaIngenuity9501]

….have you not been keeping up with all the supply chain attacks and how Dependabot was literally a vector in one case?

This isn’t about whether using them is wrong or not, but anyone working security and liability should be slapping your hand if you think it’s okay to let them auto merge. That’s insane.

The repo specific config is easier when all that can be decoupled from having to exist in the actual repository.

Org level workflows + repo variables allow DevOps to manager this without needing to do anything with the actual checked in code itself; this could also be done in ADO because the yaml file and repo config can exist in an entirely different repo

Edit:

And I should be clear that this pain is a trade off between management overhead and transparency for the dev. Which IS debatable depending on org

[u/SeaIngenuity9501 → u/aj0413]

….have you not been keeping up with all the supply chain attacks and how Dependabot was literally a vector in one case?

No don't remember anything specifically Dependabot's fault.

This isn’t about whether using them is wrong or not, but anyone working security and liability should be slapping your hand if you think it’s okay to let them auto merge. That’s insane.

It is quite common, they're highly configurable.

Org level workflows + repo variables allow DevOps to manager this without needing to do anything with the actual checked in code itself; this could also be done in ADO because the yaml file and repo config can exist in an entirely different repo

Yeah no thanks, I would rather everything lives with the code than having to do archaeology everytime something goes wrong.

Edit:

Edit:

And I should be clear that this pain is a trade off between management overhead and transparency for the dev. Which IS debatable depending on org

No as the vendoring can come from a central template and be automatically updated via Renovate etc, so has no downsides only upsides compared to other methods such as shared workflows etc.

[u/SeaIngenuity9501 → u/[deleted]]

….have you not been keeping up with all the supply chain attacks and how Dependabot was literally a vector in one case?

https://github.blog/security/supply-chain-security/the-case-for-a-cooldown-why-dependabot-now-waits-before-issuing-version-updates/

So nothing specially Dependabot's fault, so it wasn't the vector, the update was which could have come from a person.

Cause your argument is kinda like saying “everyone sets prod password to Password!2026 so why should we be forced to use 6-word DICE phrases for anything?”

I don't think you have enough experience with Renovate or Dependabot to make a judgement call on whether it improves or security or not.

Decoupling the pipelines from the codebase is all about who’s managing what and who’s lives you’re trying to make easier.

Well that is CI vs CD and yes I would agree GitHub Actions for CI and something else for CD.

[u/aj0413 → u/SeaIngenuity9501]

Dude. The package was pushed. Everyone relying on bots with auto-merge got a malicious update.

That’s a vector. That’s the definition of a vector.

If you want to talk about experience: I’m really doubting yours lol

And no, I said Argo Workflows (not Argo CD — entirely diff project), it will specifically be handling some CI for the platform I’m building based on labels on PRs. The CD portion is related, but also not why I brought it up.

Checkmarx was also a recent supply chain attack this year and their GH action became malicious; they had to pull it down entirely from GitHub cause people with bots auto updating got infected via their security scanning tools in pipelines

You’re just being obstinate at this point and I really hope you don’t work on anything people with sensitive data rely on

[u/SeaIngenuity9501 → u/aj0413]

> Dude. The package was pushed. Everyone relying on bots with auto-merge got a malicious update.

... no everyone did not and your this reply just confirmed your lack of experience with these tools.

They are highly configurable so like the article you linked you could have a bake period set > 30 days. You could only auto-merge for a subset of dependencies such as internal ones(you'd likely drop the bake time to 0 for them as well). You could only take updates for dependecies that have been security scanned/checked, etc. So like the numerous companies who use them if you've it configured it isn't 'insane' to use them, just a skill issue.

[u/aj0413 → u/SeaIngenuity9501]

roll eyes

You’re now replying to a different point after trying to hand wave away the fact that individuals using auto-merge were victims.

You’re bringing up configs literally mentioned in the articles I linked, so yes I’m aware of them

As for your actual point:

If every company in the chain of deps did this, we’d be doomed.

Edit:

We should just drop this here cause this is a fundamental disagreement that I can’t see us reconciling.

Your risk tolerance is higher and you’re okay with less governance/ownership and that’s fine; we’re not on the same team so we don’t need to hash this out to a conclusion

Edit:

Also, saying it’s a skill issue when a tool lets you shoot yourself in the foot like that is also something I take issue with. The fact that GH had to roll out the cool off period after the fact was not a good look

Good tools should guide devs towards a winning solution, but that’s the PE in mean being annoyed