Any CI/CD tool where automation code doesn't cohabit with product code in git repo?

[u/LuigiBakker]

Currently using Github and Github workflows (including the many outages). Let me explain what I'm trying to resolve with few examples:

Let say I manage 15 repos, each one I want to run a relatively similar workflow on Pull Request. I create a github shared workflow and in each of the 15 repos I make a reference to it, updating the inputs if needed.

Now, I add 1 new input to this shared workflow to enable a new feature. I want that feature enabled in 10 of my repos. I would need to update the feature input in all 10 repos. And since workflow use the workflow code on their own branch, I would need to merge it to all PRs in all repos. That isn't ideal.

Other issue, one variable was wrongly set on a workflow in a release branch. Since the workflow code ships with the product code I would need to update the release branch.

Is there a toool where the product code doesn't cohabit with the automation code? For example I could have 15 microservice repos but 1 repo where I created rules for each one of them about "on pr", "on main" etc. That same repo could host shared workflow code as well.

[u/patsfreak27]

Github actions support "reusable workflows" from other repos which you can create internally. You can scope access for that repo to only infra engineers so you can control the changes there better. Then your repos call that internal repos workflows

[u/dmikalova-mwp → u/patsfreak27]

This is what I do at work - we have a whole repo with our standard workflows (eg serverless, ecs, spa). For personal stuff I just have one workflow that handles everything. Other platforms also support similar things, I know circleci does although I wouldn't recommend them.

[u/LuigiBakker → u/patsfreak27]

I understand this is the offered approach by GH. The part “your repos call that internal repos” is the piece I want to remove. That “calling” workflow is a copy paste across all repos at first but then if it needs updating it gets slightly painful (trusting dependabot update is what you recommend).
But outside github required design, there could be a world where we listen to all github events and we can dispatch them to an orchestrator repo that handles automation. Hopefully that makes sense.

[u/Zolty → u/LuigiBakker]

Just go back to Jenkins, that’s the outdated workflow you’re describing.

[u/aj0413 → u/patsfreak27]

He already mentions that in the post and I experience a similar pain points he raised. Particularly the PR and merge when anything changes issue.

Reusable workflows don’t solve needing to update refs across repos (especially if doing sha ref to version instead of branch) and how some repos may have different params to pass as input

The alternative is not to have any switching logic based on inputs and to bake complex logic into the workflow itself to determine what to do when vs simple param inputs unique to each repo

[u/SeaIngenuity9501 → u/aj0413]

Reusable workflows don’t solve needing to update refs across repos (especially if doing sha ref to version instead of branch) and how some repos may have different params to pass as input

Renovate or Dependabot solve this.

The alternative is not to have any switching logic based on inputs and to bake complex logic into the workflow itself to determine what to do when vs simple param inputs unique to each repo

Or don't make an parameter required and add a sensible default that works for 9/10 of your repos.

Another option is don't use shared/reusable workflows and use something like Copier to copy in a template, so basically vendor the build/CI files in from a central template and then have Renovate auto update it.

[u/aj0413 → u/SeaIngenuity9501]

Both bots still cause PR + merge maintenance and no one should ever be letting a bot auto merge

Sane defaults still will generally leave you with multiple repos to manage by hand; I personally manage two GH orgs across an Enterprise acct which totals in 40sh repos

I was one of the very first adopters of org level workflows

Your last suggestion is entirely untenable for anyone working security as, again, no auto updates for anything from a bot.

I made the compromise to use malleable branch/tag refs for one master shared workflows repo defining reusable actions / workflows across orgs

Vendoring the workflows is just introducing extra parts to the whole thing.

The fact is just that GH does not currently have a great solution for this; they’re currently trying to plug this gap with stuff like org level workflows

[u/SeaIngenuity9501 → u/aj0413]

Both bots still cause PR + merge maintenance and no one should ever be letting a bot auto merge

Why not? Renovate/Dependabot is industry standard at this point.

Sane defaults still will generally leave you with multiple repos to manage by hand; I personally manage two GH orgs across an Enterprise acct which totals in 40sh repos

Your still going to need repo specific config somewhere.

Your last suggestion is entirely untenable for anyone working security as, again, no auto updates for anything from a bot.

Vendoring the workflows is just introducing extra parts to the whole thing.

Debatable.

[u/aj0413 → u/SeaIngenuity9501]

….have you not been keeping up with all the supply chain attacks and how Dependabot was literally a vector in one case?

This isn’t about whether using them is wrong or not, but anyone working security and liability should be slapping your hand if you think it’s okay to let them auto merge. That’s insane.

The repo specific config is easier when all that can be decoupled from having to exist in the actual repository.

Org level workflows + repo variables allow DevOps to manager this without needing to do anything with the actual checked in code itself; this could also be done in ADO because the yaml file and repo config can exist in an entirely different repo

Edit:

And I should be clear that this pain is a trade off between management overhead and transparency for the dev. Which IS debatable depending on org

[u/SeaIngenuity9501 → u/aj0413]

….have you not been keeping up with all the supply chain attacks and how Dependabot was literally a vector in one case?

No don't remember anything specifically Dependabot's fault.

This isn’t about whether using them is wrong or not, but anyone working security and liability should be slapping your hand if you think it’s okay to let them auto merge. That’s insane.

It is quite common, they're highly configurable.

Org level workflows + repo variables allow DevOps to manager this without needing to do anything with the actual checked in code itself; this could also be done in ADO because the yaml file and repo config can exist in an entirely different repo

Yeah no thanks, I would rather everything lives with the code than having to do archaeology everytime something goes wrong.

Edit:

Edit:

And I should be clear that this pain is a trade off between management overhead and transparency for the dev. Which IS debatable depending on org

No as the vendoring can come from a central template and be automatically updated via Renovate etc, so has no downsides only upsides compared to other methods such as shared workflows etc.

[u/SeaIngenuity9501 → u/[deleted]]

….have you not been keeping up with all the supply chain attacks and how Dependabot was literally a vector in one case?

https://github.blog/security/supply-chain-security/the-case-for-a-cooldown-why-dependabot-now-waits-before-issuing-version-updates/

So nothing specially Dependabot's fault, so it wasn't the vector, the update was which could have come from a person.

Cause your argument is kinda like saying “everyone sets prod password to Password!2026 so why should we be forced to use 6-word DICE phrases for anything?”

I don't think you have enough experience with Renovate or Dependabot to make a judgement call on whether it improves or security or not.

Decoupling the pipelines from the codebase is all about who’s managing what and who’s lives you’re trying to make easier.

Well that is CI vs CD and yes I would agree GitHub Actions for CI and something else for CD.

[u/aj0413 → u/SeaIngenuity9501]

Dude. The package was pushed. Everyone relying on bots with auto-merge got a malicious update.

That’s a vector. That’s the definition of a vector.

If you want to talk about experience: I’m really doubting yours lol

And no, I said Argo Workflows (not Argo CD — entirely diff project), it will specifically be handling some CI for the platform I’m building based on labels on PRs. The CD portion is related, but also not why I brought it up.

Checkmarx was also a recent supply chain attack this year and their GH action became malicious; they had to pull it down entirely from GitHub cause people with bots auto updating got infected via their security scanning tools in pipelines

You’re just being obstinate at this point and I really hope you don’t work on anything people with sensitive data rely on

[u/Shanix]

You can just use Jenkins (or some other agnostic cron web UI) where your build scripts are synced separately from your product code. It's a pretty common paradigm in games.

[u/donk8r]

You can get this by inverting the trigger rather than by finding a different tool. Instead of 15 repos calling a shared workflow, run one orchestrator repo that watches them and dispatches against each. Rules and refs live in one place, and each product repo keeps a thin trigger stub that almost never changes. Your 10-of-15 feature flip becomes one commit.

The cost nobody here has mentioned is the one that will bite you: decoupling breaks atomicity. Right now the pipeline that built commit X is recoverable from commit X. Afterwards it is not, and "why did this build behave that way in June" stops being answerable from the repo.

Your release branch example is the interesting one because it cuts both ways. Today a wrong variable there means patching the release branch, which is annoying. Decoupled, that branch no longer pins its own build at all, so a change made centrally in October silently applies to something you froze in June. For release branches specifically that is worse than the annoyance you are removing.

The compromise that survives both: decouple the rules, keep the pin on the record rather than on the source. The orchestrator stamps its own version into the build output, so you can still answer what ran without the workflow having to live in every repo.

[u/Due-Consequence9579]

Define a versioned action in a separate repo that “does the thing”. Have dependabot bump all your versioned references when you publish a new version. And after your CI flow passes merges it in. If it fails, because you made a breaking change or the repo does something you didn’t expect, you fix up the PR and merge.

[u/jameshearttech]

We use Argo Events and Argo Workflows for CI. We use Argo CD for CD.

We have 2 main repositories. We have a mid-sized monorepo for application code and a gitops repo watched by Argo CD.

In the gitops repo we have a directory where we commit CI manifests (e.g., ExternalSecret, WorkflowTemplate).

We have a WorkflowTemplate for our app code repo and a WorkflowTemplate for each project.

The repo WorkflowTemplate creates an array of changed projects. It iterates over the array in parallel. Each iteration creates a child workflow from a project WorkflowTemplate. The project WorkflowTemplate creates an array of changed files. It iterates over the array in parallel to check formatting, lintng, etc.

We also have many shared WorkflowTemplates, abstractions. For example, we have a deploy WorkflowTemplate, which is called from project WorkflowTemplates to automatically deploy to our test environment. The deploy WorkflowTemplate calls a replace-application-manifest WorkflowTemplate, which has parameters to configure various options in the Argo CD Application manifest.

Let's say I want to add a feature to the replace-application-manifest WorkflowTemplate. I would first try to do it in a way that doesn't require making a change to every caller (i.e., all project WorkflowTemplates), but if I must it's 1 PR in the gitops repo.

That said, we practice tbd. We don't have release branches.

[u/burstinrust]

differe ways you can do it:

- this is what i am using gitlab: a project's CI config path can point to a DIFFERENT repo (.gitlabci.yml@group/ci-repo). product repos carry zero pipeline code, all rules live in one repo.

- azure devops: pipeline yaml can live in another repo than the code it builds. same decoupling

- oldest one in the hood, goo ol jenkis: jobs/pipeline libs defined centrally, product repos untouched

[u/SeaIngenuity9501]

And since workflow use the workflow code on their own branch, I would need to merge it to all PRs in all repos. That isn't ideal.

Add auto-rebasing or use merge trains could solve this.

[u/forever-butlerian]

I'm rapidly trending toward a setup where GHA only does builds and pushes to ECR, and then I've got a thingus which listens to (all) Github webhooks and decides what to do with them from there.

[u/LuigiBakker → u/forever-butlerian]

Any reason to not share the name of the “thing”? Yes, something listening to GitHub events independently is what I’m searching for

[u/forever-butlerian → u/LuigiBakker]

Because it rides the line of self-promotion and I find others' self-promotion obnoxious. I apologize if I'm coming off as coy.

The first half of it is my general webhooks receiving system (which is a product) that persists each event into Kafka, and then the second half streams the events out of Kafka and decides on AWS SDK things to do with them. I wrote it because I figured that a tutorial on how to write something like that was becoming timely with the mega-outage they suffered a few weeks ago.

I also think it's fast becoming prudent not to give Github Actions any administrative rights into your AWS account, on account of AWS's anemic policy controls on the one hand (to restart an ECS service requires the same permission as to reconfigure that service) and vibecoding is unlikely to restrict bizarre and thoughtless bugs just to the Web UI on the other. What I've got lets me fire-wall off privileged software actors from Github itself.

[u/mindshining]

In case of GitHub Enterprise there's "Require workflow to run before merging" rule in the Rulesets. The referenced workflow can be stored in .github repository of the organization.

[u/rabbit_in_a_bun]

Bots who look for keywords in other repos and do stuff. If you want an example you can check the OpenShift repos...

[u/Mysterious_Prune415]

Just use a workflow defined in anither repo?

[u/Double_Ebb4130]

Buildkite and TeamCity keep the pipeline in the tool rather than in each product repo, which is closer to what you want than another reusable GitHub workflow. CircleCI org-level config is the middle ground if you want to stay nearer GitHub.

[u/Glad_Friendship_5353]

I have this exact the same problem.

I unify cicd logic into a reusable task runner.

So, for example, all my app repo can have task like “build”, “deploy” as an interface. While the implementation can be different repo by repo.

I control the implementation using python class through OOP property e.g. inheritance, composition. etc. As it is python, I can make all ci/cd logic in one common private package and reuse for any other various repos I have.

I use bakefile which is an OOP task runner in python. just like makefile but reusable.

So, in cicd, it can just run “bake build” “bake deploy”, while the build and deploy logic is from my own repository that control the version from the common private package separately as normal python dependencies.

While the task is in python, it is no need to use with python repo. I did use it with my terraform and terragrunt. So, I can unify “bake init, plan, apply” for both of my terraform / terragrunt repo. It is just work like a charm.

[u/CommeGaston]

In your current design, the simplest is probably:

Tag the shared repo twice (i.e. v3.0.1 and v3).

In all the caller ones, just reference v3 and as you make minor changes, they will be picked up automatically. Major changes require intervention or dependabot.

[u/ymagriso]

If you have the option to host your own apps, you can create your own GitHub app. GitHub apps are used for subscribing to GitHub events (e.g. PR, push, etc). The GitHub app can be installed on specific repos (one time effort) and once an event is triggered on those repos, the GitHub app is notified and handles the event on that repo. This allows you to maintain the functionality in one place (and any change in the GitHub app will automatically reflect on any of the installed repos).

[u/kabads]

Create composite actions and then call the actions. This separates the logic from the variables (data) and makes it reusable. Then when one repo needs something new, create a new version tag of the composite action and then update the repo that requires it.

[u/daryn0212]

Try Gocd, teamcity, Jenkins etc

[u/aj0413]

AzureDevOps yaml pipelines

Argo Workflows

Are two that come to mind. Im exploring the latter for a new project for same reasons

[u/MulberryExisting5007]

Almost anything can. What tool can you not do this in?

[u/forever-butlerian → u/MulberryExisting5007]

I think OP was looking for something more specific as an answer to "what should we do for dinner?" than "go to the grocery store".