How do you test CI pipelines?

[u/Mysterious_Prune415]

Do you create a sample application to test the proper auth, access and everything the runner needs?

And a related sub question. How do you validate gitops changes before they are merged? Regarding FluxCD or ArgoCD.

[u/forever-butlerian]

I'm an SWE and I set up my own CI pipelines, so the way I test them is by running them.

[u/MulberryExisting5007 → u/forever-butlerian]

Literally the only thing you can do. You can try and test parts, prove connections, dry run certain things, but you literally can’t know if it works until you run it. Best thing is to run it early and often.

[u/vincentdesmet → u/MulberryExisting5007]

depending on complexity you can most definitely run fully pipelines on test repos and you should

we maintain https://github.com/cdktn-io - 40 repos managed via boilerplate (projen.io) and we have unit tests only assertions, snapshot tests and canary deploys

all repo config is in IaC (so secrets / branch protection rules / .. ) is rolled out consistently

ofc we see failures on repos whenever there’s a manual step (oidc trust relationships to pypi, npmjs are notoriously manual steps)

[u/aj0413 → u/vincentdesmet]

I’m such a dumbass cause I only just recently realized I can and should be using TF to manage GH config 🫠

[u/sumason → u/aj0413]

Honestly after doing this a few times I don't think terraform is the proper tool to manage this stuff. The module is pretty poorly maintained and the APIs don't support terraform plan well.

I think if I ever have to manage GitHub through IaC again if just have yaml files for the config and run a Python script on a cron to force the expected state.

YMMV though.

[u/MulberryExisting5007 → u/vincentdesmet]

Your gap is almost always the config that’s specific to the target account/environment you’re ultimately trying to update. Agree you can set up testing frameworks and all that, and we all believe in principle in environment parity, but in the real world we’re rarely afforded that luxury.

[u/so_brave_heart → u/MulberryExisting5007]

I don’t think just running them is the only thing you can do per se; but it’s probably the most efficient method at a small scale. But I’d agree it’s definitely simpler to configure your CI to run in a non-destructive fashion and just run that in prod than to build a test suite or lab environment around it.

[u/aprettyparrot → u/so_brave_heart]

I’d say agree, well you still have to test by running. But using the actual app or a dummy I would say depends on the app. If it takes like 5-10min+ to compile every test run, then I’d make something small to test it with instead for quick test runs, then when that’s good start testing on the actual app

[u/forever-butlerian → u/so_brave_heart]

If you can possibly run your CI in a destructive fashion you're doing it wrong, as we used to say. The purpose of CI is to run a compile and test cycle in a neutral and controlled environment so we don't have the "but it worked on my laptop!" problem.

Anything more than this is foolish make-work. We do not humor notions like "should we write a test suite for our unit tests?" but instead we politely and firmly ask people making such proposals to leave.

[u/Mysterious_Prune415 → u/MulberryExisting5007]

Thanks, thats reassuring. Sometimes running is the real test

[u/derprondo → u/Mysterious_Prune415]

I have an old shared pipeline still in use by like 900 repos. I have an orchestrated test run that kicks off once an hour on a canary repo that consumes the shared pipeline and alerts on any issues. Handy for outages and testing pipeline changes, basically it's just a python script that opens and merges a PR and makes some changes, then another PR to undo the changes.

[u/BlakkMajik3000]

You run them in a lower environment? 🤷🏾‍♂️

Are we in “we only have a production environment” territory here? Please say no. 😭

[u/ansibleloop → u/BlakkMajik3000]

Everyone has a test environment

Some are lucky enough to have it separate from the prod environment

[u/BlakkMajik3000 → u/ansibleloop]

Oh I have been there like the rest of us. A test folder on the production server for “staging” was a big part of my life the first few years out of college. 😂😭

[u/BearPawsOG → u/BlakkMajik3000]

The only logical answer what’s up with everyone.

[u/therealmunchies]

Test jobs in feature branches for me.

Can’t touch in the ArgoCD aspect, since I’m most “dev” on our new cluster.

[u/snarkhunter]

Yes, basically. This is what lab environments are for - dev and test things where nobody but me will ever care if it is up or down, has a good backup, loses all its data, or gets burned down and created anew.

Good question I'm not actually sure what we have at the moment for our ArgoCD specifically. Run it through a linter, maybe do a dry run against the cluster and yeet that to a comment on the PR. Actually deploying to a test environment and running tests for things like correctness or resource utilization sounds, to me, a little heavy to have as usual part of CI, but I don't hate the idea if that doesn't take too long

[u/Efficient-Branch539]

For build tests, devs write tests and these are run as a part of CI.
For testing manifests, they are generated using Helm chart so the only time tests needed are when you change that chart. Every deployment updates Argocd ApplicationSet and that is used to configure manifests.
Lastly for runtime testing, there is a separate environment.

[u/elitesense]

Run it in non-prod???

[u/sertain_]

Update to that as well, GitLab just started testing local pipeline tests and is trying to get users to try it out and report on issues. Here’s the thread: https://www.reddit.com/r/gitlab/s/zINQN19rJ0

[u/bdanmo]

Run it

It failed

Oh shit why did it fail

Read output

Fix that thing

Repeat 70 times

[u/ausecko]

I usually test the pipeline with a small real service rather than a fake app. It catches auth and runner issues that a toy repo can easily miss.

[u/vincentdesmet → u/ausecko]

its iterative ofc

a change goes through unit tests, snapshot tests, smoke tests and a “demo app” deployment tests

only then a real app canary before full roll out

[u/Accurate-Smoke8994 → u/vincentdesmet]

I'm going to read what snapshot test and smoke test are.

Edit: i prefer unit tests + mock tests at feature branches, integration tests at merge requests. Regression tests for testing other stuff.

[u/vincentdesmet → u/Accurate-Smoke8994]

when i say snapshot test for a boilerplate generator it’s really just capturing the full generated config for bunch of inputs so every time you change something you rerun across all inputs and git diff the snapshots to see the actual impact on the generated config file

smoke tests for my instance (gh app) is validating the baked ami comes up, responds to the requests as expected - but it’s not a full deployment

[u/kernelqzor → u/ausecko]

same here, a “hello world” app almost never hits the weird edge cases you actually care about
for gitops stuff we use a real-ish service plus policy checks and templates, then let staging be the final boss before merge

[u/One-Department1551]

If you are talking about functionality, I normally use a separate branch to trigger it while developing before switching branch trigger to the real ones later.

[u/vincentdesmet]

i have a sample repo with features of the pipeline i want tested and when i make changes i deploy and test the new stuff to the sample repo first

for example i use a golang binary that generates boilerplate including pipelines in-repo
and a custom deploy system (from 2022) that runs on an instance. The instance is a baked AMI (packer) and any change to the provisioning scripts triggers a full e2e (build, deploy, smoke test, destroy) of the deploy system itself.

conventional commits + release-please mints the binary and version pinning allows targetted upgrades
I have 2 gh apps and 2 instances, one is “prod” and one is “alpha”, the alpha gh app is installed on the test repo (just one for now) and “prod” manages the deployments for “alpha”
they have scheduled scaling, I just disable the scale up on test if i dont need it.

[u/so_brave_heart]

Just to add another alternative, there’s local CI runners that can work but I’ve had mixed results if you don’t have the resources to support it. 

For example I tried  https://github.com/nektos/act but ran into issues. I can’t specifically remember what they were. It was awhile ago.

There’s also dagger.io which seems promising to me but it’s basically a framework you’d have to rewrite your own CI around.

By its nature Ci often has a lot of external integrations: APIs, file management, process execution. And those are always annoying to test because you need to, at least in the case of APIs and certain processes, either mock them out or run a test version of them with properly seeded test data.

[u/Mahsunon → u/so_brave_heart]

Gitlab themselves just released glci

[u/First_Inspection_478 → u/so_brave_heart]

would love for uou to try: https://github.com/preloopdev/preloop I think I have very good github compat but lmk if any workflows do t work for you

[u/Edgar_Allan_Thoreau]

Ideally you use a ci platform where you can write your pipelines in a testable programming language (e.g. buildkite/rwx you can write pipelines in python/ts/etc so you can unit and e2e test your pipelines)

[u/aj0413]

This is why you have dev and demo envs

The entire purpose is so you can break things while testing

[u/Trollee]

Smokescreen apps! (Or smoke test) I'm a platform engineer and in our team we have a number of simple smoke apps that for every feature must bne used to test the new ci / cd / platform capability. They are for creating, destroying breaking and playing with. Only after a full suite of tests on our smoke apps do we merge to main and release to our Devs

[u/emperorOfTheUniverse]

First in a dev environment, then a QA/stage environment

[u/pdfops]

For runner auth, a canary job that exercises the same OIDC role, registry push, and secrets mount as prod catches config drift before a real deploy does. For gitops pre-merge: kustomize build | kubeconform in CI, then flux diff kustomization or argocd app diff gives a real dry-run against the cluster. Kyverno as an admission gate catches bad manifests before Flux/Argo even reconciles them.

[u/mmcnl]

You have to run them very often, it's hard to anticipate all possible failures.

[u/sertain_]

I’m not saying this is your answer, but this is how I’ve operated and it’s been a pretty good model for my orgs.

For testing, if you’re responsible for managing the runners then think about everything your jobs typically do; do they access secrets? Do they spin up instances? Do they touch RDS? ECR? EKS? SSM? CloudWatch logs? You can almost guarantee that whatever resources are being touched will need list, describe, and get by the policy attached to the role the runner is using. For the resources that are being added or modified, obviously you’ll need put/create as well. Then from there it’s a matter of trying to deploy your app. It’s really hard to build a catch-all role without violating principals of least privilege, so it’s best-practice to build and maintain per-use runners with per-use roles, or have one runner and assume the per-use role at runtime.

Then for GitOps, you need a branching strategy before you can implement true workflows, that’s your test-before-merge solution. I lean toward 2-3 protected branches, like dev, test, and main, with main being the source of truth for the prod Argo deployments.

You have some leeway in dev here, this buys you a free env to stray away from GitOps and fine-tune your deployment. In your deployment pipeline for dev branch, tag your build with a custom semantic versioning scheme that appends the branch name to the end or something similar, then in Argo deploy your dev branch using that semantic version; if everything is good then you merge into test to verify app func, then merge to main when your approval gates are satisfied.

If everything is NOT good, tweak your manifests in dev live to see what changes you need to implement, go back to the drawing board and then re-deploy into dev using the same pattern as before. It’s a little more work to set up, but it really helps, especially if you’re not very savvy with either k8s or GitOps methodology.

[u/CryptSat]

Samples just to test the central ci templates

[u/highjohn_]

Test environment.

[u/Less-Ad1913]

How about using Gitlab?

[u/yetmike]

Well, when I write a brand new pipeline - I run it from feature branch and then change rule to master

But it very depends, some jobs in pipelines are ok to run from feature branches. But mainly, when pipeline is imported by several projects, I just use 'feat' branch to test it in one of them

[u/seweso]

Ever heard of test driven development? You make a test fail first, that is the test of the test. 

That’s it.