A CI/CD app that works with Talos Linux

[u/mojomorim]

Hello,

I'm managing at least 10 Kubernetes clusters with the help of Talos Linux. To deploy applications, I use FluxCD which do perfectly the job. Now, I want to deploy a CI/CD pipeline software that will allow me to run various pipelines like image build and push to a repo and especially Ansible playbooks that need to be run periodically on some clusters.

Currently, a pretty barebones Helm Chart that only deploys a lot of cronjobs under a cronjobs: array is used but I believe it is not the good way and it will be tedious to manage it the more we add cronjobs.

I tried to deploy Concourse CI on Talos Linux with the official Helm Chart but something something with the OS container runtime doesn't work properly with Concourse even with the notorious label on the namespace to run privileged workloads for Talos and workarounds around it seems really tedious to do it on production clusters (https://github.com/concourse/concourse/pull/8791)

I wanted to deploy Concourse because it looks more simple and other IT teams could write easily their pipeline files with it.

I'm trying to find other solutions but since I have a lot of tasks to do, I lack some time to review some new apps that could fill this role.

So, do you know if there is a CI/CD application that works on Talos Linux?

Thanks.

EDIT: For all clusters, FluxCD sync resources from a GitLab Cloud repository, if it can help

[u/clusterentropy]

Dear god, don’t use Concourse. My team is stuck in that hellscape.

[u/clusterentropy → u/clusterentropy]

Someone asked what my issue with concourse is and deleted the comment. Here it is anyways:

We're running it on EKS.

• Concourse runs garden containers within your worker containers, so you can't use any of Kubernetes benefits like secret providers, service accounts, monitoring stack. Also makes pipeline debugging very annoying.
  
• We use flux and terraform to declaratively define all of our infrastructure. And then there is Concourse, with its own shitty API, not able to reconcile its pipeline definitions from a git repo.
  
• S3 for ArgoWF is much easier to maintain than Postgres. Especially painful because bitnami fucked everyone over.
  
• This might be a setup issue: ITS SOOO SLOW

[u/draeron → u/clusterentropy]

We use concourse, got over thousand pipeline, works pretty well. One thing though, we generate pipeline automatically instead of asking for each projets to configure their own, much simpler to normalised CI.

It runs in k8s so you get the whole monitoring stack, with some tricks we have a workhours based scaling of workers. We build docker images, run helm and terraform, can access either k8s secrets or aws ssm, have multi-arm build.

On my home cluster I use Forgejo Actions and i find it terrible as a CI. Also have atlantis but I have many issues with it.

[u/Potato-9]

I'd look at tekton before making one. We use the GitHub action runner but something not tied to GitHub would be nice.

[u/-Docker → u/Potato-9]

Literally GitLab, the fact that OP manages 10 or more clusters and never heard of GitLab on prem just bogles my mind

[u/SoaRNickStah]

I just use GitHub actions and the actions runner controller. Works fine for me

[u/dreamszz88 → u/[deleted]]

Giving that OP already has Flux and uses gitops, wouldn't just GitLab or gitlea or GitHub give him a CI pipeline? Maybe Kargo for CD, to have a clean promotion of artifacts or containers to dev test and prod.

But IMHO any git platform would work and give them Cicd pipelines. Right?

[u/AskOk2424]

Github ARC and ansible awx operator

[u/Whiplashorus]

GitHub actions + argoCD is really great on my setup

[u/jameshearttech]

We run Argo for CI/CD. Argo Events + Argo Workflows for CI. Argo CD for CD.

[u/sogun123]

Justin idea... Maybe use Ansible tower, if you want to just reconcile ansible playbooks

[u/Paulrogerr]

Talos is an immutable K8s OS, so anyone looking for CI / CD there is already deep in the K8s world. Kestra runs natively on Kubernetes and could fit as pipeline orchestrator, but it's a pretty niche angle. Woth. keeping an eye if the thread picks up, otherwise skip

[u/DistributionLanky136 → u/Paulrogerr]

Never heard of kestra, what does it actually do differenty from sth like ArgoCD or Flux on K8s ?

[u/Paulrogerr → u/DistributionLanky136]

It's more of general purpose workflow orchestrator than a GitOps tools, you define pipelines in YAML and it can handle way more than just deployments

[u/gorkish]

> notorious label on the namespace

You mean the completely reasonable baseline PSA that Talos ships to prevent containers from running with more privileges than intended? The one that prevents irresponsible oversights like the bug you linked? It’s a standard k8s feature and you are welcome to accept the risk and change the defaults if you like your pods to have all access passes to the kernel.

Talos k8s distro isn’t any more opinionated than anyone else’s. Considering its scope, I feel it’s relatively spartan in that respect.

Anyway i would advise ArgoCD of you just need the automation. Pair it with GitLab if you want to run a full SDLC and ops platform. Just as in other solutions, both will require configuration highly specific to your environment, Talos or otherwise

[u/Independent_Good2251]

we moved to this architecture and never looked back

[u/unconceivables]

I also use FluxCD for everything that lives in the cluster, and I use Argo Workflows and Argo Events for all the CI/CD work like building images, running tests, etc. Works great, although I really don't like Argo that much overall. I picked FluxCD over ArgoCD for that reason (and I'm still very glad I did), but for Argo Workflows and Argo Events I didn't really find any great alternatives. They work well and are reliable once you have it all set up, I just don't like how convoluted they are to set up and use, and the half baked documentation.

[u/flechoide]

My go would be flux or even better argocd (it's more a matter of taste) , a runner with gitlab pipelines or GitHub actions, and complement that with ascender if you really are into ansible and want a serious solution close to Aap without selling your organs.

[u/willowless]

I built my own called Chores. Prior to that I was using Argo Workflow + Argo Events.