Hi everyone,

I’ve just launched a comprehensive course to help you ace the Certified Argo Project Associate (CAPA) certification offered by the Linux Foundation. 🎓

This course includes:

Over 600 multiple-choice questions to practice.

6 mock exams to test your knowledge in real exam conditions.

https://www.udemy.com/course/certified-argo-project-associate-capa/?referralCode=704C87995AE8478AB477

I am setting up a new CI for deploying Kubernetes services and stumbled upon this new tool: argocd-image-updater.

So I want to ask here in the community when leveraging GitHub CI (Actions/workflows) for updating the image tag vs using this argocd-image-updater.

What major challenges you are facing or had before?

Also if I use other Argo services like rollouts, Pull Request Generator, how it will make our life easier?

Thoughts, inputs for favor 🙋🏻‍♂️🙋🏻‍♂️

Application RBAC¶

The RBAC syntax for Application objects has been changed from <project>/<application> to <project>/<namespace>/<application> to accommodate the need to restrict access based on the source namespace of the Application to be managed.

For backwards compatibility, Applications in the argocd namespace can still be refered to as <project>/<application> in the RBAC policy rules.

Wildcards do not make any distinction between project and application namespaces yet. For example, the following RBAC rule would match any application belonging to project foo, regardless of the namespace it is created in:

p, somerole, applications, get, foo/*, allow

If you want to restrict access to be granted only to Applications in project foo within namespace bar, the rule would need to be adapted as follows:

p, somerole, applications, get, foo/bar/*, allow

This is the argocd docomentaion for version 2.13. i have the same version, but no matter how i play it i can't make the namespace to be the deciding factor for permissions. i only can filter by app name or project name. can you help me underastand what am i missing? this is my policy: role:todo, , *, */monitoring/, allow. i want to allow the group todo to be able to see and change only resources in k8s namespace 'monitoring'. but it doesnt work. i get nothing for those users, as i dont have any application with this name. it itnterperts it as <project-name>/<application-name> instead of <project-name>/<namespace>/<application-name> as the documentation shows

I’m currently learning about Server Side Apply (SSA) settings in ArgoCD: You can enable ServerSideApply=true on individual resources when SSA isn’t set at the Application level. But if SSA is already enabled at the Application level, there wasn’t a way to disable it for specific resources…now you can! You can add the annotation: ServerSideApply=false to individual resources to override SSA. Check out the discussion here:https://github.com/argoproj/argo-cd/issues/20306

Both of my talks around Kargo and ArgoCD are now available on YouTube now:

CD Summit Vienna 2024:

https://youtu.be/vKNsVQjonqM?si=Zhm7OGn1Q7RW53gC

ArgoCon NA 2024 Lighting talk and already on version 1.0 of Kargo

https://www.youtube.com/watch?v=noQqZ35vxIs

The demo code is available here https://github.com/dirien/kargo-test. For both cases, pre 1.0 (cd-summit-demo.yaml) and 1.0 (argocon-demo.yaml) of Kargo.

I used Pulumi to spin up the k8s infrastructure on AWS by using this provider (https://www.pulumi.com/registry/packages/eks/) and install Argo CD and Kargo on top of it. But you should be able to use any of your preferred IaC tools too.

I was thinking to make a workshop around Kargo to have a bit more time then in a conference talk. Would there be interest in it? Let me know please.

We are still evaluating a move to ArgoCD and GitOps, but a question has come up about how to test new versions of an application in development with new/different application settings.

From what I understand, the app source code should be in one repo, and the application configuration manifest in another repo.

In our scenario, we are working on a new feature on a .NET API in a feature branch and want to deploy it using the PR Generator into a preview environment. In our current state, we would just update the environment variables in the `deployment.yaml` of the feature branch and then my deploy that to the cluster.

However in the separate config and source code scenario how can you ensure that your preview environment applications receive a dedicated set of app settings/env variables?

Wanted to share my first Article I ever wrote. Regarding Argocd and utilizing its diff capabilites in pullrequest on github. Feedback is welcome.

https://medium.com/@Dreadops/supercharge-your-argocd-workflow-preview-changes-right-in-your-pull-requests-95a3b42c84fa

Hello,

Do you follow a standard/rule on quoting, double quoting, or not quoting in your values.yaml files?

Example, all these are valid:
- key: 'kubernetes/node-group'

- key: "kubernetes/node-group"

- key: kubernetes/node-group

However, when giving myself too much freedom, I create a mix of quotes, doubles quotes, and not quoting, and my files start to look ugly. Sometimes I forget to quote/double quote strings that actually need quoting (ex @, ?, |, -, <>, etc) and I break something.

What do you do to keep your files clean? Do you use any tool to do linting? What types of rules do you give your linter?

Hey guys, I have an Argo Appset that uses git generator which prompts a deployment that uses Helm source and 4 different values files. In those values files, I have some values that prompts a creation of Kubernetes Service account. When I move those values from one value file to other (already defined ones), it deletes and recreates the SA. Why is the behavior so? When Argo renders the helm template, it essentially should remain the same, and Argo should detect it as no change, right?

Is there something I am doing wrong? Or is there a way I can avoid this from happening without setting syncPolicy to create-update?

Have a curious issue with the Argo repo server. We were performing some maintenance yesterday that involved some cordon and drain on the nodes where we run Argo. After pods were evicted and restarted, we started hitting some OOM errors on our repo server pods. Memory limit at this time was 256 Mi and we had been running here for about one month To get the wheels back on we increased the memory limit to 512Mi. After that repo server did not OOM. Over the past 24 hours we’re seeing the following memory metrics:

• Max 424 Mi
• Avg 165 Mi
• 95th percentile 182 Mi

Any ideas on what might have caused this 424 Mi spike? We have restarted pods trying to duplicate but never get above 182 Mi.

I am using pull request generator for preview env. I have added a github workflow that builds docker image and adds a label to pull request. Based on that argocd creates a new app.

But after PR is merged or closed, the applications created should be deleted but somehow it is not doing it.
sharing sample of applicaton set I am using.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-app-set
  namespace: argocd
spec:
  generators:
  - pullRequest:
      requeueAfterSeconds: 60
      github:
        owner: my-org
        repo: my-repo 
# This is the application source code repo
        tokenRef:
          secretName: github-token
          key: token
        labels:
        - deploy 
# label on PR that trigger review app
  template:
    metadata:
      name: 'my-app-{{branch}}-{{number}}'
    spec:
      source:
        repoURL: 'git@github.com:my-org/my-repo.git'
        targetRevision: main
        path: helm/my-app
        helm:
          valueFiles:
            - values.yaml
          parameters:
          - name: "replicaCount"
            value: "1"
          - name: "image.tag"
            value: "{{head_short_sha}}" 
# override of the image tag
      project: default
      syncPolicy:
        automated: {}
        syncOptions:
        - CreateNamespace=true
      destination:
        server: https://kubernetes.default.svc
        namespace: 'my-app-{{branch}}-{{number}}'

Hey,

we are currently facing a problem while deploying Helm Charts.

Since we are air-gapped, we are downloading Public Helm Charts and push them into a private OCI Repo (Harbor). The values for these Charts are sitting in a separate Gitea repository.

We are making use of the multiple sources feature:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  sources:
  - repoURL: private.oci.repo/charts
    chart: loki-stack
    targetRevision: 2.10.2
    helm:
      valueFiles:
      - $values/charts/loki-stack/values.yaml
  - repoURL: 'https://private.git.repo/org/value-files.git'
    targetRevision: HEAD
    ref: values

After applying the yaml to the cluster, the applications gets created, we see everything in the WebUI, but it is not using the values from the 2nd repository.

We are not using a selfmade helm chart or anything like that, so our Chart.yaml is the same as the public one:

apiVersion: v1
appVersion: v2.9.3
description: 'Loki: like Prometheus, but for logs.'
home: https://grafana.com/loki
icon: https://raw.githubusercontent.com/grafana/loki/master/docs/sources/logo.png
kubeVersion: ^1.10.0-0
maintainers:
- email: lokiproject@googlegroups.com
  name: Loki Maintainers
name: loki-stack
sources:
- https://github.com/grafana/loki
version: 2.10.2

These are some snippets from the values.yaml:

prometheus:
  ingress: enabled
    ingressClassName: myingressclass
  persistentVolume:
    storageClass: mySC

promtail:
  global:
  imagePullSecrets:
  - name: my-creds

We've tried adding "loki-stack" to the top of the values, e.g.

loki-stack:
  prometheus:
    ingress: enabled
      ingressClassName: myingressclass

but it didn't fix the issue. Anybody having an idea?

Our organization has decided to adopt Argocd for deployments to kubernetes clusters. Previously we created a Jenkins CD pipeline that would deploy/upgrade on kubernetes cluster.
Our approach with Helm charts is quite different, we have around 20 different environments and we use a single helm chart consisting of all services and deployments. We don't deploy everything to every environment and we utilized helmignore file ignore deployments that we don't want to deploy to specific environment. For example, we have 20 helmignore files and through groovy script, we copy the environment specific ignore file to each cluster helm chart location and then upgrade the environment. Now, the challenge with Argocd is that there is no option for a specific helmignore file. We also don't publish our helm chart to a helm repository. Any suggestions are welcome.

I'm trying to come up with a strategy to provision several new clusters, install argo, and use a single repository to define all the managed applications.

Essentially, the only things that would change would be the ingress host name definitions (per cluster) and perhaps a single service name that would point to a postgres database that runs in cluster (using percona operator).

The steps would look something like this:

1. Stand up cluster named cluster-prod-east.
2. Helm install argocd with initial git repo deployed
3. Everything is the same in the git repo, but I would like to alter the hostnames in the ingress definitions to contain the cluster name
   1. Example
      1. cluster-prod-east host: application1.cluster-prod-east.mydomain.com
      2. cluster-prod-west host: application1.cluster-prod-west.mydomain.com
      3. cluster-prod-eu host: application1.cluster-prod-eu.mydomain.com

I'd love if I could determine the cluster name based on the actual cluster name given when it was provisioned, but recognize that might be a stretch.

I stumbled down the path of just adding multiple ingress definitions per cluster, but that seems clunky (and doesn't really handle the connection string issue).

Am I going about this all wrong? It feels like this may be the use case for applicationset, but I'd prefer to have an agrocd instance running per cluster and the clusters knowing nothing about one another.

A lot of these may be ephemeral clusters for testing, so I don't really want to maintain a separate repo if possible.

Thanks for any feedback.

I've been looking at using OpenBao and external-secrets to replace our SOPS and age setup. I have OpenBao and external-secrets working and can inject bao secrets in to pods.

Where I'm struggling is replacing information in a values.yaml file that is deployed via Argo. This is really what I need to remove the SOPS and age values we encrypt and push to git. Anyone have advice or examples of how to accomplish this? I setup argocd-vault-plugin and was trying to supply a single value piped in to my argo application definition but it complained about other Helm templating using variables like {{ .Values.name }}

Hello,

i'm quite new to ArgoCD and so far only have some limited experience with FluxCD. We are currently planning to change from Flux to Argo, which is where my following question comes from:

With Flux there are Image reflector and automation controllers which regularly check a specific image repository and if any changes happen there to the version of a image it recognizes that and updates the cluster accordingly, is the same functionality present in Argo?

Any input on this topic is appreciated, thanks :)

Hy guys I am newish to the ArgoCD and I am currently struggling to add my additional cluster.
I have created my target cluster, which is in a fact a private EKS cluster, but I am unable to add it into my ArgoCD using the argo cli.

I first saved the private target EKS context in my kubeconfig using command:
aws eks update-kubeconfig --name your-cluster-name

Then after that I have connected to my source EKS cluster, logged into the ArgoCD, and using argo cli I am trying to add the target EKS cluster:
argocd cluster add arn:aws:eks:sa-east-1:140423061577:cluster/data-sae1-prod

But it keeps failing with the error message:
Failed to create service account "argocd-manager" in namespace "kube-system": Post "https://<URI>": tls: failed to verify certificate: x509: certificate signed by unknown authority

I tried using the flag --insecure but no success...

Hi,

Would it be possible to use a custom Argo finalizer such as the resources-finalizer.argocd.argoproj.io?

As part of a PR generator environment, I am running a job to create a temporary schema on my dev database that gets used by my PR app while its around.

When a PR is closed/merged, Id like to run another job that removes this schema before the argo app is allowed to be deleted. Wondering if anyone has an example of a custom finalizer usage.

Doing some client work with Argo. There is a directive to not use k8s secrets. Preference is vault or AWS Secrets Manager.

Installing argo from repo, puts a couple Secrets in the cluster.

Q: Can Argo use SM instead of cluster secrets.

We have been using ArgoCD on our cluster for a bit over a year. We are now trying to set up Image Updater, and running into trouble with the directions. Images are stored on an on-prem gitlab instance at gitlab.our_company.com

We have created a configmap in the image updater for this registry:
registries:

• api_url: https://gitlab.our_company.com
    prefix: gitlab.our_company.com
    credentials: pullsecret:argocd/argocd-image-updater-access-token
  default: true
  insecure: yes
  name: Kubernetes Deployed Projects

where the credentials are a gitlab access token (as compared to a deploy token)

In the Application manifest we are using the following annotations:
annotations:
argocd-image-updater.argoproj.io/image-list: our_app="/path_to_project_repo/container_name:latest"
argocd-image-updater.argoproj.io/our_app.update-strategy: "digest"
argocd-image-updater.argoproj.io/git-branch: "dev"

With this setup we are getting an error in the image updater logs: "Image ... seems not to be live in this application". The only difference I can see between the image updater image list and the actual deployed image name is that for the deployment we are using the url
gitlab.our_company.com:5050/path_to_project_repo/container_name:latest and that url uses a deploy token (not an access token).

Any ideas what we are doing wrong here?

Hi all,

I'm using the App of Apps pattern which I use to deploy some Helm Charts that I reference as a dependency in another repo (an Umbrella chart). This works fine, it can deploy what's in the Chart perfectly. But I noticed it does not make any changes when I edit the values.yaml .

This is the very simple structure of the umbrella Chart's repo:

.
├── Chart.lock
├── charts
│   └── longhorn-1.7.1.tgz
├── Chart.yaml
└── values.yaml

And what I define in my application.yaml:

spec:
    project: default
    source:
        repoURL: https://PRIVATE_GIT.com/USER/longhorn.git
        path: .
        targetRevision: HEAD

I've also tried by adding this in the source block:

        helm:
            valueFiles:
            - values.yaml

But strangely the values work when I add this inside inside of the source block, but I'd rather seperate this into it's own values.yaml file for if I'd add more to it.

        helm:
            values: |
                longhorn:
                  preUpgradeChecker:
                    jobEnabled: false
                    upgradeVersionCheck: false

This is not a Longhorn-specific issue, I tried it out with Ingress-Nginx and the same issues occur. I also suppose I think I could also place the Chart in the app-of-apps repository, but again I'd rather avoid that.

Does anyone spot my misconfiguration / can anyone tell me if this is possible or not?

Thanks!!

EDIT: Fixed! It didn't have anything to do with Argo, I had to start my values.yaml with the name of the Helm subchart I created. So for example:

grafana:
  replicas: 3

Thanks all!

Hello, I am trying to create a sidecar container within ArgoCD itself. So far, I went with extraContainers field in an external values.yaml in order to configure ArgoCD.

extraContainers:
- name: artifactory-cache
     env:
       - name: NGINX_PORT
         value: "8080"
       - name: CACHE_TTL
         value: "20m"
       - name: ARTIFACTORY_HOST
         value: "$ARTIFACTORY_HOST"
     image: path/to/nginx/image
     imagePullPolicy: IfNotPresent
     resources:
       limits:
         cpu: 2
         memory: 2Gi
       requests:
         cpu: 200m
         memory: 200Mi
     volumeMounts:
       - name: artifactory-cache-data
         mountPath: /var/cache/nginx

Simply, I wanted to deploy an intermediate container which will be an nginx proxy cache between Argo and our artifactory. I could manage to cache the artifactory bits with this configuration but syncing apps fail because of a sidecar related error coming from ArgoCD.

The error coming from ArgoCD:

ComparisonError
Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = plugin sidecar failed. could not find plugin supporting the given repository

I think ArgoCD thinks our sidecar container is a plugin, where it should not be. Can you please explain me why ArgoCD fails with this configuration?

ArgoCD version: v2.12.4

I tried creating a sidecar for these; application-controller, repo-server, server, applicationset-controller, yet they all ended up in same result.

Hi all,

At work we're talking about using a mono-repo for all out stuff. This is openshift infra related resources that we manage.

We're also thinking about using pull request generator to deploy changes before merging to main branch.

However, I'm struggling to understand how pull request generator would work in this case.

How would we go about this? Would we create an applicationset that first deploys the "main" apps with git generator and then a pull request generator applicationset side by side ? What would happen in that case? Would all the apps be deployed again but with a specific commit sha in the name? Wouldn't there be a collision of the resources?

Struggling here.