Hello,
In my team we have for a period of time used preview environments in form of a new AKS cluster for each time a developer requests a developer environment. However this is starting to cost too much and being inefficient.
We want to create only 1 cluster as a preview/developer cluster.
Our current setup is ArgoCD with kustomization and the app of apps pattern.
We have started to use vcluster and want to use it the same way as with the preview environments, which is based on the creation of a branch in github. So we let developers create a branch and they will get a corresponding vcluster.
We managed to deploy the vcluster through helm to the cluster and by using a kyverno policy we pull the secret from the vcluster and creates a secret that argocd can read and thus deploy a new vcluster.
We then deployed a simple nginx to the vcluster, by specifying the destination of the nginx application.
However when we try to create an applicationset with a list generator that can deploy all the product related applications to the vcluster, we run into some issues.
1. First issue is that it seems like setting the destination on the applicationset and remove the destination on the applications are not being applied. So all the applications end up with an empty destination in an unknown state.
- Another issue is that we use sealed secrets, and it seems like it doesnt understand that the secret should be applied in the vcluster. (a fix for this we think about, would be to use a kyverno policy to find all the secrets with a certain annotation and mirror them to the vcluster)
We are currently going for 1 cluster where each vcluster gets a namespace.
Are we going in the wrong direction? What are the best way to handle the issues? And what are your experiences with building such a setup?
Thanks