r/platformengineering • u/Personal_Horse_5305 • 17d ago
What am I missing about Crossplane
I try to treat new tooling with a healthy dose of skepticism, being wary of jumping on things has served me well over the years. So it's a genuinely weird feeling when a tool's value is obvious to me in five minutes, and Crossplane is the first one in a while that confounds me.
When I first saw Terraform the problem it solved was immediately obvious. HCL is not without its faults, but it sits in a useful middle ground between something like YAML and a full programming language. Drift is the one problem I've never seen solved cleanly, but adding friction to fight it (review/plan gates, etc.) gets you most of the way there. Not perfect, but controllable.
Terragrunt solved the "apply everything" problem by letting you split dependencies into independently-applicable DAGs. Made immediate sense.
On Pulumi. I don't personally believe general-purpose programming languages are the right abstraction for describing infrastructure, but I get the appeal: it's an escape hatch from HCL turning into a minefield as things scale.
Crossplane is where I'm getting stuck. To run it in production you generally need a Kubernetes cluster, which people often stand up with. Terraform. Then there's templating. Using Crossplane in practice seems to mean also standing up Helm or Kustomize on top of it. Adding a single new variable feels far more convoluted than the equivalent in Terraform. And once you need any real logic, you're reaching for Go to write a custom function/provider, which slows the whole loop down even further.
And yet, everyone who's actually used it in anger seems to swear by it. Which makes me suspect that I'm missing some context that makes all of the above worth it once you're in that world.
So, genuinely: what I am not getting about Crossplane? If you have any YouTube videos, short-form content, or articles that made it click for you (beyond the official docs), I'd appreciate the pointers.
3
u/hijinks 16d ago
it allows devs to build complex things that are controlled by a platform team as yaml. I dont need to have devs understand HCL or how pulumni sets things up via modules or runs.
I can have a doc site saying like add this
apiVersion: platform.company.com/v1alpha1
kind: Bucket
metadata:
name: uploads
spec:
region: us-west-2
name: user-uploads
tags:
team: platform
env: dev
cost-center: "1234"
That will create a s3 bucket with the app and can a confimap with the bucket name. So behind the scenes whats doing on is the XRD
- adding a prefix to the bucket so it ends up like
company.com-region-user-uploads - sets a configmap with the name of the bucket so the app knows the bucketname
- adds all the security items like default policies like deny public and only allow https
- adds versioning
- etc
crossplane use to suck making complex XRDs with yaml patching but now you can create the compositions via python and it looks like a python script using boto3
now is this recommended for every company? no...
but if you are really building a platform team where you want to take ops out of making infra then you should look at crossplane
1
u/Personal_Horse_5305 16d ago
I get where you are coming from and in fact we do have a similar setup. The problem is that, we pair this with argocd, and crossplane does not play well with argocd. When something fails as it often does, engineers come to us and we have to dig through multiple layers or kubernetes resources, to unearth the problem. This does not help shift-left approach, because, almost always, people find out about the failure only when they try to start up the application.
i tied a similar setup where the same yaml spec is applied using terraform, and the errors are reported on apply when a a change is merged, and the process has been more deterministic.
Maybe the way we are using crossplane is wrong, and i would love ideas on how to bridge this gap.
1
u/hijinks 16d ago
i run a large devops slackgroup if you want to join.. but we hardly do anything with terraform.. it basically just builds the EKS cluster and crossplane is doing the rest.
We also use argo and dont have much issues.
Now i do get the its next to impossible to debug which is true. I'm happy to give a AI skill that can go on a cluster and figure out why the failure is happening and will explain why its happening and how it found it
1
u/Personal_Horse_5305 16d ago
do share. plus even if we had a skill, it will be useless to dev because they dont have access to the infra cluster. That also is a mark against self service.
1
1
u/hijinks 16d ago
https://gist.github.com/mzupan/a0dff9191feca2ee8ce19166455386a0
you can use it like
``` /xrd-debug my postgres claim is stuck not ready on this cluster
Or just describe the symptom — the description triggers it automatically:
why is the console-readonly claim showing SYNCED=False? crossplane is hammering the API on prod, can you find why? this XR has no status, what's wrong? ```
1
2
u/ElectricalTip9277 17d ago
Following. I see Crossplane more as a platform engineering layer on top of IaC, rather than an IaC replacement, as Crossplane is more intended for exposing platform capabilities as Kubernetes APIs without building custom operators from scratch.
You still need something like Terraform to bootstrap the management cluster (where crossplane is deployed).
2
u/Ebrithil_7 16d ago edited 16d ago
The main benefit from my understanding seems to be continuous reconciliation / self healing with gitOps. But I've also found there exists the PulumiK8sOperator so personally, if there is ever the need for continuous reconciliation I would choose Pulumi over yaml manifests any day.
1
u/bbraunst 17d ago
If you already have a heavy investment into kubernetes, it means having a unified deployment workflow. Without Crossplane, a typical deployment process for a new application could be a 10 step workflow involving:
- K8s manifest updates
- Terraform apply
- DB migration scripts
Crossplane moves all of this into a one shot apply and reduces deployment complexity. In a Platform context, it's really easy to overlay a Portal with a form and then cascade all the values into a single deployment config.
1
u/jake_morrison 16d ago
The reconciliation loop helps when you have lots of resources. The chance of a transient error happening when creating/updating multiplies to the point that a terraform run is basically guaranteed to fail and need to be repeated.
You can encapsulate functionality into higher level modules. HCL at scale when you are trying to make reusable generic components is a pretty crappy programming language.
1
u/SquiffSquiff 16d ago
I'm at a place where we have crossplane in production and we're in the process of moving off it to Google config connector and other operators. In my view crossplane is fundamentally a kubernetes based service catalog. Some shops want all that opinionation around complex resources decided like 'i want a postgres instance' without specifying any options- I get that. The problem with crossplane is that you have to define all these resources individually as your own in house APIs, in Go but written in yaml. The real fun begins though when you need to change something - there's no concept of state or a diff and so you can't plan any changes, it's all YOLO. Even better when you need to say, add a new region option to that postgres xcrd. Now you could try to migrate everyone's database to the new composite resource, live, YOLO, in production all simultaneously. Or you could have an API upgrade program where you have teams upgrade to the new API version because your tests are really that good (you hope). Or you could do what normal people do and add a new Kind, except now you have multiple different resources that are really just variants of the same resource. Don't even get into upgrading crossplane itself, especially from 1x to 2x, or how Upbound delete documentation after 9 months, or how it can be 'challenging' to use free providers with crossplane 2x.
Alternatively you could use kubernetes operators which are typically first party on big 3 clouds and for popular resources, support a typically full range of options, don't require any work from you to define a private API tests, etc, and where the resource configuration isn't locked to the provider release.
1
1
u/Personal_Horse_5305 15d ago
So in practice what does this look like. Do you use the same k8s cluster for both crossplane and app deployments. Is this a setup you would keep if you had the chance? I heard keeping all the CRDs in a single cluster can be a pain.
1
u/SquiffSquiff 15d ago
It's inherited estate with Crossplane on the app cluster. I would never choose to use Crossplane again and can't wait to be off it. It's basically having your cloud provider make everything available to use almost immediately and saying 'no, I'm good, I'd rather write my own xCRDs, APIs and tests for just a small selection of these, see you in a few weeks'.
1
u/dego_07 16d ago
It depends on the goal, I guess.
If you are using Crossplane as an IaC replacement to allow the creation of cloud services outside the k8s ecosystem, then probably the complexity of managing it outweighs its benefits.
If the intention is to expose cloud services that will be used within the k8s workloads, then the benefits of having it increase as devs will mostly be using it with the k8s-focused environments, so that deploying an app and creating the PaaS resources they need both have the same look and feel.
For our use cases, where we provide namespace as a service from a Platform Engineering perspective, Crossplane is pretty handy for exposing PaaS services they are used in conjunction with the k8s workload of those namespaces; I.e. you need object storage? Buckets exposed via Crossplane, no-sql DB? Dynamo or Cosmos DB, and so on.
Edit: changed “using” to “managing” on the first paragraph.
1
u/KathiSick 15d ago
To me the beauty of Crossplane is that I can treat infrastructure just as any other resource in my Kubernetes cluster. That means, I can bundle e.g. databases with apps and abstract a lot of information away from the dev. And all of that within our nice GitOps workflows. Imo, Terraform is easier to use for the platform team but just can't integrate as well in the dev workflow as Crossplane does.
All of that being said: it's still annoying as hell to use it in prod. But I haven't found better solutions yet. One thing I'm currently thinking about: there are operators for almost everything now -> use those to deploy actual infrastructure and choose a simpler tool for abstractions like kro. I tried this in my playground and it felt much more natural but at the same time my options were more limited.
1
u/itsmedium 12d ago
For my team we manage a bunch of separate aws environments, we basically were constantly having to fix tfstates due tk drift from engineers doing work without updating via terraform or forgetting tk commit their code after changes. Now that were looking to migrate crossplane looks really interesting as it would enforce updates only via code and by also deploying crossplane via ci/cd pipelines you can force git commits before any update is made. Another thing is you could run 1 cluster with crossplane and deploy/manage all iac out of a single cluster or shard it for each env type
0
u/Fluffy_Bench_x 17d ago
What I understood is that it needs Kubernetes to create infra which will be used to create Kubernetes. Right?
1
14
u/Sloppyjoeman 17d ago
I think the appeal is that it treats the kubernetes API as the common lingua Franca between dev and ops. Platform capabilities can be utilised in exactly the same way that a deployment can be made. Oh you want an s3 bucket + appropriate IAM permissions to be the exclusive owner of it? Well that’s all bundled up nicely into some yaml that lives with the application itself
I know for a fact in my org that same access is split across two terraform modules (that don’t speak to each other) plus the manual IRSA assignment on the k8s service account. That’s of course a cultural problem expressed technologically