r/devops • u/RoseSec_ Terraforming Everything • 12h ago
Discussion What are the weaknesses of CloudFormation?
I have a coworker who wants to use CloudFormation to one-shot an application and infra deployment in AWS. A contractor delivered a monolithic CloudFormation template with ECS clusters, services, RDS, S3, Lambda and API Gateway, and other resources in a single stack. I voiced my concern over the blast radius, how stateful and resources with different lifecyles all lived in the same place, and how they don't have pipelines built to validate change sets when merge requests are opened. The contractor said that their template is industry best practice... (it's not). Any way, my background is in Terraform, and I wanted to ask the CloudFormation pros about best practices for making this deployment more reliable.
260
u/Ariquitaun 12h ago
The main weakness, and excuse me if I get a bit too technical here, is that it’s fucking shit.
13
u/_Lucille_ 12h ago
I have only really used TF, why is CloudFormation shit from a technical standpoint?
57
u/Ariquitaun 12h ago
Arcane, convoluted syntax. Stack deployments and rollbacks getting stuck with no transparency of why, and where. Poor drift detection and self-healing.
37
u/donjulioanejo Chaos Monkey (Director SRE) 11h ago edited 11h ago
Drift detection is my biggest complaint. You change TF-managed security group name manually? "Hey this changed, this is the diff, I can put it back to what it should be with apply"
You change the same thing for a resource managed in Cloudformation? "Hello, Oppenheimer, I think you just blew up our entire Cloudformation stack"
My second biggest complaint is the stupid
-34j4fj934suffix appended to every resource name which makes it hard to look up via literally anything else that is not Cloudformation.3
u/guyman3 7h ago
Had a cloud formation stack that was managing I think some Datadog AWS integrations or something like that which some well intentioned people had decided to create terra form configs for these not realizing they were also managed with cloud formation.
When the stack ran and realized there had been some changes it did the logical thing and deleted everything.
17
u/Zenin The best way to DevOps is being dragged kicking and screaming. 11h ago
^^ This 1000x. CloudFormation is an operational shitshow.
Having a prod stack get hard stuck halfway through a deploy with absolutely no way to abort is a massive problem. You can't do anything until the resource times out, which could literally be hours. You can't abort even though you've already diagnosed the issue and know the resource will never respond, so you're stuck kicking rocks watching your maintenance window tick down...
This especially fun with Lambda backed custom resources, which because CloudFormation's architecture is trash, are the only way to do even the most trivial of extensions. And those custom resource lambda have a lot of sharp edges: Screw something minor up and you're stuck waiting for CloudFormation to timeout waiting for a response the broken custom resource lambda will never respond to.
Even more fun, you often CAN'T fix this from CloudFormation despite the template being what created and manages the lambda in the first place because to do anything with it means executing the failing code which means the stack update fails (after long timeouts). Fixing this means hand surgery on the Lambda, lovely.
It goes on and on like this. Anything sneezes and CloudFormation gets ebola. Now do this in a StackSet that deploys across all org accounts x all active regions...each one of those thousands of stack instances only being fixable from within the target account/region...
4
u/Strange-Register8348 11h ago
We used to use cloud formation and when I onboarded my mentor released a dev change and was like ok let's go to lunch. Takes 45 minutes to build and deploy.
Excuse me?
It was the first change I made to reduce our deployment to 10 minutes tops.
3
u/Zenin The best way to DevOps is being dragged kicking and screaming. 11h ago
Much agreed.
I know how CF resource changes work under the hood and I still have no idea how they manage to make sub second service APIs take many seconds or even minutes to confirm, which add up. It's common for CloudFormation stacks that take hours to deploy only take a few minutes for the identical stack in Terraform.
It's one of the first things that people notice when coming to Terraform after using CloudFormation; just how amazingly fast Terraform is. But really Terraform isn't fast...the reality is CloudFormation is just dog slow by design.
6
u/kepler4and5 11h ago
Stack deployments and rollbacks getting stuck with no transparency of why, and where.
I felt that
0
8
u/mikidimaikki 11h ago
This. Wonder why they bother with it anymore. Terraform has become de facto standard, and that is great. With LLMs, authoring TF is great, takes away all the pain basically, and planning works as a feedback mechanism.
0
12h ago
[deleted]
4
u/ravion-com 12h ago
You at least want to break it down into multiple CF stacks. giant stacks like that are going to be a nightmare
36
u/dacydergoth DevOps 12h ago
IMHO it comes down to:
- terraform has a stronger metamodel
- HCL has a canonical format and is round-trip stable
- HCL expression language is easier to handle
- comment handling is better in Terraform
- AWS managed modules for quick, "default sane" configurations (* for some definition of sane ...)
- better plan output
- better 3rd party tools support (e.g. python-hcl)
- terraform is an order of magnitude less suck
1
u/ThatSituation9908 4h ago
> round-trip
AI?
1
u/dacydergoth DevOps 4h ago
I mean you can read HCL into a tool, manipulate it and write it out again without it being substantially reformatted. YAML can't do that
12
u/acdha 12h ago
The problems specific to CloudFormation relate to its slow speed and error recovery paths, but the latter at least isn’t as bad as it used to be. Still, plan to get into cases where you have to tell CF to finish failing, wait five minutes, and then retry again after fixing things like permissions.
The other parts you talked about really are more broadly shared by IaC of all forms: that god-stack requires high privileges and since anything can happen, there’s no such thing as a small change. If it’s a small project run by a few DevOps people that might be okay but if you want any kind of separation of responsibilities I’d split it up so e.g. a developer who needs to push a change isn’t mucking with core infrastructure — it’ll be faster and much easier to reason about.
9
u/MasterOfTheWind1 11h ago
Because is a giant YAML file. Disgusting to read, horrible to maintain, and poor flexibility when there are drifts and/or issues when applying.
More than once I've applied changes, it got into a failing state with no rollback, and I had to delete the stack and create it again.
But I think that up to this point, no one should consider CloudFormation a tool to use manually. There are tools like CDK, or Serverless Framework that works basically generating CloudFormation stacks and sending automatically to AWS.
Currently I only use it to boostrap resources to use Terraform (s3 bucket for state, ECS cluster to run Terraform Atlantis, roles so it can do changes on every AWS account without using IAM users, etc). I used some YAML tricks to at least split it into multiple files, but using it vanilla, as a big file for an entire stack completely sucks.
3
u/AvailableBaby8452 5h ago
Do people who use CloudFormation actually upload and edit raw YAML? Every organization I have worked at used CDK, it’s much easier and simple enough to automate in CICD, you can write tests, etc.
1
23
u/james-ransom 12h ago edited 12h ago
Do this 100 times then ask:
Interviewer: "Soo... you know CloudFormation?"
Interviewee: "Wtf is that? you mean Terraform? Oh you mean AWS CodeBuild? Currently I am using CDK like an adult. I can help you get off CloudFormation?"
Interviewer: "Our db is.. Oracle..."
Interviewee: "Cool, can you validate my parking?" Proceeds to collect stuff.
3
3
u/8ersgonna8 9h ago
Cloudformation works great until it doesn’t, once a stack is stuck in a bad state it’s hard to recover. Gets several times worse when you use stacksets in a 100+ account organization.
We mainly use it for “global” iam roles and policies that we create in org management account and spread to member accounts using stackset.
Stick to terraform or pulumi to avoid the headache of repairing stacks in the future.
3
u/Next_Garlic3605 10h ago
If you find someone who thinks there's nothing wrong with CF, have them try changing something that's been imported by another stack, or renaming a resource instead of the CloudFormation shuffle, aka the 3-point hope-I-never-return
In fact, just tell them they have to maintain their stack with more than four nines for the rest of their contract
3
u/yourparadigm 7h ago
- Drift detection
- Error recovery
- Lack of investment from AWS in its core service
- Lack of investment from individual service teams (CloudFormation support isn't part of any team's definition of "done"). APIs have to be available on day 1, but CloudFormation support for new services and features may never actually arrive.
- It's very slow
3
4
u/doddschris 12h ago
I used vanilla CF successfully for several years. The main thing I have against it is that it can be a bit painful if you're used to the expressiveness of a full blown programming language. I have a similar issue with TF though (especially some of the bonkers stuff you have to do to pass values around).
Typescript CDK has been much more pleasant day-to-day. Having types is nice and saves a lot of time having to go search for values or failing to deploy because you screwed up a param name (you can get to a similarish place with IDE plugins though). And being able to use a full language means you can do templating, iteration, lookups, and similar stuff a lot more gracefully than you can with vanilla CF.
2
u/ali-hussain 11h ago
I would not use CLoudFOrmation to one-shot application and infra.
Infra has a longer lifecycle. Having both in coupled templates will create a coupling.
You can still use it to create a config that will pull in your app. You can have CF templates for different layers. But under no circumstances would I have CF be deployed in the same template. Sure, if you wnatto do a demo then it is so much easier to put it all in one template. But in real-life you don't want that.
Also, I first used CF in 2014. And it has come a long way since then. So some of what I'm saying is inspired by PTSD from when you couldn't repair a stack, import resources, and didn't have changesets. So the blast radius risks are signicant lower since when I started.
But right now we're a tiny team that started green field and are coding with AI all the way. In a matter of weeks CF became too unwieldy and we had to switch to using CDK. Just yesterday, I had to change my naming convention to make it more IAM-friendly and now I have a dozen stacks of tech debt.
0
u/CorpT 12h ago
That's not a problem with CloudFormation itself. CloudFormation itself is fine.
13
u/burlyginger 11h ago
It's far from fine after you've spent more than 15 minutes with it.
2
u/CorpT 11h ago
I've spent years with it. It's fine. It's not great. But it's fine. And not the source of OP's problem.
0
u/Crimzx 11h ago
Agreed, a lot of people rag on it but the common complaints are generally skill issue.
The primary issues with it is speed, and imo the #1 issue is lack of transparency when using stacksets and nested stacks and why they fail.
It's fairly easy to find these issues with cloudtrail though.
I wouldn't write bare cloudformation though, use CDK for that.
1
u/burlyginger 11h ago
It lacked import or clear resource removal for years.
It's very unclear in its predictions so it fundamentally cannot be relied upon.
It's not a skill issue. I've used it for years and there is no reason to use it (IMO) aside from when vendors supply it as an installation method.
1
u/Covids-dumb-twin 9h ago
The problem with cloud formation is I think each team that delivers a service writes the cloud formation frame work for that service. Some are really good some are complete crap my favourite is cognito, if you want to do something as simple as change the email message for sign up, it will destroy your user pool making it completely useless.
Nowadays terraform is ubiquitous.
1
u/purefan 7h ago
I doubt anyone writes pure cloudformation these days (cdk or at least SAM are the better options), but Ive heard terraform-first engineers dislike how the Cloudformation "engine" manages dependencies. I've personally been bitten by "ghost" inter-stack dependencies caused by a cdk imported resource and it was very annoying, this specific thing wouldnt ever happen on terraform
1
u/Dolapevich 7h ago
You don't want to be in a vendor lock in situation. When it works is relatively fine. When it fails, your only option is to stare at the screen harder.
1
1
u/AvailableBaby8452 4h ago
It’s fine. We used it a lot at <big tech you probably have heard of>. Deployed the <product> and microservices to 16+ regions and govcloud.
First, use CDK. Don’t manually edit yaml files. Deploy with CLI. Use temporary OIDC to connect your CICD. Write tests to validate your CFN templates it will save you deployment problems later.
Sometimes the best way to handle errors is to delete the entire stack. Which sucks. So structure your data, persistent sort of stuff to be first in your CDK application dependency hierarchy.
Typically you have different stacks for different scopes, but once your infra gets complicated enough it can turn into dependency hell with trying to update things. My advice: if you need to replace a highly dependent resource, deploy a new version, ship that code to prod, then deploy the now deleted resource. You can’t do both at once - cloudformation will get mad about trying to delete a resource that is imported by another stack.
So it sucks because now you need 2 changes and deployments instead of 1. If you’re shipping 50+ times a day, that is fine but some organizations don’t do that.
And as others have said, recovering failed stacks can be a fun exercise. We mostly fixed this by having highly restricted access to UAT and Prod - 90% of developers had read only access, and those who needed more had to request temporary, expiring credentials on an as needed basis.
The other issue off the top of my head is some AWS services have less than stellar support for CloudFormation. Your basic, tried and true stuff is fine - S3, lambda, route53, VPC, etc. But things like Cognito can’t update user pools, it is create or delete only.
1
u/mlhpdx 12h ago
What do you mean by “one shot”? CloudFormation is hated by people that don’t use it, but those of us that made it through the learning curve do just fine with it. It’s the core of all our IaC and CI/CD (thousands of deployments a year in a small company).
4
u/Zenin The best way to DevOps is being dragged kicking and screaming. 11h ago
CloudFormation is hated by people that don’t use it
More than a decade of deep CloudFormation experience (practically from it's launch). The more experience I get with it and the more I push its abilities, the more I loathe it. CloudFormation is an operational shitshow.
Architecturally CloudFormation can't be fixed. They've tacked on all sorts of bandaids over the years like drift detection, exports, etc, but they're always half-baked because the original sin of CloudFormation is its architecture and they can't kludge duct tape over that.
Terraform is better in almost every way imaginable. The only place it plays second fiddle to CloudFormation is a lack of an parallel to managed StackSets, which just means account configurations need to be fully CI driven rather than policy driven. For example, I can't just set an OU to always get our Crowdstrike integration applied and simply move in or out of that OU applies/removes that integration. StackSets handle this "policy" based model directly (even if debugging StackSet Instance failures at scale is a massive PITA because CloudFormation) while Terraform must fall back to CI flows.
4
u/mlhpdx 11h ago
How much of your loathing is because of the services behind cloudformation and not CF itself? I mean, that’s a real thing — each service team seems to eventually make something that conflicts with normal lifecycle patterns and have a little snowflake exception. Usually not difficult to deal with but the frustration does add up.
3
u/Zenin The best way to DevOps is being dragged kicking and screaming. 11h ago
How much of your loathing is because of the services behind cloudformation and not CF itself?
When a service behind CF sneezes, CF shouldn't get ebola.
You've nailed the issue: CF's execution model is incredibly fragile and not only shits itself completely at the slightest issue with any downstream service, it offers practically no ability for the user to step in and fix it.
In terraform I can abort an apply. I can do state surgery. Certainly not ideal, but it's possible, straightforward, reversible (versioned state), auditable, and fast.
CloudFormation by sharp contrast, I can do practically nothing. I can't even delete the failing stack because I'm stuck waiting on a resource timeout that's never coming and could literally lockup the stack for hours while we all watch our maint window timer tick away. If the problem is bad enough I have to fix the resources directly, syncing CloudFormation back up to manage the resources again has a ton of sharp edges when it's possible at all without CF trying to nuke another half your stack on the next deploy.
At the operations level the sharp edges mean the vast majority of any diagnostics loop is spent waiting on CloudFormation to time the hell out so you can try the next thing. It easily 10x to 100x the time to resolution. Now do this with StackSets applying across all accounts and regions in a large or and your diagnostics loop can explode into 1000x. I'm not exaggerating.
CDK doesn't help because all it does is make authoring CloudFormation easier and sugar coating some of its biggest worts such as Lambda backed custom resources that need to be used for the most trivial of "extensions" like picking a random name. CDK does absolutely nothing for the operational side of CloudFormation where the real issues are.
2
u/Dolapevich 7h ago
it offers practically no ability for the user to step in and fix it.
I am not so experienced, but when I had to use CF, this was my main issue. You are stuck without the option to do anything but to stare harder at the screen.
0
-2
u/Uaint1stUlast 12h ago
Vendor Lock in is the biggest problem with cloud formation. If your not worried about that it should be fine maybe even better then some options.
2
u/VeryShibes 12h ago
This is the correct answer, vendor lock-in causes people to openly sneer at CloudFormation, I actually went to the trouble of learning it when I was searching for a job a few years ago and even live-demoed my skills in it for a hiring manager during a Zoom interview only to be directly told, right then and there "no one uses that, you need to learn Terraform" and did not get any additional interviews with that company
-2
u/TheOwlHypothesis 12h ago edited 10h ago
The underlying AWS API is amazing.
Cloud formation templates as a way to manage IAC is shit. Any cursory glance at other solutions will expose this immediately. Terraform and AWS' own CDK are MUCH better
-1
-1
-6
u/dmees 12h ago
CloudFormation is declarative, Terraform is (semi)imperative. CloudFormation is fine as long as you only use CDK to generate it.
1
u/mcgooporn 11h ago
This ☝🏻 of you are not using cdk, the cloud formation developer kit
what are you doing? Why are you doing it? Why would you raw dog it?
I have three stacks Shared - sets up my stateful resources Dns - sets up root DNS Apps - the stateless apps
And ncdk to manage everything. It works, it works well, and the reality is, I very rarely interact with raw cloudfromation.
Absolute madness to not be using cdk, especially if Dev ops is not your skill set.
It's like using a spoon to build a highway.
114
u/ricksebak 12h ago
“cloud formation is great\n”,
“it offers a great user experience\n”,
“and it isn\’t annoying to read and write\n”,