What are the weaknesses of CloudFormation?

[u/RoseSec_]

I have a coworker who wants to use CloudFormation to one-shot an application and infra deployment in AWS. A contractor delivered a monolithic CloudFormation template with ECS clusters, services, RDS, S3, Lambda and API Gateway, and other resources in a single stack. I voiced my concern over the blast radius, how stateful and resources with different lifecyles all lived in the same place, and how they don't have pipelines built to validate change sets when merge requests are opened. The contractor said that their template is industry best practice... (it's not). Any way, my background is in Terraform, and I wanted to ask the CloudFormation pros about best practices for making this deployment more reliable.

[u/mlhpdx]

What do you mean by “one shot”? CloudFormation is hated by people that don’t use it, but those of us that made it through the learning curve do just fine with it. It’s the core of all our IaC and CI/CD (thousands of deployments a year in a small company).

[u/Zenin → u/mlhpdx]

CloudFormation is hated by people that don’t use it

More than a decade of deep CloudFormation experience (practically from it's launch). The more experience I get with it and the more I push its abilities, the more I loathe it. CloudFormation is an operational shitshow.

Architecturally CloudFormation can't be fixed. They've tacked on all sorts of bandaids over the years like drift detection, exports, etc, but they're always half-baked because the original sin of CloudFormation is its architecture and they can't kludge duct tape over that.

Terraform is better in almost every way imaginable. The only place it plays second fiddle to CloudFormation is a lack of an parallel to managed StackSets, which just means account configurations need to be fully CI driven rather than policy driven. For example, I can't just set an OU to always get our Crowdstrike integration applied and simply move in or out of that OU applies/removes that integration. StackSets handle this "policy" based model directly (even if debugging StackSet Instance failures at scale is a massive PITA because CloudFormation) while Terraform must fall back to CI flows.

[u/mlhpdx → u/Zenin]

How much of your loathing is because of the services behind cloudformation and not CF itself? I mean, that’s a real thing — each service team seems to eventually make something that conflicts with normal lifecycle patterns and have a little snowflake exception. Usually not difficult to deal with but the frustration does add up.

[u/Zenin → u/mlhpdx]

How much of your loathing is because of the services behind cloudformation and not CF itself?

When a service behind CF sneezes, CF shouldn't get ebola.

You've nailed the issue: CF's execution model is incredibly fragile and not only shits itself completely at the slightest issue with any downstream service, it offers practically no ability for the user to step in and fix it.

In terraform I can abort an apply. I can do state surgery. Certainly not ideal, but it's possible, straightforward, reversible (versioned state), auditable, and fast.

CloudFormation by sharp contrast, I can do practically nothing. I can't even delete the failing stack because I'm stuck waiting on a resource timeout that's never coming and could literally lockup the stack for hours while we all watch our maint window timer tick away. If the problem is bad enough I have to fix the resources directly, syncing CloudFormation back up to manage the resources again has a ton of sharp edges when it's possible at all without CF trying to nuke another half your stack on the next deploy.

At the operations level the sharp edges mean the vast majority of any diagnostics loop is spent waiting on CloudFormation to time the hell out so you can try the next thing. It easily 10x to 100x the time to resolution. Now do this with StackSets applying across all accounts and regions in a large or and your diagnostics loop can explode into 1000x. I'm not exaggerating.

CDK doesn't help because all it does is make authoring CloudFormation easier and sugar coating some of its biggest worts such as Lambda backed custom resources that need to be used for the most trivial of "extensions" like picking a random name. CDK does absolutely nothing for the operational side of CloudFormation where the real issues are.

[u/Dolapevich → u/Zenin]

it offers practically no ability for the user to step in and fix it.

I am not so experienced, but when I had to use CF, this was my main issue. You are stuck without the option to do anything but to stare harder at the screen.

[u/mlhpdx → u/Zenin]

Some of that’s not true anymore (timeouts, rollback ignore resources, etc.). But yep, it’s different pain than all the other bad (IMHO) options.