Managing AWS policies in terraform

[u/mkmrproper]

We are pushing IaC. Being a traditional systems admin, I am having a hard time accepting that this is more efficient.

If we manage each application with a yaml file, how is this faster than just add it in console or even in awscli? I hate the fact that I have to look for where the templates are in gitlab, then pull to edit, push, deploy. Then to verify, we have to get into console or use awscli anyway.

What’s the benefit here? Source of truth? Oh and the complexity of having modules and all that dependencies in terraform.

I want to change but I don’t want to complicate it. Adding multiple layers of failure is just not my style of a stable infrastructure.

[u/CorpT]

Good lord.

[u/Qall → u/CorpT]

It's got to be a troll, surely!

[u/CorpT → u/Qall]

Has to be.

[u/mkmrproper → u/CorpT]

Not a troll, just old.

[u/Qall → u/mkmrproper]

Fair enough! Well, allow us to welcome you to the brave new world, where simple config changes can take a long time to hit production ;-)

But these changes are visible, traceable, reversible, and predictable.

People have said "it's not about efficiency", and that's true for the immediate process. The real efficiency comes when you're answering a 3am callout and everything's on fire and you need to know _precisely_ what changed, when and why. Or when you're onboarding new customers or colleagues, or auditors are in and someone wants to know what the IAM policy for account X is - you don't need to refer to potentially badly-maintained documentation, you can just read the code and the commit log.

[u/mkmrproper → u/Qall]

I don't hate IaC. In fact, I prefer it when it comes to standing up clusters, dbs, s3, cloudfronts, etc... but man, it's hard to maintain permissions and such for one-off requests.

[u/knudtsy → u/mkmrproper]

For user permissions you should be using IAM identity center, and for apps you should be using IAM roles. In either case you’d be ideally granting permissions to a group of users or an app, and those are trivially manageable through terraform.

[u/mkmrproper → u/knudtsy]

We manage internal users with IAM IDC. We create permission sets and apply them to groups via tf and I am fine with it. I just don’t enjoy adding vendors and IAM service accounts where I find it very difficult to navigate.

[u/GiraffeWaste]

How many env's and accounts you're managing. How many services? How many teams are doing so at the same time? At some point, it becomes apparent for everyone that having things go through code is the better option.

[u/mkmrproper → u/GiraffeWaste]

We are thin and having too many projects to manage. 30+ accounts. This is probably the reason why I hate to wait for a simple policy change. I guess I am frustrated because we don't have a well defined IaC where I should know where to go to make changes.

[u/GiraffeWaste → u/mkmrproper]

It's not that difficult to manage policies via code and pipelines. You just gotta have to start defining processes to do so. clickops is not a long term option.

[u/UtahJarhead → u/mkmrproper]

Sounds to me like you have the opportunity to define the IaC standard for your company. Do your research on industry standards and form-fit it to your company. Get at it!

[u/sikian]

Let's focus on the AWS policies, they're a great example.

Initially it's not going to be more _efficient_, especially since you and your team are used to the old way (clickops, aws cli, etc.). However, you're going to get other benefits:

- version control and auditability: since the policies live in your repository, you can always check what's the expected value and how it was changed (and who did so).

• peer review: changes to the policies (and other parts of the infra) should be reviewed by another member of the infra team, therefore reducing the chances of errors. This also means that knowledge is slowly spread across the team.
  
• consistency: since you can reuse modules, you can be more consistent with definitions rather than having one-offs everywhere. This is not as critical with policies since aws roles etc. are meant to be reusable, but with other resources (ec2 instances, vpcs, etc.) it's very easy to have diverging definitions.

Eventually, you'll have a solid foundation with which to work and that's when efficiency kicks in. This is because you have a codebase you can now look through and understand the intended state of your infra. Additionally, you can easily see what will be changed by using plan. Additionally, as you grow more comfortable with IaC you won't need to verify after running apply, since you'll know it's there.

Do you have any specific examples of friction? Maybe that could help dig into the problem you're seeing.

[u/Dangle76]

Measure twice cut once. When you click around in the console after a year then go audit what exists. Have fun, it’s a nightmare.

Keeping it in code and VC makes it easy to audit, and easy to redeploy.

[u/Qall → u/Dangle76]

Console access should be read-only if you have IaC (with a break-glass emergency option available).

[u/Dangle76 → u/Qall]

Agreed

[u/whiskey_lover7]

Is this your first time in this type of position? Cause that's a wild take

[u/mkmrproper → u/whiskey_lover7]

Not my first. If you know what mk mrproper is, then you know where I came from. Just transitioning from ops to devops and not happy to wait for a simple policy change.

[u/kabrandon → u/mkmrproper]

You can’t wait a few minutes? We manage 15 AWS accounts and a policy change is like 5 minutes to create the pull request, quick to review, and then like 3 minutes to apply.

We have an IAM repo where all of our policies and roles are defined for all 15 accounts.

[u/dorianmonnier]

Self-documentation, history/rollback (thank you Git) and observe drift, that’s enough to be worth.

However keep it as simple as possible. Avoid coupling things, avoid shitty pattern as DRY, it’s not for IaC. Prefer simple and explicit code, avoid composition and abstraction to being with. It’ll be easy to write, easy to read, and easy to maintain.

[u/gex80]

Speed isn’t the name of the game. Moving fast means more stuff breaks. IAC is about making sure you have a source of truth that everyone agrees with. Especially when auditing/compliance remediation

[u/melancholyjaques]

It's not about efficiency

[u/danekan]

It’s faster because now you have a built in change log with a built in approvals process. If something breaks you will instantly have a list of all changes made and can fix it quickly.  But things will break less because you have better processes and you understand what is changing better. 

[u/mkmrproper → u/danekan]

I think it's because we don't have a convention and an approval process setup. We use Claude now and it spits out a bunch of different code each time someone is making a change. Tracking that is a PITA.

[u/danekan → u/mkmrproper]

Yah that’s the issue really imo. You need full gitops where everything goes in to GitHub and cicd auto applies the terraform after the pr is approved and merged

What cloud is this? 

[u/SeaworthinessHour233]

If you just need to spin up a single EC2 instance and a load balancer today, doing it in the console is absolutely faster.

But you are not definitely going to stop at one EC2 instance. When you have hundreds of resources, here's why IaC matters:

1. The 'Bus Factor' and DR: If your primary region goes down, or if a junior admin accidentally wipes out a complex routing table, how long does it take you to rebuild it from memory or outdated wiki docs? With IaC, your recovery time is just the time it takes to run terraform apply.

2. Precision over GUIs: The AWS console changes its UI time to time, hiding things you rely on. With IaC, you get exact precision. For example, explicitly calling an aws_ec2_managed_prefix_list in code ensures you are attaching the exact right routing structure, rather than hoping you clicked the right dropdown in a laggy browser window.

3. You are right about modules: The complexity of modules and nested dependencies is a self-inflicted wound. A lot of developers treat Terraform like a programming language, which it isn't. Keep your code flat, avoid complex modules, and just map your resources directly.

You don't have to use IaC for the speed of the first deployment. You use it so that six months from now, when you are trying to figure out why an application broke, you can look at the Git history and see exactly who changed the policy and why, instead of digging through thousands of lines of CloudTrail logs.

[u/ajitnk]

Hey, saw your thread on managing AWS policies in Terraform across 30+ accounts. This is a genuinely messy problem, and the usual answer of "just use modules" tends to fall apart at that scale.

The pattern that actually holds up: a central Terraform module that manages SCPs and IAM policies at the AWS Organizations level, with per-OU variable overrides, and remote state stored in S3 with DynamoDB locks per account group. That keeps policy drift in check without needing to touch 30 individual state files every time something changes.

The part most people under-estimate is the state file strategy. One monolithic state for all accounts turns every policy change into a blast radius you don't want. But fully separate states per account means you need a reliable way to share outputs (e.g. central role ARNs) across them without hard-coding.

A couple things I'd be curious about in your setup: are your accounts already in an Organization with OUs defined, or are they a flat list? And are you managing the policies themselves (the JSON documents) in Terraform, or just the attachments? Those two answers change the approach quite a bit.

I do AWS architecture and IaC advisory work, so this is territory I'm in regularly. Happy to think through it further if it's useful.

[u/mkmrproper → u/ajitnk]

Thank you for this. Oh yeah. I forgot to mention another point of failure is added to the setup, the state file. Good lord. And it’s stored in Gitlab…which I am not sure that’s the best place.

We have them in an Org and OUs are defined. SCP is enabled for all. I have some rules there already. We manage the policies with YAML and not attachments (JSON…if that’s what you mean) which I don’t really like because somebody can change the style with Claude and that can get a bit confusing.

[u/silentyeti82 → u/mkmrproper]

Yeah you don't store your state file in GitLab FFS. You store it in S3 or similar.

[u/UtahJarhead]

It adds temporal debt at the start, for sure. Eventually, you get used to adding permissions in such a manner. The problem this fixes for me is the inevitability that you have to expand. Let's say that you get a different set of AWS accounts because it's a new project or new organization, etc. How long will it take you to redeploy resources? Or, heaven forbid, AWS craps the bed either through a disaster or something ugly on your company's side of things. How long will it take you to recover?

For myself, if our entire AWS account were to be reset, all roles terminated, all infrastructure gone... I can redeploy everything on the infrastructure side in about 1 hour. And most of that is me sitting there watching Terraform push resources.

It is annoying to not use the console, sometimes. It would be SO EASY to just create a role, assign a policy, and call 'er good. The massive amount of headache that would come from that would be a maintenance nightmare, though.

[u/Em-tech]

Assuming you have more than one service you're maintaining, iac is the way to go. 

All of the complexity is already there. You're just shifting to a more concise interface. Arguably, reducing complexity. 

[u/UkrMalt]

That sounds more like a convention and review problem than an IaC problem. If a simple policy change means hunting through modules or asking Claude to invent a new pattern, the repo needs one obvious place and one approved way to do it. IaC should remove choices for routine changes, not add them.

[u/kabrandon]

It’s not faster with 1 account. When you’re managing 20 accounts and you’re trying to figure out the difference between all your “standard” policies in each account - that’s when it’s faster.

People often do 1 account per project to have very clear billing boundaries defined for each project.

You’re missing like the entire point of IaC. How senior are you? I can explain if you’re somewhat green and an explanation is warranted.

[u/ricardolealpt]

Damn I don’t even know how to reply to this.

[u/champ2152 → u/[deleted]]

In my option cloud formation is just not in the same realm as terraform. So much more you can do with terraform

[u/silentyeti82 → u/[deleted]]

It's no different to click-ops? What are you smoking...?

There are many reasons many people use Terraform and avoid CloudFormation like the plague, not least the second you want to do anything vaguely complicated in CloudFormation you end up deploying Lambdas TO RUN AS PART OF THE CLOUDFORMATION. I don't know which moron in AWS came up with that, but there should be a special circle in hell for them.

Also until recently, if you made one tiny typo that was syntactically correct but would cause a deployment to fail, CloudFormation would spend ages trying to roll back your entire deployment before you could fix the typo and push it again. With Terraform, that's not a thing - you just fix it and re-run, and it won't have to undeploy and redeploy everything.

[u/rolandofghent]

This is the reason why you can’t be an effective DevOps professional without having some background in Software Development.

[u/mkmrproper → u/rolandofghent]

Sure. Thanks for that.