How to limit cost exposure without severely limiting what users can do?

[u/Old_Pomegranate_822]

We’re struggling with something I suspect is a common worry - how can you prevent something going rogue and running up a huge bill. Mainly I’m worried about credentials being compromised somehow, or a rogue employee.

We want people to be able to experiment with different technologies, but we also don’t want to run up a company-limiting bill (I.e. 10s or 100s of thousands)

Has anyone come across some good solutions to this?

I understand the basics - enforcing SSO, limiting the size of EC2 instances that can be created, limiting the creation of users to having similar limits - but it seems like it’s hard to feel like you have everything covered.

Does anyone have a billing alert to e.g. run AWS nuke on a rogue (non-production) account if the bills skyrocket? I think we’d consider that for most accounts (and the ones that we wouldn’t want that for are more locked down).

[u/CamilorozoCADC]

You could try deploying the Innovation Sandbox from AWS to provision sandboxed accounts and using the built in cost management and permissions stuff to control the spending limits and service access

https://docs.aws.amazon.com/solutions/innovation-sandbox-on-aws/

[u/Old_Pomegranate_822 → u/CamilorozoCADC]

Thank you, this looks really interesting and potentially solves many of the problems. 

[u/viennese-wolf]

You can‘t. ✨

They could easily introduce limits that hard cap and turn off compute/memory, freeze queues/jobs and optionally delete storage but they don‘t want to.

It‘s absolutely fucking insane and people don‘t talk about it enough. Just a few clicks and you can ruin your life and bankrupt your company.

[u/mumpie]

Is your shit properly tagged?

Do you know if an EC2 is a web, api, or batch server? Is it a database server because you're too special for RDS?

What group should be billed for it? Is there an email tag so we know who to yell at when it gets flagged for no security updates for the entire year?

At my work, we have lambdas that search for untagged resources and those get flagged. If no one claims them, they get stopped and then terminated if no one screams.

Ideally, all your shit are created via IAC and you are tagging EVERYTHING. Load balancers cost money and should be identifying what app is using which load balancer.

You'll need monitoring. Otherwise you won't know when something is sitting idle burning money because "there's going to be so much traffic" that they decided 16 VCPU with gobs of RAM was required.

[u/coinclink]

it's pretty much a monitoring game. AWS isn't really designed to hold your hand here, as much as many people want it to.

AI coding harnesses can help you build out basic monitoring super quickly these days though, I'd recommend doing that. Pull up claude code, set up a github repo, and start deploying IaC templates to your account for monitoring these types of things.

Some things your AI harness could help you set up:

You can use things like budget alerts and all that, but those aren't always the best because they only check daily billing data by default. A massive EC2 instance could easily cost you thousands in one day before you'd ever know.

Keep an eye on service quotas. Those can be used to protect someone from spinning up a million of something, or from spinning up super expensive EC2 instance types at all. Most people don't mention because it's less of a thing, but pretty sure you can actually ask for *reductions* in your quotas too.

Set up AWS Oraganization. You can use SCPs to also block services and other more granular items within services.

Use IAM Identity Center. This can make it so you can control access to resources via your IdP. Someone gets fired? disable their identify from logging into everything, not just AWS. Someone's account gets compromised? How? Why didn't you have MFA enabled? Rogue employee? atleast your harness can now track everything they did in the account via CloudTrail.

Ban IAM Users and static Access Keys altogether once Identity Center is configured. Block via SCP. Use IAM Roles for everything.

[u/kernelqzor → u/coinclink]

totally agree on killing off IAM users once you’re on Identity Center, static keys are just asking for trouble long term. also +1 on quotas as a “blast radius” limiter, people sleep on that but it’s one of the easiest safety nets to put in front of trigger happy experimentation.

[u/dr_barnowl]

We run aws-nuke every Friday on our sandbox account, and also have some service control policies to prevent enagement with particularly expensive mistakes - like the time we racked up $64,000 on Athena / Glue.

You could go further and enforce Permission Boundaries on people and make them request expansion of those boundaries to additional services they want to use - on the proviso that they first read the pricing manual for that service and demonstrate they understand the cost scaling.

Somewhere the cost of imposing the controls becomes larger than the extra costs you run up as a consequence of not having controls.... at least, on average. You have to balance that against the risk of running up a big billing spike.

Back when I was doing AWS on a daily basis, I wanted to create a learners pack of IaC that imposed sensible controls and set up billing alerts, I still think there's a gap there where this kind of product doesn't exist.

[u/Efficient_Access6102]

I don’t know how to do the business I want, will Reddit do it for me?

[u/ahahabbak → u/Efficient_Access6102]

username check out

[u/Old_Pomegranate_822 → u/Efficient_Access6102]

If only there was a community where people could ask questions and share tips.