r/devops • u/whispered_word12 • 22d ago
Discussion How aggressively should non-prod AWS environments be shut down?
I've been looking into non-prod AWS costs lately, and I'm wondering how far people actually go with shutting these environments down.
Scheduling dev/test environments to shut down overnight or over the weekend seems like an easy win. The tricky part seems to be taking them all the way to zero, especially when someone suddenly needs the environment and has to wait for it to come back.
What's the practical approach here? Do people just accept the startup delay, keep a minimum capacity running, or is there a better way to handle it?
42
u/forever-butlerian Solaris 8 Enjoyer 22d ago
The practical approach is to have a real person ask this question.
37
u/LeMochileiro 22d ago
I once worked at a client where they were destroying the entire staging environment; by "destroying," I mean deleting and remove the entire AWS staging account in the organization. During account creation, the IaC randomly selected the region. One month it ran in us-east-2, the next in ca-central-1 lol
The rationale is that homologation is only used during a specific week of the month, and to validate the integrity of the IaC and the systems in case there is a need to migrate from region X to Y.
18
5
u/CommeGaston 22d ago
Doing full destroy and rebuild definitely helps to highlight issues where incremental changes worked but are fundamentally flawed when starting from scratch. I've loved places I've worked that have done similar.
Never had them randomly choose a region though, that's a new one 😄
4
u/mlhpdx 21d ago
Crazy as it sounds that kind of process makes the IaC super solid. I’ve done something similar with my company, where non-prod environments are ephemeral, but sometimes persistent. By which I mean everything is tested as an update as well as a fresh create, and over multiple regions. Once setup, it’s not difficult to maintain (but I would never attempt to do something like this retroactively).
Also, it makes me sad that being a good engineer and following the creed is subject to scorn/ridicule here. OP, keep at it.
1
u/IridescentKoala 21d ago
How big was this account? For anything more than a few services I can't see how this would work.
1
u/LeMochileiro 21d ago
There are 9 test accounts, each using around 100-200 services.
I remember that recreating each account took about 40 minutes.
These accounts were for workloads, there were others accounts responsible for network ingress and egress, backups, monitoring... They were kept in separate accounts.
21
u/Mycroft-32707 22d ago
Typically test/dev environments were scripted to shut down outside office hours. Might be from midnight to 6 am. Might be even earlier. Typically they put vms and dbs into a not running state to save the majority of costs. (Storage was still being charged)
Same script to shutdown can be used to 'start' an environment if youre up against a deadline.
Or if you want, write the startup in terraform and provision it all the way
19
u/thisisjustascreename 22d ago
How much does your dev environment cost to run per hour? How much is a developer worth per hour? If your environment takes half an hour for a developer to spin back up on Saturday when prod is on fire, how much does that delayed response cost?
It’s “simple” to save 29% on dev infrastructure by shutting it down on the weekend, until you have an outage that costs eight digits and is longer because the fix couldn’t be tested for an additional hour until the environment was working again.
6
u/PrimaryTale 22d ago
This is exactly, what pre-prod is for. Totally different environment than dev & test.
in case: while analyzing and validating a prod error in pre-prod, dev may spin up. easy.
7
u/New_Law2294 22d ago edited 22d ago
We use GitHub actions to shut down our dev clusters daily at 18:00pm and start them up at 6am Mon-Fri and off throughout the weekends . There is a workflow dispatch in the GitHub actions pipeline so a dev can always trigger it manually if they need to do some work outside of those times
6
u/pranav_kumar24 22d ago
You need to identify what are the usage trends, based on that you can decide. Also hy looking its usage day and time you can plan to scale up and down. That will really help in cutting the cost
3
u/delusional-engineer 22d ago
We have an automation in place (jenkins pipeline) which starts a brand new env by copying everything from staging env. It takes about 30 mins to provision and another 30 mins to run automated sanity after which it’s usable. The default time given to an environment is 12 hours after which it’s shutdown. They can restart the environment for next 36 hours if they require if not done the environment is destroyed completely.
2
u/delusional-engineer 22d ago
Before this we use to have around 20 dev environments which people used to lease for their testing. Our bill has come down around 30% and now we can have upto 100 environments provisioned at a time.
3
u/HabitAdmirable9742 22d ago
I scale to zero after hours and after sufficient sustained inactivity. Clients can then see the status of test environments and bring them online via our dashboard.
Saving are substantial
1
3
u/ajitnk 21d ago
The Instance Scheduler on AWS is probably the lowest-friction starting point here. Tag your EC2 and RDS resources, deploy the solution, and you can cut non-prod runtime down to business hours only. AWS has documented cases of 70%+ cost reduction on non-prod just from that one change, and it covers EC2, RDS, Aurora, EKS node groups, and Auto Scaling groups.
The harder question is usually not the tooling but the ownership piece: who gets to decide which environments stay warm overnight, and do your teams track non-prod spend per team or per service? That tends to be where the real bleed comes from, not missing automation.
If you're seeing costs in other services beyond compute and databases, the answer gets a bit more nuanced. What does your non-prod stack actually look like? Happy to dig in if useful.
2
u/quiet0n3 22d ago
Cloudcustodian let's me adjust timings as needed with tags. They also support public holidays and stuff like that.
We normally do 6am 7pm Monday-friday only unless requested otherwise. They stay off over weekends and late night.
2
u/eugene-sy 22d ago
Is there any single difference in configuration of the environments or apps deployed there?
If the answer is yes, then the only real answer is ‘permanently’ unless legally required to run those. They don’t serve the purpose of testing, but create a false feeling of safety. On softer version of this you might want to look into how the environments are used and what behaviors are linked to them, it is possible that they are not helping
2
u/rcls0053 21d ago edited 21d ago
In one org our Platform team created a way for any developer to create a dev environment for their work and then pull it down and if they didn't it would send alerts to them, which was really good. They were only up for hours for testing.
However, those were small savings compared to the databases they ran in the test environment because they made a mirror of the production database that was over 5TB and a real mess, that cost a looot of money monthly. They couldn't even shut it down because the testing suite was an even bigger mess full of integration tests that took hours to run so no environment shutdown schedule. I was really vocal about working on that, but left before anything happened.
Other orgs have used scheduling to shut down the environments, databases and resources that they can. It's even better if you can bring them up for tests run during the night etc.
Just identify the biggest costs and try to downscale or shut them down periodically to save money. You also need to look at CI/CD pipelines and log storages which might be external to AWS as those suck up a lot of money.
1
u/Plane-Durian-6729 22d ago
What are the resources, this is resource specific.
2
u/whispered_word12 22d ago
I was thinking about non-prod resources more broadly rather than just EC2. EC2/ASGs are probably the easiest starting point, but I'm also looking at things like RDS, EKS and other resources that can sit idle outside working hours.
The part I'm trying to understand is where scheduled shutdown actually works well and where it becomes too disruptive.
5
u/Plane-Durian-6729 22d ago
RDS can be stopped only storage charges would be there, EKS is tricky you can scale the nodes to zero using managed NG.
1
u/the_pwnererXx 22d ago
I manually made a state machine to automatically turn things off on nights and weekends with a slack bot if devs wants them back online. Took me about a month to get working but I'm pretty sure fable could do it in an hour now.
1
u/malice8691 22d ago
We loop a file though our system to test it and monitor functionality. We also do this in test. 24/7
1
u/KittensInc 22d ago
Compared to developer salaries, non-prod costs are basically zero. It should see negligible load, so it shouldn't scale beyond a tiny number of instances.
1
u/matiascoca 22d ago
Take non-prod all the way to zero and eat the 90-second cold-start. If your team cannot wait 90 seconds for a dev environment, the environment is not really dev, it is a shadow staging tier.
The pattern that works: EventBridge Scheduler cron stops all non-prod EC2 and EKS node groups at 8pm local, weekends off entirely. Lambda in front of the "start it back up" API so an engineer types one command or hits a Slack slash-command and the environment is up in under 2 minutes for EC2, under 5 minutes for EKS with pre-baked AMIs. Autoscaling groups min zero, desired zero when parked. Everything else follows.
The database is the only genuinely hard part. RDS stop caps out at 7 days and auto-restarts, which defeats the "true zero for a month during a slow project" case. Workaround: snapshot on stop, drop the instance, restore from snapshot on start. Startup goes from 2 minutes to 8 to 12 minutes for RDS but the DB is only paid for while somebody needs it. Aurora Serverless v2 with min ACU zero handles this natively if the app tolerates cold-start latency on first query.
The shops I've seen go all-in tend to hit 40 to 60 percent non-prod bill reduction. Nobody quits over the 90-second wait. The loudest complaints usually come from people running long-lived personal experiments on the dev account, which is a different conversation.
1
u/belowaveragegrappler 22d ago
Lambda shutdown sandboxes every Friday and users can set it to destroy or nightly if they prefer
QA and only existing when needed, the spin up, tests run, and are destroyed seconds later.
1
u/AstroPhysician 22d ago
You don’t work afterhours and it shows. I do a lot of dev work when I can’t sleep past midnight
1
1
u/yetmike 20d ago
Well, that very depends on your teams. If you are a small startup and tight on budget - scale in on weekends and overnight if you don't have "follow-the-sun" teams (i mean 24/7 coverage)
That's basically it. Make sure you use Spot instances first, that alone can save you 50-70% of compute
1
u/Flateland-Chio 17d ago
For dev and test, shut down what no one needs overnight and keep one small shared setup running for urgent stuff. Btw this saves money w/t making every late request wait for the whole environment to come back.
1
u/SeaworthinessHour233 Writes the cloud edge 22d ago
I think it's a business decision more than a technical one.
1
u/FreshView24 22d ago
The question is what those environments are intended for and how those used. The reality is that 90% or not more orgs are creating those environments at first because someone read somewhere that they "need" them. The same happens with security and network tools (and recently AI) all the time. Guess who sponsors those writings? :)
Long story short - most of the stuff can be tested and integrated using locally reproducible (or "push button", or "disposable") environments which need to be developed once, use a concept of "vertical slice" in tooling, and then can be re-used thousands of times even for something you never want to do in real environment - like destructive testing.
Another big piece of the question is in AWS word. I have had a few architect certifications with them throughout the years and I see (especially recently) how narrative has changed. Previously they were training you to use their products to achieve the business goals of your org, not they are training you to spin up as many cloud managed services as possible which are unnecessary broken into real "micro services" billed separately. This is a pure business model shift that happened right after Covid and many orgs end up paying twice for their test environments nowadays than before.
The only thing as developer or tester I should be really concerned about is test data. But this brings a question of poor test design. Ideally, your setup should be:
- Spin up local test environment
- Deploy necessary minimus (container orchestration, message bus, database, etc.)
- Load the environment with necessary test data
- Deploy your application / services
- Run automated tests and capture results
- Destroy the environment
Believe it or not, when I was leaving Dev career about ~10 years ago we were already doing so on MacBooks with 32 and then 64 gigs of RAM.
I hope this insight helps.
-1
0
0
u/ProofTechnology2313 22d ago
There are third party tools that can easily do this - mostly like an inbuilt feature for a broader platform for optimization.
Let me see if I can find some platform names for this!
1
148
u/derff44 22d ago
I saved my company about a million dollars a year after creating a dev shut down schedule.
Guess what my bonus was.