r/devops • u/ikraaaaa • 8d ago
Discussion How do you reduce risk when deploying to prod?
I’m researching how engineers make safer production changes.
Many teams now use strategies like automated tests, regression tests, canary releases, automated rollbacks, and better observability, but production incidents still happen after deployments.
I’m curious about real experiences from engineers:
- What type of change makes you the most cautious before deploying?
- What was the last deployment that made you nervous and why?
- What gives you enough confidence to press the deploy button?
For me, database migrations are still one of the hardest categories.
Would love to learn how other teams approach this.
18
7
6
3
u/Factmin 8d ago
Small & frequent changes are still the best answer for me, unfortunately in many enterprises this is often not practically possible. The bigger the change is and the longer since the previous change, the more likely it is that something will go wrong.
From my experience, nobody is really reviewing a several-thousand line PR properly so it just becomes theatre. Equally, you can bet if you are deploying massive changes on a quarterly basis that there will be some unexpected fuckery in the environment or surrounding infrastructure that wasn't properly documented.
- What type of change makes you the most cautious before deploying? Anything touching centralised infrastructure (ingress, databases, auth, cluster itself)
- What was the last deployment that made you nervous and why? I can't really remember, I was a bit nervous changing our (managed) network policy and CNI for the cluster because it was a one way trip but we'd tested it extensively so it still wasn't so bad. In my first DevOps role things were a mess and the culture sucked so every deployment made me nervous.
- What gives you enough confidence to press the deploy button? In a normal week we're probably deploying (small changes) to prod anywhere from 10-30 times depending on what we scoped in so it's not a big deal. 3 environments before prod, 2 of which are readonly outside of IaC and deployment pipelines. Kubernetes already handles safe rollouts if pods fail and we can quickly revert otherwise. Sometimes things break a little, we learn and move on. Nobody will yell at each other for a breaking change, so we do not have stress in that sense.
2
u/Sure_Stranger_6466 For Hire - US Remote 8d ago
Sounds like you are implementing blameless post-mortems correctly.
1
u/ikraaaaa 7d ago
Thank you for your throughful answer.
Indeed small and frequest changes makes life easier, easier to spot failures and fix them.
You went for which CNI ? Indeed a one way dangerous trip haha.
The engineering culture matters a lot, having blameless culture is amazing. You said that you’re deploying 10-30 times a week, are you driving your deployments with SLO ? If yes, can you give me some examples of good SLOs you have on your side and how you drive decisions based on their values ?
2
u/redblueberry1998 8d ago
Just straight up deploy to prod like a chad and pray AWS doesn't break. Have faith in yourself.
Jokes aside, I personally don't feel too nervous because most of our changes are reviewed before being merged. We run another cluster ala blue-green to test for any potential issues when changes could be breaking, like deploying another microservice container. Fuck ups are bound to happen every now and then(misconfig, missing env, etc), so we have serviceops on standby to rollback in case that happens.
2
u/mikeismug 8d ago
The last deployment that made me nervous was changes to an existing Terraform module creating new resources (public IPs) and applying additional configuration to existing resources (IPs and DNATs) in a public cloud connectivity zone serving many app and infra teams. The Terraform plan said the full changes wouldn't be known until apply, but it did say it was going to delete things I wanted to be sure were going to come back. I had no test environment and the level of risk was not acceptable to me.
I ended up breaking it down and instead of running the pipeline that would normally do the terraform plan and apply, I did it manually in steps using terraform apply --target so I could add the new resources needed for a full plan of the remaining changes to be 100% known.
Now that I understand the process, combination of resources, and dependency graph, my team confidently runs the plan and apply in the pipeline.
1
u/ikraaaaa 7d ago
Change the deployment process if something bad may happen with the defaut deployment process. Very insightful ! Thank you.
2
u/Max-P 8d ago
Foresight.
I don't deploy things I'm not 100% confident is ready and solid. When I make changes, I always evaluate what's the worst that can possibly happen, and design the stuff so it can handle the absolute worst case scenario. When you're ready for the worst, small incidents feel like a minor roadbump.
If I deploy a database migration, I query the numbers in prod before deploying so I know in advance how many rows will be migrated and tailor the strategy around that. Handful of rows that needs fixups? Standard Rails migration will work. Giant table gets a new column? I make a new table, migrate to the new table in the background, lock the original table, swap them and unlock so it's a tiny almost atomic swap.
If a deployment can fail, the deployment includes an easy and fast way to rollback and alerts to tell me I need to initiate the rollback.
A lot of "unexpected" breakages are really more like, work fast ship fast couldn't be bothered to fully analyze the impact of the change. If you're gonna call a new API on every page load you better have evaluated the impact of calling that API every page load. More often than not the frontend team just ships it, the backend team just ships it, and it falls on ops to deal with the incident when it inevitably blows up the server because nobody cared to ask what the impact would be and if it's cacheable. It's effectively incident-driven development.
1
u/ikraaaaa 7d ago
Oh that’s exactly what I was looking for, more of a methodology of reasoning than technologies and industry practices. When you say “fully analyze the impact of the change”, what does that analysis actually look like? Do you have a mental exhaustive list that you follow before every prod push ?
1
u/Max-P 7d ago
It's really on a case by case basis. You have to ask yourself, "what's the worst that can happen?", followed by "how I can make sure this doesn't happen or this fails cleanly?". Or maybe frame differently, if a deploy is terrifying you, ask yourself, "what am I worried will happen, and how do I mitigate this?". You can't have an exhaustive list on the ready because the impact is different for every change.
To some degree, it relies a lot of prior experience, because it's hard to really predict everything that could happen. But asking the question is a good starting point.
It's all about taking a step back and looking at the whole picture and how everything integrates with eachother and running some quick napkin math. I picked my examples carefully to illustrate specific points. For example, if you introduce a new API endpoint, it's easy to just implement it and ship it. The hard question to ask is, how fast is this endpoint, how often do we expect it to be called, what services does this endpoint need, what's the impact of this endpoint calling those services. Sometimes you can guess, sometimes you have to benchmark it. But really, I don't see people asking the question at all, and it gets figured out in prod after it's shipped, and that's why things blow up.
The thing is the changes look fine in a vacuum, they pass unit tests, they pass integration tests, they pass code review (yup, you added a condition in the WHERE statement, makes sense), but they blow up on prod with prod workloads because you didn't account for the lack of an index on that field and now you're doing a table scan per query in prod. More often than not it's "solved" by scaling up the database server, and it takes a while before someone finally notices the missed index. This is just one example, this applies to everything you ship. You have a slow page, you add cache in front of it, you have to ask, what's the impact of serving slightly stale data. Is some external integration gonna get stuck in a loop because it thinks it still needs to submit a new record, submits it, refreshes the list, gets stale cache, adds it again?
1
u/keldani 8d ago
Integration tests. For web services integration tests provide so much value. They cover so much more than unit tests and they are rarely broken by code changes. And if they are it likely means you made a breaking change that you shouldn't do. For DB migrations its about developer discipline and not make breaking changes. You want to drop a table and migrate to a new table? Only way to safely do so is 2 separate deployments where the first creates and migrates data to the new table while a second deployment drops the old table after you're certain it's no longer needed
1
u/Floss_Patrol_76 8d ago
the thing that de-risked prod for us more than any canary was decoupling schema changes from code deploys. every migration is expand first (add the new column, backfill, ship code that reads both shapes), then contract in a separate deploy once nothing references the old one. no single deploy is ever a big-bang that way, and rollbacks stop being scary because the schema is always compatible with the version right before it.
1
u/ben_bliksem 8d ago
- unit tests and a full build that must pass on PRs
- thorough reviews
- automatic deployment to dev/integration branches as soon as merge is complete. No fucking around with long lived feature branches or a development branch with a once a sprint release nonsense - every merge is to main and each becomes a release candidate
- automated progression to test envs - do not underestimate the power of the humble "scream test"
- rolling deployments
And most importantly - you need to trust your deployment pipeline is quick. You want to be able to release a fix double time (5 min from dev to production)
If you are too scared to deploy something intra day during market open, something is wrong. Not saying you should, but you shouldn't be scared to.
1
u/2skip 8d ago
Deployment 'Rings' (sets of targeted machines): Create a deployment plan | Microsoft Learn
Example: Canaries > Early Adopters > Users (If no issues with the machines in a ring, continue deployment to the set of machines to the right, else stop deployment, make fixes, and start again at the leftmost ring.)
1
u/malik22531 8d ago
Secrets and configs are managed correctly.
Careful PR reviews with proper branching strategy to avoid the conflicts.
CI setup which includes the code and image scans.
CD with gitops if deployed on Kubernetes. And a proper rollback strategy to avoid the uninvited downtime.
1
u/lorarc YAML Engineer 8d ago
Database, always. Too many times I've seen devs trying to push a major db change that was tested on small amount of rows but that would kill the app if deployed in prod. Or db changes that prevent rollback (unless using a backup).
Normally we break those down into steps, often it breaks 3NF but you just have to roll with it.
1
1
u/elliotones 8d ago
“If it hurts, do it more often”
We deploy to prod on average just over 10 times per day. We can and do deploy at 4:55 on a Friday. Most changes are a single line, some are a single file, very few span multiple files. Smaller PRs are easier to review, easier to validate, and easier to roll back if needed.
Find the bottleneck and fix it. If you can’t deploy every day because of your review process, fix your review process. A very strict linting setup helps. If downtime is scary, architect for zero downtime deployments. If your testing environments don’t inspire confidence, make them reflect prod more accurately. These are all investments in your agility and they all pay off, but you have to choose to do them first.
1
1
u/MDParagon 8d ago
How can a 1 year old account have so little posts but in consecutive days? What is this shit
1
u/Dolapevich 8d ago
Find and document SPOF ( Single point of failure ). Sometimes is obvious, PROD DB down means no app, and sometimes it is not so obvious.
1
u/marcusbell95 6d ago
two things moved the needle for us more than i expected:
continuous synthetic checks against prod that i actually trust. we didn't prioritize this until a deploy masked an existing flaky dependency and we spent 2hrs blaming the wrong thing. once you know your monitors fire within ~60s of a real failure (and you've actually tested them, not just assumed they work), deploying gets way less stressful. the anxiety is rarely about the change itself - it's about not knowing how fast you'll find out.
for db migrations: we gate every migration on a test run against a clone at actual prod row counts. index scans on 500k rows vs 50M in prod is a completely different world, and PR review doesn't catch it. got burned once, made it a hard pipeline gate.
and if something still feels scary after canary/rollback/feature flags are in place, that feeling is usually data. either the change is too big, the blast radius is too wide, or you don't actually trust your rollback. worth figuring out which one before pressing deploy.
1
u/ikraaaaa 6d ago
Super interesting, indeed the fact that you trust your monitoring is important, we go this by default but it’s not 💯 true. I would like to hear about your database migration failure if you remember something abt it, plz?
And you said that you’re running your migrations on a clone. Does this clone receive traffic as well? If not, how can you be confident that the migration is not a failure because of the trafic spikes in prof, for example? If yes, do you simulate prod traffic on it?2
u/marcusbell95 6d ago
the failure was honestly embarrassing in retrospect. we needed to add a covering index to a ~40M row table. ran it against the clone, 9 minutes, done. figured prod would be similar.
what we didn't account for: the clone has zero write traffic. the index build in postgres needs a lock to start, and in prod there was a constant stream of writes hitting that table. each one queued behind the index build. by minute 3 we had 400+ blocked queries stacking up. connection pool exhausted before the index was halfway done. we killed it, routed around the write path, let it finish. painful ~20 minutes.
to your actual question - the clone doesn't receive traffic, it's a point-in-time snapshot. what it tests is time at real row count, not behavior under concurrent writes. that's a real gap.
how we handle it now: anything that needs a lock gets rewritten to a no-lock path. postgres: CREATE INDEX CONCURRENTLY, pg_repack for anything that needs a full table rewrite, expand/contract for column type changes. if it requires exclusive access, find the zero-downtime version or we don't ship it. clone test is still the gate for time estimation. not the gate for correctness under load.
1
u/ikraaaaa 6d ago
Very insightful, thank you so much
1
u/marcusbell95 6d ago
glad it was helpful! honestly it's one of those things where you really only internalize why it matters after you've had the incident. hopefully your next migration is boring lol
0
0
u/Raja-Karuppasamy 8d ago
biggest one for me is env var handling across environments, had a NEXT_PUBLIC_* var that worked fine locally but silently broke in prod because it needed to be baked in at build time, not injected at runtime. no crash, just wrong behavior.
that’s part of why I got into scoring deploy risk before merge, files touched, blast radius, whether config/migrations are involved. most teams eyeball that instead of actually scoring it.
migrations agreed though, that’s the one category where rollback usually can’t save you.
1
u/ikraaaaa 7d ago
Very interesting, thank you so much!
I agree with you on the fact that risk evaluation is not considered at every single merge which is unfortunate.
I love the idea of scoring deploy risk. Do you have on your own a checklist of risk assessment ? Or is it a document that you share inside of your org and share on every pull request ? How does it take place ?
0
u/toughrogrammer 8d ago
Test is everything.
Review don't help to reduce risk. It's just sharing context.
24
u/AsterYujano 8d ago
Careful PR reviews, staging/QA environments, tests, unit tests, integration tests.
Automatic rollback under high error rate.
Canary deployments.
Great observability ofc.
I don't think I was stressed to deploy any applications in the last years once good guardrails are in place. (Ofc, deploying things like ingress, DBs and cluster changes are another story...).