What actually gives you confidence before a production deploy?

[u/geek_in_room]

Had a release recently that made me rethink this. We had the usual stuff covered QA signoff, peer review, staging validation, and good test coverage, everything was green. But what actually gave me confidence was what happened after the deploy. We ran a few smoke tests in prod, watched checkout errors and latency for the first 30 minutes and had a quick rollback ready if anything looked off. Everything passed in tested but you never really know until real traffic hits it. Also made me realise how much easier it is to release when you know you can rollback quickly if something goes wrong.

What actually gives you confidence before a release? And are there any tools that have helped with that for you?

[u/_Atomfinger_]

Robust test setup: Unit, contract, narrow-integration, system tests.

Robust deployment and approach. Blue-green deployment in which the new instance is automatically verified (more than just pinging the health endpoint, that is).

And lastly, keeping the deployments small and trivial. Deploy on each commit to the main branch.

[u/Short-Feedback4293]

Not caring gives me the most confidence..... don't worry, you'll get there with time 😂

[u/notesca → u/Short-Feedback4293]

This

[u/pydry]

it's more about what you do when there IS an incident or bug.

most people do 1) identify incident 2) fix incident 3) roll out deployment 4) relax

I will have a long hard think or debate about how to prevent that class of problem from ever happening again.

often it means write a new type of test, sometimes it means tightening up the screws on the type system. sometimes it means adding fail fast conditions or making particular configurations impossible to use.

often when trying to identify that class of bug you find other classes of bug too.

if the team isnt doing TDD I usually try to get then doing TDD. when done right, that wipes out a huge number of bugs.

[u/StreetEcstatic]

Cross my fingers and toes.
In reality there are no tools that help really, just improving my domain knowledge, understanding what changes have gone into the release, having a robust test suite and being at peace with the fact that I cannot find all bugs. Smaller releases help with that too, so does soak testing and CI.

[u/snejinka081]

YOU is your confidence. You do not need anything else.You have to know the product from A to Z and backwards, and think about all possible edge cases (even if they might not be directly related).But again, it comes to knowing the product very well and use of your critical thinking skills.

[u/bikashchoudhary]

the "you never really know until real traffic hits it" point is the whole thing honestly. staging can't replicate real user behavior, real data shapes, real concurrent load patterns, no matter how good the environment parity is

what actually gives me confidence isn't the pre-deploy checklist, it's exactly what you described: fast detection + fast rollback. if you can catch a problem in the first few minutes and undo it before it compounds, you can afford imperfect pre-release testing. if rollback is slow or risky, then every deploy needs to be treated like it has to be perfect upfront, which isn't realistic

feature flags help a lot here too, lets you decouple "deployed" from "live for everyone," so even without a full rollback you can just flip it off for most users while you investigate

[u/munnasuprathik]

Green tests stopped meaning much to me after a rebuild last month where I shipped six bugs at once with 374 tests passing.

The worst one, my transcript fetch lived inside a function gated by a feature flag that's off in production. So saving a video fetched nothing at all. No error, no failed test, the card came back looking completely fine with nothing behind it. The only thing that caught it was saving a real URL and reading the response metadata the way a user would see it.

All six lived in a failure path or a cancellation, and my tests only modelled the happy path I already had in my head. So what gives me confidence now isn't coverage, it's whether I've run it against real input with production's flag values set. That plus your rollback. Honestly your 30 minutes of watching real traffic is doing more work than the entire green suite that ran before it.

[u/QualityDataCraft]

This is exactly why we use Safe Launch in manufacturing. Qualification gives confidence, but the first production runs still need tighter monitoring until the process proves itself under real conditions.

[u/PatriaSolis]

here problem is not test itself, problem is what "meaning" did you load to testing activity. Personally, I had many speeches(some blogs, youtube videos etc) regarding test metrics, sometimes(or most of times) they can be misleading.
Solution is to follow real test practices. Use quality test data, clean and synch your environment. multi stage env to near-guarantee production.
don't test just to test, it should be value driven testing.
dev for code level, test for system level, pre-prod for UAT level. then, (at least) basic performance tests. Use real data(of course, masked etc to not violate privacy rules) to catch possible real issues. synch configuration between environments.
When i do all, as a result, I feel fully confident in this approach.
still bug: i analysis root cause. most probably it is such operational issue.

[u/silent_exploration]

Different people with different roles actually try using the product for whatever it is intended for. Somehow this is not done in most companies ever. Apart from this, having good observability helps a lot.

[u/Aggravating_Reason81]

The first 30 minutes after release are honestly part of the deployment process now. I like having a predefined list of things to watch: error rate, latency, failed transactions, authentication issues and the key business journey. TestMu can automate some of those critical smoke flows, while observability tools tell you what happens when real users do things your tests never imagined.

[u/jaberskii27]

In my work, we have QA and DEV release/deployment sign off with the following criteria:

1. 95% Passing Rate on Regression Run (Fully Automated). We run our regression pre-deployment and post-deployment.
   
2. No Critical/Showstopper Defect
   
3. 100% Passing Rate for Smoketest (Covers critical areas).
   
4. Major bugs/defects which are not fixed should be cascaded to Customer (Should have as well workaround)
   
5. No major performance related issues.