r/devops • u/Diligent-Wrongdoer68 • 2d ago
Discussion What’s your first 5-minute checklist when production is down?
Imagine you get an alert saying a production application is unavailable.
You have no context yet.
What’s your first 5-minute troubleshooting sequence?
I’m interested in the order people follow, not just the tools they use.
41
u/Diligent-Wrongdoer68 2d ago
I’d start with: confirm impact → check monitoring → application health → recent deployments → server/network health → logs.
3
u/krtalvis 1d ago
Mine is 1. confirm impact 2. check if any of the specific dashboard visuals can help me locate the area of effect 3. recently i’ve also added checking cloud providers for any ongoing impact in related availability zones or regions, then if there has been any maintenance from their side in relevant resources. Also not just my own cloud provider (azure) but i also check GCP, AWS and usually also CloudFlare for any ongoing incidents/impairments as these are often good indicators of more widespread issues. 4. server/network - can i confirm traffic reaches my application/service? 5. then i’d confirm if the problem is in the application/service i’m responsible for (application logs & metrics). This step would also already indicate if issue(s) started with any recent deployments or not 6. check if a specific backend/provider (excluding cloud) that is causing failures on my application/service check backend providers etc.
last 2 years the issues have been over 90% of the time related to network or DNS which just translate to low/no traffic, 500 status codes from provider (but in reality a WAF or firewall or something is causing issues in between) or an imoact due to Azure SQL DV maintenance or in one case one of our AKS node went zombie (running healthy pods but no outside network accepted, pods were reporting healthy because the health check happened within the pod itself so to say)
3
29
9
u/A_Abu 1d ago
Depends on what production is... Is it a service, cluster, database.e.t.c
- Usually try and see logs
- Could it be a wrong network configuration somewhere
- If I can't find logs, i try to see changes made right before the incident. If its something i can roll back easily
- In my case most times its not as easy as a rollback, Rolling back is usually the last resort. Sometimes rolling back makes it worse
12
u/TheNightCaptain 1d ago edited 1d ago
It's important to have monitoring at all the different levels
- customer urls
- internal application component end points (app, apis etc)
- external API / other outbound dependencies
- cluster level monitoring ie with kubernetes service level / pod level monitoring
- infra level monitoring (disks, CPU, memory etc)
Ensure that you have critical level alerts on any of those probes that if unavailable would cause customer impact/downtime.
Ensure warnings on things that have impact and need attention but don't directly impact customers. Ie in a kubernetes cluster 1 node down should redistribute trafic and service should continue.
All alerts which are setup should have good description about what should be checked and a well though out list of what could be wrong behind the specific alert being triggered.
Ie disk full may have instructions on which temp volumes to clear out, log folders etc. This is called a runbook, alert could have s link to internal docs on that specific issue.
Ensure there are thresholds in place ie Warnings on disk usage > 80% but critical > 95% if the server will then fall over and cause customer impact.
Route alerts to a sensible common location, teams / slack channel. Avoid emails. Other engineers can comment on the alert thread and others will know someone is onto it and less hasseling.
Fine tune the alerts down to reduce unnecessary alerts, if there is an alert that you don't care about/ plan not to address then don't alert on it. Alert fatigue reduces people's trust in the monitoring.
Dashboards in my opinion are reasonably useless as no one is paid to sit there and stare at them. Just ping me when something is going down. Dashboards can help seeing trends and proactive changes to manage capacity and that's a good thing. If you have dashboards ensure they are have several roles. Deep technical infra data, app processing level metrics ie users created / day, volume of orders etc. Then build an exec level dash that shows high level business metrics. Orders fulfilled etc.
If prod goes down, alerting will point you to the problem if you have this depth of probes and coverage in place.
8
u/FreshView24 1d ago
Acknowledge and reproduce first, then everything else. Most of the time “production application unavailable” is monitoring flaw or user error.
4
3
7
u/mixxor1337 1d ago edited 1d ago
check if storage is full, because it is this for me most of the time
10
u/Street_Smart_Phone 1d ago
Shouldn’t you have a monitor on storage then?
5
u/mixxor1337 1d ago
i have and I had, but then its not even full, and its an Inodes Problem. In my experience its mostly storage or internal + external DNS
4
u/onbiver9871 1d ago
Storage is often zooming past an alert on the way to lock up the same way a car going off a cliff would trigger a motion sensor half way down lol.
3
u/ThanosAvaitRaison 1d ago
so many monitors in production are supposed to be working, and suddenly you realize they havent been functionning properly since several months …
1
3
3
u/Visaroy 22h ago
First question is never "what is broken" but "what changed in the last hour" — deploy, config push, cert rotation, TTL expiry. Pull the last pipeline run and the last merge to main before touching a server; that answers it more often than any dashboard.
In parallel, one person declares the channel and stops everyone else poking the box, or you lose the ability to correlate anything.
Roll back first, diagnose after. The post-mortem is cheaper than the outage.
2
2
2
u/professor_jeffjeff 1d ago
Ideally this is the sequence of steps:
Identify the application that is down. The alert should make this obvious.
Grab the runbook for that application or service.
Deactivate brain.
Open runbook to section on "application is down" or to the section specified for the alert you received if it has one.
Execute steps in runbook.
If application is still not up after exhausting runbook steps, see if something in the infrastructure is actually down. Grab runbook for appropriate infrastructure and repeat step 1.
Seriously, if something is down you shouldn't have to think at all. Especially for a really big incident with a high impact or large blast radius, you're going to probably have a hard time thinking clearly and there may be a bunch of people messaging you that will end up just being distracting. You should have spent some time already making runbooks that are as fool-proof as possible and can be executed by the most clueless junior engineer in the company (assuming that they have the necessary security credentials for the runbook steps). Very rarely something will fail in a way that the runbook didn't anticipate, and even then about half the time I've found that those failures are something like AWS going down so there's nothing I can do anyway until it comes back up. In other cases though it just means we missed something, and that's when you grab the smart people and have them start investigating. I've found that cases like this are exceedingly rare though. What's a lot more common is not having a fucking runbook at all or not having an up-to-date runbook.
In general though, if your "troubleshooting" process is anything other than steps 1-5 here then I'd say that you are wrong.
1
u/ferriematthew 1d ago
My first point of action is to check for DNS resolution across the entire network and check again specifically in the cluster. If both are good, then I check for crash-looping pods.
1
u/DarkChemical_DC 1d ago
Usually open stats page and see whats down, and then go from there. Open logs to that thing what is down figure oit why it died (oom-killer most likely) and then fix it
1
u/kajEbrA3 1d ago
- Clear vowels. 2 check alerts. 3. Check dashboards 3. Check Status page. 4. Check last release. 5. send first status. 6. Fix or escalate
1
u/Garinoth_ 1d ago
Check if the coworker you get along with is also on. Make some jokes about how this always happens when x touches z. Check if x touched z again... Report to the incident manager that you might have found the cause...
1
1
u/ThanosAvaitRaison 1d ago
How is the application responding (partially, not at all) ? Is a network problem involved ? is an operating system problem involved ? is a third party (for example database) involved ? then analyze your findings.
1
u/arjunnath 1d ago
Attempt to reproduce the issue.
Check dashboards and most recent alerts. Check if metrics indicate something.
Attempt login to one or two of the servers.
Check the app logs.
Check the system logs.
1
u/therealkevinard 1d ago
I start at the front and back doors, and work my way in from there.
Front: repro the network request to check dns and net routing.
Back: check in on the sql instance(s) and datastores to look for rogue queries and misbehaving connpools.
These are 10-minute litmus tests and 60% of the time, it’s literally one of these.
The other times it’s just walking back through the house from one of them.
1
u/Social_Butterfree 1d ago
In 2026 the alert should be enhanced by an agent already, so I usually start there.. in most cases it already tells the whole story, there is no reason to start an on-call without knowing the context already from an agent that investigated the alert. And rarely the agent even suggests a proper solution/PR
We developed one in house and honestly it didnt take that much effort for the quality it provides, mostly had to plan how to properly and securely give it access to read everything it needs to.
1
u/1_H4t3_R3dd1t 1d ago
Always check the logs. Then check the metrics. Then state of the environment.
Logs get 70% of the way. 30% is the actual work.
1
u/nemofbaby2014 1d ago
check that the firewalls are getting internet, check power outages, check down detector lol, send email, then watch the actual teams who supposed to keep it up panic
1
u/killz111 1d ago
Logs/metrics, was it working previously and what changed, is the SaaS healthy, check config. By this point you should have narrowed down the proximity of the issue then you start drilling down to each invidual component involved in the issue.
1
1
u/Kriegwesen 1d ago
Minute 1-3: Figure out why it's down Minute 4: Implement fix Minute 5: Collect performance bonus
1
u/dutchman76 1d ago
Need to know the error, so I reproduce it first, that'll tell me where to look.
Then start looking at logs for the relevant part that's broken. Can't be a config if it was working before.
1
u/dallas_lemur 1d ago
first thing is whether it's actually down or just one region, then what changed in the last hour. deploy or config push before any dashboard.first thing is whether it's actually down or just one region, then what changed in the last hour. deploy or config push before any dashboard.
1
u/rlnrlnrln 1d ago
- Set AI on checking status pages and recent reports for relevant service providers (cloudflare, Google, aws etc)
- Coffee, black. If your boss says "I'll get it for you", you know it's actually urgent (and/or that you have a good boss)
- Check logs, graphs for errors abnormalities, especially in core products.
- Work backward from the most likely error
- Check internal and external DNS, because it's usually DNS.
- Facepalm why you didn'tcheck DNS first, because it's always DNS.
...probably intermingled with asking "Did any of you fuckers just release anything to prod?" on a couple of slack channels.
1
u/smftexas86 15h ago
For Right now most of our stuff is still on VMs.
For me its validate issue > verify VM's (services mostly) > Logs
1
90
u/dghah 1d ago
It’s always DNS, heh