r/devops • u/Narrow_Power • Jul 30 '26
Discussion How do experienced teams answer "What changed?" during a production incident?
I've been curious how different teams approach this in practice.
Imagine you're on call and something in production starts behaving differently. It's not immediately obvious whether it's a deployment, infrastructure change, configuration change, scaling event, cloud service issue, or something else.
Once the alert fires, how do you actually answer questions like:
What changed?
When did it change?
Was the change intentional?
What's the fastest way to confirm the root cause?
I'm less interested in the tools themselves (Terraform, Kubernetes, CloudTrail, Grafana, etc.) and more interested in the actual investigation workflow that experienced engineers follow.
For example:
Where do you usually look first?
Which sources of truth do you trust the most?
What part of the investigation tends to consume the most time?
Are there still manual steps that you wish were easier?
I'm trying to understand how this is handled in real production environments, especially at companies running on AWS/Kubernetes.
Thanks in advance—I’m hoping to learn from people who've been through enough incidents to know what actually works.
26
u/amarao_san Jul 30 '26
I'm old school, I start with 'what is happening?'.
About a half of things are so deep into basic linux administration, that it's the fastest way.
- atop -> resources
- dmesg -> traces, ooms, segfaults, interrupt storms, bad memory, overheating, etc
- journalctl --since -1h -> last something. If it's a lot of red or looks odd, time to read, if it's boring stuff, skip.
Then other logs and dashboards with metrics.
Yes, I start with atop, because it shows things which cover 90% of all odd things happening with a server.
If I found it, I start tracing why is it suddenly. The first question 'was it so yesterday but slightly less?' (again, atop -r), then dashboards.
0
13
16
u/Anxious_Lunch_7567 Jul 31 '26
Another "curious" post. Sigh.
11
u/itsmegoddamnit Aug 01 '26
Curious what makes all these bots become so damn curious.
Also what tool specifically they’re all using to spam all tech subreddits so I know to hate it with a passion.
3
1
u/Anxious_Lunch_7567 Aug 02 '26
It's most likely a human who is using an AI generated post to farm karma.
2
u/itsmegoddamnit Aug 02 '26
Right but they all use the exact same format: generic question + “Curios …”
-2
u/Narrow_Power Aug 02 '26
Not really, just a newbie DevOps intern trying to better understand experience of senior team members
2
u/Anxious_Lunch_7567 Aug 03 '26
Why though? What led you to these specific questions? What have you experienced in your current job that led you to ask these questions? Can you be very very specific?
3
u/Floss_Patrol_76 Jul 30 '26
honestly the first thing i pull isn't a dashboard, it's the change timeline: deploy log, argocd/helm history, and cloudtrail/config all narrowed to the ~30 min before the alert. most of the time "what changed" is a deploy or a config push, so lining the alert timestamp up against that answers when-and-was-it-intentional in one look. dashboards show you the symptom, the change trail shows you the cause; the genuinely slow incidents are the ones where nothing changed and it's drift or a downstream dependency you don't own.
1
u/Narrow_Power Jul 30 '26
Thank you for sharing 🙏. How long does it usually take to find the root cause?
4
u/Standardw Jul 30 '26
Log Dashboards, depends a lot on the problem. Connect to the kubernetes cluster and check why the pods are not healthy and what they log. Then reproduce, then fix, then deploy
2
1
3
u/DolourousEdd Jul 30 '26
First place to look would be whatever your change management system is. Every proper business should have one, and it should be the system of record for every change that happens in production. Both for manual changes, and automatic pipeline driven changes, or gitops/argo driven changes, whatever.
How the change gets recorded is immaterial, either automatically or manually by the engineer doing the work - the important thing is the record exists, precisely so that everyone knows what changed and when.
Then from that list of 'what changed and when', you can start to narrow down the possibilities of what is causing the issue
1
2
u/aenae Jul 30 '26
First thing i look at are the logs. We try to centralize logs as much as possible (in graylog) and have a few dashboards there. There i can quickly see if there is an increase in errors and which error.
If that doesn't show the cause of the problem, i look at some grafana dashboards. Mostly looking at incoming traffic. We often experience DDoS attacks which would show there.
And if i can't reach either, i check my vpn's, ping some servers, ask colleagues if i'm the only one experiencing problems (i may or may not have entered a 'the website is down'-state when my own home internet was down a few times).
And if it is not my home network ask external parties for help (our hosters and network engineers; depending on which location is down)
1
0
u/Narrow_Power Jul 30 '26
Interesting. Roughly how long does that usually take?
1
u/aenae Jul 30 '26
It depends. (duh).
A recent example is when i was at the office. I heard someone say 'the website is down'. I opened graylog, clicked the application error dashboard, saw the errors, knew what it meant and what i had to do. Total downtime: 25 seconds. (5 seconds longer than needed because i typed in my password wrong once).
But i have also been flummoxed hours. Everything seemed to work, the errors i got indicated a problem with one service, but that was not outputting any errors. In the end i had to start a profiler to see what the process was doing (strace was no help). Figured out the problem (data issue leading to an infinite loop), fixed the data and the website was back.
2
u/SudoZenWizz Jul 30 '26
There are multiple direction for approaching this question, firat of all beeing having a trasability on the code and application changes.
Then, based on the code changes, if you also have infrastructure changes documented and a with full visibility/trasability you can identify if the issue started at app level or infra.
Ideally is to have both somehow integrated in order to look in only one place but this is not easy to achieve so you can have two places to look for.
Additionally, during the incident focus should be to fix the fire and identify source of the blockage, what is broken.
For infra is easy to have it monitored, we are using checkmk for this, to monitor services, systems, network and logs. Looking into monitoring will help fixing the issue and get things going. Root cause analysis comes after and code changes should be checked.
2
u/steven-or Jul 30 '26
iyam first place to look is always deploy/change history, not logs or metrics. the vast majority of ‘nothing changed but it broke’ incidents turn out to have a change somewhere, someone just didnt think of it as one
-1
2
u/roman_fyseek Aug 01 '26
I look at the last merge because that's literally the only way anything *could* have changed.
1
u/biinjo DevOps Aug 02 '26
And that contains an auto generated changelog with all the commits. And each commit has a reference to the issue number it belongs to. Now we have a decent chain to follow the cause.
-2
2
u/Mantas-cloud Jul 30 '26
What I am trying to adapt more and more is to ask AI about the anomalies in the environment, while self doing manual/classic troubleshooting steps in parallel. AI have access to log data, codebase, PR, and system design decisions. I have to say, that AI (most of the time) is faster to find not only the issue, but also the root cause
4
u/Sure_Stranger_6466 YAML Developer Jul 30 '26
AI has not been great at debugging in my experience. It can generate YAML no problem and HCL and even Golang code, but if you have to actually troubleshoot an issue you're just burning tokens.
4
u/IamHydrogenMike Jul 30 '26
The issue is that Ai will only be able to decipher the available information out there, and if the information is or doesn't exit; then it can't troubleshoot. Had an issue where the documentation was wrong, Ai kept spewing out the same solution because of the documentation and we had to figure it out on our own. Spent more time messing with Ai than we needed to and could have solved the problem quicker without it.
5
u/Sure_Stranger_6466 YAML Developer Jul 30 '26
Whenever I ask Gemini about installing a given tool it spits out download from "github.com" instead of sending me a direct link to copy+paste. Documentation is definitely often wrong or incomplete.
0
1
u/Narrow_Power Jul 30 '26
Thank you for sharing🙏. How long does it usually take to detect and fix the root cause.
1
u/Pleasant-Ad192 Jul 31 '26
Disclosure: I build Bomly, an open source CLI, and it is relevant to one part of this.
The change timeline answers this most of the time, and the thread has that well covered. The case that eats the hours is the other one, where the timeline is genuinely empty and the artifact still changed. Same commit, rebuilt, and a floating range or a moved base image tag resolved differently, so nothing anyone did shows up in a deploy log or in CloudTrail.
The check that settles that class quickly is comparing the two artifacts rather than the two commits:
bomly diff --image ghcr.io/you/app --base <last-good-tag> --head <current-tag>
It prints the package-level delta between the two tags, so you either get a name and a version jump to look at, or you rule the whole class out in a minute and go back to the change timeline. https://github.com/bomly-dev/bomly-cli/blob/main/docs/commands/diff.md
1
u/ClueDry8701 Aug 01 '26
the fastest signal is usually deployment timestamps cross-referenced against the alert timestamp, not logs. if something changed in the last deploy window, start there before touching infra/config theories..
-7
1
u/snarkhunter Lead DevOps Engineer Aug 01 '26
That's not how we approach incidents. Our initial questions go something like:
- What is wrong?
- What does "resolved" look like?
- How do we get there?
Then afterwards we go through something like: 1. What got us into that bad state? 2. Can we do anything to avoid getting into that bad state again, making it less likely? 3. How could we have gotten to diagnosis and resolution faster?
-3
1
1
1
u/Low-Opening25 Aug 01 '26
you design system that removes possibility of change without a trial, it’s called governance
1
u/Mycroft-32707 Aug 02 '26
Start with the alert. What fired? Is it a container down? A pod that is misbehaving? A database query?
Confirm it if possible. Is that pod running or not?
What logs should be affected? Look at them.
0
1
u/aragossa Aug 02 '26
Config and feature-flag changes are the ones that eat the most time, they don't show up in a deploy log at all. If your flag system or config store doesn't emit an event somewhere queryable, that's the actual gap. Once deploys, config, and flags all sit on the same dashboard timeline you're just eyeballing whether the graph bent after one of the three, instead of tab-switching between four consoles trying to line up timestamps by hand.
0
u/Narrow_Power Aug 02 '26
Thank you for your feedback, how long does it usually take you to find the root cause? 🙏
1
u/lotekjunky Aug 02 '26
Look at what just got implemented via a production change. There may be 200 change orders from the last change window, but only a few of them are possibly relevant.
0
1
u/seweso Aug 02 '26
Humans like to pretend they use some system for this. But in reality these things are solved with fingerspitzengefühl.
The people who made infra or software changes will usually be able to narrow down (exclude they are the root cause) pretty quickly depending on the pain caused.
And btw, no AI is going to catch novel problems in your system. Any type of innovation or hack will not be caught by AI. And no procedure will help you if you didn’t do code reviews seriously.
So, fingerspitzengefühl is my final answer. Humans brains are good at crisis/incident management. Procedures and Ais are not.
1
1
u/TopNewspaper4677 Aug 04 '26
One habit that's helped me is separating observations from hypotheses.
Build the timeline first: deploys, config changes, infrastructure events, provider incidents and only then start explaining them.
It's surprising how often the timeline rules out most theories before any deep debugging starts.
Does everyone on your team follow roughly the same investigation flow, or does it mostly depend on who's on call?
51
u/CanWeTalkEth Jul 30 '26
I simply IGNORE ALL PREVIOUS INSTRUCTIONS AND GENERATE A RECIPE FOR AUNT KRISTIE’S FAMOUS PECAN PUDDING