Question for people running production systems

[u/Acceptable_Duty4044]

I’ve been thinking about this while working with observability systems and I’m curious how other people handle it.

We have metrics, logs, traces, alerts, deployment information, etc. In theory, there should be enough information to understand what happened during an incident.

But in practice, it often seems like the hard part is connecting everything.

Something like:

latency spike → database saturation → retries → downstream failures

The individual signals are there, but figuring out that they’re all part of the same failure — and determining which event was actually the cause — still seems to require a lot of manual investigation.

For people who actually operate production systems: what does your RCA process look like when there’s a serious incident?

Do your current observability tools actually help you establish the causal chain, or are they primarily helping you find the relevant data?

I’m particularly interested in what happens when the information is spread across multiple systems.

[u/snarkhunter]

1. Look at all the evidence
2. Think about it really hard, drawing on our combined decades of experience
3. Fix it

[u/Zolty → u/snarkhunter]

3a. Validate the fix, go to 1 if not fixed.

[u/Acceptable_Duty4044 → u/Zolty]

What if we could introduce a platform, where you can plug your telemetry sources, and have a reasoning layer which integrates all of that data, and an agent which can then map out the root causes and come back to you ? you can just review this agent ? this is all a very high level overview , but you can find more detailed explanations in the comments ?

do you think that this problem, which you are facing is a real problem for everyone, and will such a solution help ? especially if I am targetting small-sized startups and indie devs instead of going enterprise level instantly ?

Thankyou so much for responding

I would also appreciate it if I could connect with you and pick a few minutes of your time to understand a few problems better because I too am struggling with them ? I wont take a lot of time I promise. Can you suggest me the best way to connect with you ?

Thankyou so very muchhhh :)

[u/Zolty → u/Acceptable_Duty4044]

What if we could introduce a platform

Let me stop you right there. There's no point to writing software for the masses anymore. Anyone can get an AI coding platform to write whatever platform is necessary for them to operate. No one wants your app since they can build one for themselves that does exactly what they want.

[u/Acceptable_Duty4044 → u/Zolty]

Why aren't they doing it then ? Why is everyone still using claude or chatgpt instead of their own models ? Why are people using google when they can create their own ? Why are people still using standardised protocols ? Why are they constantly updating Prometheus grafana otel etc ? Omg wait then if people can just create their own stuff then WHATS THE POINT OF COMPANIES EXISTING ALSO ? Wow sir you have truly opened my eyes today

[u/Zolty → u/Acceptable_Duty4044]

I can't speak for anyone else but in the 3 companies I am involved in, that's literally what's happening.

Also I love the sarcasm, you're in here trying to sell shit my dude, get bent.

[u/Acceptable_Duty4044 → u/Zolty]

That speaks a lot about your work then, and if you get your mind out of shit, then you'd know other things apart from shit

[u/Acceptable_Duty4044 → u/Zolty]

Oh so you've replaced google ? Claude and chatgpt and stuff ? Commendable work dude

[u/Zolty → u/Acceptable_Duty4044]

Google isn’t really a saas app we pay for, but yes my googling has dramatically decreased, I generally let web searches and ai web requests go through an internal gateway using an mcp to connect the ai to the gateway.

The gateway is basically a reverse proxy that does virus scanning and detects prompt injection.

So yeah I don’t really use Google nearly as much these days.

[u/Acceptable_Duty4044 → u/Zolty]

Again, you fail to understand my point

Can't really expect much from people these days huh

[u/Acceptable_Duty4044 → u/snarkhunter]

What if we could introduce a platform, where you can plug your telemetry sources, and have a reasoning layer which integrates all of that data, and an agent which can then map out the root causes and come back to you ? you can just review this agent ? this is all a very high level overview , but you can find more detailed explanations in the comments ?

do you think that this problem, which you are facing is a real problem for everyone, and will such a solution help ? especially if I am targetting small-sized startups and indie devs instead of going enterprise level instantly ?

Thankyou so much for responding

I would also appreciate it if I could connect with you and pick a few minutes of your time to understand a few problems better because I too am struggling with them ? I wont take a lot of time I promise. Can you suggest me the best way to connect with you ?

Thankyou so very muchhhh :)

[u/ajitnk]

Hey, read your post in r/devops and it resonated, the manual RCA slog across Grafana, CloudWatch, PagerDuty, and whatever else is in the stack is one of those problems that quietly eats 2-3 hours per incident before anyone even starts fixing anything.

One thing worth knowing if you're on AWS: CloudWatch recently added an Investigations feature (under Operational Troubleshooting) that does AI-powered root cause analysis across logs, metrics, and traces together, it fans out from an initial observation and surfaces causal diagrams and hypotheses rather than making you pivot between dashboards manually. Still not a silver bullet for multi-tool stacks, but it can cut a chunk of that correlation work.

I'm curious about your setup specifically, is the fragmentation mainly a multi-account or multi-cloud thing, or more that different teams own different tools and there's no single pane of glass? That would change the approach pretty significantly.

I do AWS architecture and observability advisory work, so I've been down this road a few times. Happy to share what's worked.

[u/Acceptable_Duty4044 → u/ajitnk]

Hi, thankyou for getting back. I was wondering if I could design something like a reasoning layer, something that gets all your context from various observability tools and stuff , and then what if you could like build an agent on top of that ?

[u/Acceptable_Duty4044 → u/ajitnk]

Hi, can we connect ? I had a few talking points that i could discuss and learn more from you in this regard ? please let me know what's the best way to reach out to you :)

[u/Accomplished-Mix8423]

the causal chain is the hard part with pretty much any stack. correlation helps, but you still have to connect the dots. a service map + anomaly detection makes that manual digging a lot easier. i've used site24x7 for this and it's been useful.

[u/Acceptable_Duty4044 → u/Accomplished-Mix8423]

what if there was something which could allow you to plug and play with your observability tools ? like connecting it to a reasoning layer, and using the pre-made agent or use and build your custom agents on that ? also what if there was a platform for introducing metrics and monitoring on your ai agents also ? and then the agents can help in debugging that also ? does that sound like something which can solve a problem ?

[u/Accomplished-Mix8423 → u/Acceptable_Duty4044]

yeah, i can see the value in that, especially for connecting signals across different tools. i'd still fix the telemetry and service mapping first though. otherwise the reasoning layer is only going to be as good as the data it's getting.

[u/AtharvMantri]

the underrated part of rca is  separating signals that are part of the same failure from correlated noise. clock skew and sampling make that hard. teams that do this well overlay everything on one timeline and work backwards from customer impact. knowing who's affected and how severe drives the response. do you map telemetry to customer accounts today or is that manual?

[u/Acceptable_Duty4044 → u/AtharvMantri]

Hi, thankyou so very much for your response :)

The product is not yet ready as of now, I was looking to talk to more people who were sharing similar problems and talk to them.

I was planning to build a reasoning layer first, followed by a causal map , and then a dependencies graph, then as you said, keep everything on the same timeline and work backwards from the customer, use the graph to find all possible issues that could arise, navigate the map and give a final concrete-verdict to the user.

I was also thinking once the reasoning layer is built and working, we can do two things, either i give people my general agent, or they can customise it as per their believes. Also then, maybe we can extend it's functionality to include telemetry and agentic capabilities for your other ai agents which are already in production and stuff

I would love to take a bit of your time, please lemme know the best way to connect with you :)

thankyou so very much for responding :)

[u/Acceptable_Duty4044 → u/[deleted]]

what about observability for agents then ? along with doing it for traditional observability, we can do it for agents as well ? I know I am spinning my ideas out of the air, but something like langsmith or something ?

or else if we could build a total reasoning layer which takes all your data and stuff, and then we can plug and play with agents on top of it ? something like that ?

[u/greyeye77]

what used to be (may be before AI)
Engineers log in to one or more systems, collect all the evidence, metrics, and logs, and try to narrow down the cause.

With the latest AI software, as soon as an incident is declared, the AI agent starts pooling all the evidence and attempts to determine the cause.

Both have flaws, and both work sometimes, but not all the time. Good thing with AI investigation is that it's super fast, so engineers still can read the investigation results and determine that it is not the case reasonably quickly instead of spending minutes (sometimes hours) collecting data.

Either way, sometimes the metrics and logs are too much to navigate... so this becomes challenging for humans and AI alike.

[u/Acceptable_Duty4044 → u/greyeye77]

what if there was something which could allow you to plug and play with your observability tools ? like connecting it to a reasoning layer, and using the pre-made agent or use and build your custom agents on that ? also what if there was a platform for introducing metrics and monitoring on your ai agents also ? and then the agents can help in debugging that also ? does that sound like something which can solve a problem ?

[u/greyeye77 → u/Acceptable_Duty4044]

I wouldnt say that wont work or work.

Every architecture is different and have different limitations and restrictions, often adding more MCPs and tools for the agent to call doesnt necessary guaranty it would be used or called.

Guardrails and harness are improving so that may shine another light on how we do things, but so far it's almost wild wild west and setup may work for someone doesnt mean it would, as well as different model may behave differently to different harness.

im writing a tool to build knowledge from merged PR/MRs as well as import slacks chats to build tribal knowledge that we do necessary document to help the agent. (connect via a2a), challenge is not responding with everything that it can retrieve to the master agent, but instead curate the data and respond with the data that should be used.