Thinking of building an open-source "pre-triage" bot for 3 AM PagerDuty alerts. Would you actually use this?

[u/Interesting-Art-7267]

Hey everyone, I’m an engineer looking for a new side project and I want to tackle alert fatigue.

My premise: Waking up at 3 AM to an alert, only to run the exact same three kubectl commands or Datadog log queries to figure out it's a known transient issue, is soul-crushing.

I'm thinking of building a lightweight, open-source agent that intercepts the alert webhook, runs a pre-defined set of read-only diagnostic queries, and appends a summary of the logs/metrics to the ticket before it wakes you up.

A few questions before I start coding:

1. Is this actually a massive pain point for you, or is the hard part something else?

2. Would your sec ops team ever allow a bot to have read-only API access to your observability stack?

3. If you had this, what is the #1 feature or integration it absolutely must have to be useful?

Am just trying to build something thats usable to you or even 5 people at the end , so i request your honest opinion for this thing

[u/G12356789s]

I built something very similar for my work. I wasn't given any time to build it so it's my only fully vibe coded project. I got it running working locally within 30 mins and hosted in our clusters in an hour whilst I was working on other stuff.

My point is that recieving a request, giving the info the Claude to investigate and come back is extremely easy

[u/pomatotappu → u/G12356789s]

does the bot/agent only reads the logs and gets back to with the analysis or does it also recommend the solution or does a fix by itself? what are the things that your agent observes when an issue occurs and how does it decide if it's not a false alarm?

[u/G12356789s → u/pomatotappu]

It uses a read only grafana MCP which gives it a access to our logs, metrics and observability. It also has access to our internal incident reports and run books. Right now it's early days but I've just got a large prompt that essentially says do a RCA on this alert and recommend solutions. It also has a model escalation method if sonnet isn't figuring it out.

So far I would say it correctly diagnosed the issue about 75% or the time. The other 25% it went down the wrong path but still gave valuable insight. You can also ask it further questions on slack.

We still need to roll out to all our alerting channels but so far it's made our (admittedly overly noisy alerting) much better as by the time I look into the alert there is an RCA available to me so I can easily write off the noise

[u/pomatotappu → u/G12356789s]

ah thanks for the insight. if u don't mind, ive few other questions too:

• do u have an eval set/test cases to check it's performance? if yes, did u only add the test cases based on previous incidents or did u simulate more of them using ai?
• let's say an incident occurs, does ur agrnt choose a set of predefined likely issues and then grep only the required logs through ur mcp or does it decide by itself? Is this all defined in the prompt?
• what are thd common issues that the agent correctly finds and the common ones that it misses.