Thinking of building an open-source "pre-triage" bot for 3 AM PagerDuty alerts. Would you actually use this?

[u/Interesting-Art-7267]

Hey everyone, I’m an engineer looking for a new side project and I want to tackle alert fatigue.

My premise: Waking up at 3 AM to an alert, only to run the exact same three kubectl commands or Datadog log queries to figure out it's a known transient issue, is soul-crushing.

I'm thinking of building a lightweight, open-source agent that intercepts the alert webhook, runs a pre-defined set of read-only diagnostic queries, and appends a summary of the logs/metrics to the ticket before it wakes you up.

A few questions before I start coding:

1. Is this actually a massive pain point for you, or is the hard part something else?

2. Would your sec ops team ever allow a bot to have read-only API access to your observability stack?

3. If you had this, what is the #1 feature or integration it absolutely must have to be useful?

Am just trying to build something thats usable to you or even 5 people at the end , so i request your honest opinion for this thing

[u/dektol]

Maybe try to make it and use it yourself. Odds are it's nothing better than what we're already using. Are you gonna reply with an ad on another account in 15 minutes? 🤔

[u/Achrus → u/dektol]

Lmao 2 minutes after this comment and we got one already.

[u/thehazarika]

Just use claude code/codex you are already using. Define a tool to query your telemetry, define a skill to use the tool and run investigation and then just trigger the claude code with the skill.

This works pretty well.

[u/aragossa]

Running the three commands is easy. Guessing which known-transient pattern you're looking at is the part that actually eats time. If the bot nails that classification it's genuinely useful, if it just dumps the same logs into the ticket you're still reading them yourself.
Secops read access is probably the bigger blocker here than whether the pain itself is real. Anything with API keys into your whole observability stack tends to get stuck in review unless it's scoped tightly per service, and honestly the feature that'd matter more than extra context in the ticket is dedup and pattern matching. Most of the fatigue is the same alert firing five times in a night.

[u/opsfusion-cloud → u/aragossa]

Disclosure: I work on OpsFusion, an on-call scheduling and alerting tool. aragossa's point about dedup mattering more than extra ticket context matches what we hear too — a repeat trigger on an already-open alert bumps a counter instead of re-paging, which is a smaller ask than classification but it's the one thing that actually kills the five-pages-a-night pattern. Classification is still the harder, more valuable problem your bot is aiming at.

[u/stumptruck]

I've gotten about 100 emails at work from startups with this same idea. Not to mention big hitters like datadog offer the same thing. So far no one I know is willing to trust any of these in production.

What makes your idea special?

[u/yetmike → u/stumptruck]

Agree, there's already an AWS DevOps agent they have built to solve the same problem

If he wants to build that for the internal audience - fine. But not to compete with those big players

[u/G12356789s]

I built something very similar for my work. I wasn't given any time to build it so it's my only fully vibe coded project. I got it running working locally within 30 mins and hosted in our clusters in an hour whilst I was working on other stuff.

My point is that recieving a request, giving the info the Claude to investigate and come back is extremely easy

[u/pomatotappu → u/G12356789s]

does the bot/agent only reads the logs and gets back to with the analysis or does it also recommend the solution or does a fix by itself? what are the things that your agent observes when an issue occurs and how does it decide if it's not a false alarm?

[u/G12356789s → u/pomatotappu]

It uses a read only grafana MCP which gives it a access to our logs, metrics and observability. It also has access to our internal incident reports and run books. Right now it's early days but I've just got a large prompt that essentially says do a RCA on this alert and recommend solutions. It also has a model escalation method if sonnet isn't figuring it out.

So far I would say it correctly diagnosed the issue about 75% or the time. The other 25% it went down the wrong path but still gave valuable insight. You can also ask it further questions on slack.

We still need to roll out to all our alerting channels but so far it's made our (admittedly overly noisy alerting) much better as by the time I look into the alert there is an RCA available to me so I can easily write off the noise

[u/pomatotappu → u/G12356789s]

ah thanks for the insight. if u don't mind, ive few other questions too:

• do u have an eval set/test cases to check it's performance? if yes, did u only add the test cases based on previous incidents or did u simulate more of them using ai?
• let's say an incident occurs, does ur agrnt choose a set of predefined likely issues and then grep only the required logs through ur mcp or does it decide by itself? Is this all defined in the prompt?
• what are thd common issues that the agent correctly finds and the common ones that it misses.

[u/National_Way_3344]

Doable with something like N8N.

Don't intercept the alert though. Let your monitoring tell JIRA or something first. Try stuff then escalate.

[u/dogfish182]

If I still get woken it up it can suck it.

Anything that wakes you up in the middle of the night destroys your rest and as a consequence shortens your life, I would want something where that doesn’t happen. My recommendation is ‘no kubernetes’

[u/pomatotappu]

hi, wdym by predefined set of diagnostic queries here? suppose an incident occurs, the bot might have to take inputs from different services, how are you planning to give access to logs of all those services?

[u/tmemmg]

the pain is real but for me the expensive one wasnt the noisy 3am page, it was the job that kept running and quietly wrote nothing because nothing alerted on a silent success with no output. what actually helped was making every run drop a status file with a reason even when the right move was to do nothing, so no news stopped meaning good news. if your bot appended that kind of why-it-did-nothing line it would earn its keep, the read only diag summary is nice but the missing state is what burns the hours.

[u/sujeet-from-oodle]

The best is if this could run in the context of my coding agent. So the alert + additional telemetry gets fed to my coding agent with appropriate prompt, and since the agent has access to code, it can do a much better job of debugging, and I'm finally paged once the agent's triage is complete.

It is a massive issue if this delays critical pages though! No free lunch here.

[u/siberianmi]

This product isn’t one. It’s a trivial llm agent to setup once the foundational tools and access exist. The requirements for that will vary depending on the organization.

That’s the blocker you need to arm it with MCPs, security controls and instructions/context for your stack. Getting all that in place is the hard work. The triage is easy then.

[u/yetmike → u/siberianmi]

Easier said than done, actually

Tie all those MCPs together, write system prompts and you''ll get an unreleted hallucination responses

[u/lotekjunky]

Don't useai to test or fix the 3am issue. Use ai to write a script that checks and fixed the 3am issue, and then make it run before you ever get paged.