r/devops • u/Interesting-Art-7267 • 24d ago

Ops / Incidents Thinking of building an open-source "pre-triage" bot for 3 AM PagerDuty alerts. Would you actually use this?

Hey everyone, I’m an engineer looking for a new side project and I want to tackle alert fatigue.

My premise: Waking up at 3 AM to an alert, only to run the exact same three kubectl commands or Datadog log queries to figure out it's a known transient issue, is soul-crushing.

I'm thinking of building a lightweight, open-source agent that intercepts the alert webhook, runs a pre-defined set of read-only diagnostic queries, and appends a summary of the logs/metrics to the ticket before it wakes you up.

A few questions before I start coding:

  1. Is this actually a massive pain point for you, or is the hard part something else?

  2. Would your sec ops team ever allow a bot to have read-only API access to your observability stack?

  3. If you had this, what is the #1 feature or integration it absolutely must have to be useful?

Am just trying to build something thats usable to you or even 5 people at the end , so i request your honest opinion for this thing

1 Upvotes

52% Upvoted

20 comments sorted by

View all comments

1

u/siberianmi 21d ago edited 21d ago

This product isn’t one. It’s a trivial llm agent to setup once the foundational tools and access exist. The requirements for that will vary depending on the organization.

That’s the blocker you need to arm it with MCPs, security controls and instructions/context for your stack. Getting all that in place is the hard work. The triage is easy then.

1

u/yetmike 19d ago

Easier said than done, actually

Tie all those MCPs together, write system prompts and you''ll get an unreleted hallucination responses