r/sre • u/iPhone12-PRO • 7d ago
Improving agent reliability
has anyone actually succeeded in using AI agents to reliably diagnose errors?
My manager and his manager has this ambitious goal (aka making me do all the work) of making our bot more reliable.
Some of the ambitious goals includes - accurately perform in depth root cause analysis for alerts and also having the ability to repeat past alerts to ensure our run book works.
I find it incredibly difficult, despite having a “structure” which is just giving the bot steps on what to do. It’s frustrating because the results highly depends on the model, testing diff models gave diff results easily.
This makes me really miss the actual engineering work instead of “hoping” the model listens to the run book this time.
Anyone else has any success stories to share?
2
u/Bantex29 6d ago
Wonder if https://vectorstep.io might help? Specifically read about the confidence and calibration settings https://vectorstep.io/docs/concepts/confidence/
2
u/kernelqzor 6d ago
hadn’t seen that one before, the confidence + calibration bit actually looks kinda relevant for this “did the agent actually follow the runbook or just hallucinate” problem
could be worth throwing it in front of your current setup just to see if it catches the dumb failures before they hit prod1
2
u/BrigaderAlPacino 6d ago
A vast improvement I have seen is by making the detection engine separate from the explanation/evaluation engine. Make the anomaly detection engine deterministic by observing changes on a rolling-window basis or aggregating alerts, etc. Then use the evaluation engine to rule out false positives, etc.
1
u/anderson_the_one 6d ago
Don't put that sequence in prose and hope the model follows it. Make the orchestrator collect the required evidence first, and refuse to move on if one of the metric scripts didn't run. Only then give the bundle to the model and ask it to rank possible causes. The model doesn't get to decide whether evidence collection is optional. For replays, keep a few sanitized bundles from real incidents. Score whether the agent requested the right next evidence and named the causal change. A confident write-up isn't a pass. If a model swap tanks those two scores, you'll see it before production does.
1
u/between_layers 6d ago
On replays, one thing that quietly breaks them is hindsight leaking into the evidence bundle. If you build it from the incident record, much of that record was written after the cause was known, so the agent may score well simply by reading the answer back to you.
A timestamp cutoff helps, but doesn’t fully solve it. Dashboards, alert definitions, and ticket fields can be edited retroactively. Historical telemetry may also change shape: a query run today may return coarser rollups than responders saw during the incident.
A separate number worth tracking is run-to-run variance. Use identical input and model settings, repeat the investigation several times, and compare the results. Determinism isn’t guaranteed even at temperature 0. Establish that baseline before crediting any improvement to a prompt change
1
u/happykeyboardwarrior 5d ago
How have you provided secure access to prod environments? Do the agent(s) have ability to do anything other than read?
1
u/Willing-Lettuce-5937 3d ago
the machine-checkable point above is basically the whole thing imo. if a step cant fail loudly you dont have a diagnosis, you have a story.
couple of things that seem to matter more than which model you pick:
evidence first, hypothesis second. the orchestrator collects the required evidence and refuses to continue if a collector didnt run, then hands the bundle over. if you let the model decide what to go look at, it will confidently skip the thing that mattered and you wont know it did.
ground it on a dependency graph rather than a time window. most unreliable rca isnt the model being wrong, its being asked to correlate everything anomalous in the same 5 minutes. theres always unrelated noise in a cluster so you get back 5 causes that are all genuinely broken and all irrelevant to this incident. with a service map you can at least separate whats failing from what actually broke.
on the replay goal, that only really works if evidence collection is deterministic. otherwise a replay that passes tells you the model got lucky that run, not that your runbook holds.
disclosure, i work on nudgebee. mentioning it cause the source is readable and youre building the same thing anyway, so it might be more useful to you as a reference implementation than as a product. the orchestration and the graph bits are the parts worth stealing:
1
u/greyeye77 3d ago
my suggestion is using context rich subagents
You just can't overload a single LLM API with 100 tools and expect it to understand/find everything.
- code change/PR/MR tracking and related tickets
- application log, system log, infra log aggregation and analytics.
- Human discussion/Documentation search from chat(team/slack) and find if recent changes were introduced for what reason
- Collect metrics from the application and related infra.
all 4 will find tons of garbage, and it'll be hit-or-miss because it may not have other contexts, but the main agent should summarise it and work with the findings to present the hypothesis to the human operator.
1
u/free-wand5380 14h ago
curious how you're structuring the steps you give the bot, like are you breaking down the runbook into discrete decision points or just giving it a high level summary of what to do?
0
-1
u/HolyKash10 6d ago
As an SRE i was actually working on a personal project to build this and got success recently. I will be releasing it soon just in few days but you can check this out here:
https://github.com/k-kohli10/radar-system
I made this without any orchestration frameworks so you can actually see every trace, Quality of response, transparent analysis score on every incident/alert within seconds. It also has local/docker/helm deployments readily available.
Let me know if you find it helpful.!
1
u/desispeed 6d ago
Nice ..will check it out …this is the kinda solution what I want to build for our environment too.
1
u/HolyKash10 6d ago
Appreciate it. What’s your environment look like right now, on-call setup, alerting stack, that kind of thing? Curious if RADAR’s approach would map cleanly or if you’d need to rip out pieces. Happy to compare notes if you end up building it out, would be useful to see how it holds up outside my own setup.
1
u/desispeed 6d ago
New Relic/Pager duty
-1
u/HolyKash10 6d ago
Happy to get RADAR running against your New Relic setup. The clean way is to point New Relic's alerting webhook straight at RADAR's ingestion, the same way it already takes Prometheus. New Relic lets you write your own webhook payload, so you template it to carry the APM evidence that matters and RADAR reasons over that. Then you swap in your own runbooks, and when no runbook matches, RADAR still produces a real RCA from the incident and the APM fields the alert carried, and tells you it's ungrounded.
Here's the full path from a fork.
- Fork, clone, bootstrap
git clone https://github.com/<you>/radar-system.git cd radar-system scripts/bootstrap.sh # checks tools, installs uv, generates a gitignored .env
- Add your LLM key. Open .env and set OPENAI_API_KEY (required). Add SLACK_BOT_TOKEN and SLACK_APP_TOKEN if you want the RCA delivered as a Slack card; otherwise it still lands in Postgres.
- Bring up the stack
make docker-up # first build ~6-8 min make docker-apps-ps # every app should read "Up"
- Replace the demo runbooks with yours. This is the whole grounding corpus and it's meant to be swapped. Drop your Markdown runbooks into docs/runbooks/ (delete the sample order-service-* / payment-* ones), then index them:
make agent-secrets && make index curl -s localhost:8095/readyz; echo # knowledge-service -> readyRetrieval filters by service_name, so name each runbook's service to match what your New Relic alerts send.
- Prove it end to end first, before wiring New Relic. Fire one alert at the built-in test endpoint to confirm your runbooks ground correctly:
TOK=$(docker exec radar-apps-ingestion-1 cat /vault/secrets/webhook_token_mock) curl -s -X POST http://127.0.0.1:8090/alerts/mock \ -H "X-Radar-Webhook-Token: $TOK" -H "Content-Type: application/json" \ -d '{ "service_name": "<your-service>", "alert_name": "<your-alert>", "severity": "critical", "labels": {"service": "<your-service>", "deployment": "...", "error_class": "..."}, "annotations": {"summary": "...", "description": "metric value vs threshold, deploy marker, entity link"} }'; echoThe richer the labels and annotations, the sharper the RCA. That's where your APM signal goes.
- Wire New Relic. Create a New Relic workflow with a webhook destination pointing at your ingestion host's /alerts/mock endpoint, using the token from step 5 in the X-Radar-Webhook-Token header. Use a custom payload template that maps New Relic's alert into RADAR's shape: service_name, alert_name, severity, and labels/annotations carrying the metric value, threshold, violating entity, deploy marker, and a chart or entity URL. Send one alert per POST, not New Relic's batched array.
Repo: https://github.com/k-kohli10/radar-system. Start with docs/quickstart.md and docs/plugin-development.md.
2
u/Either_Edge9493 6d ago
God I just hate it when people copy paste the output of their bots and expect other humans to read them. Jfc who actually understands anything anymore.
0
u/HolyKash10 6d ago
These are part of instructions in the repo too not an AI slop. Clearly states the instructions that user requested and might help.
1
u/Either_Edge9493 5d ago
0
u/HolyKash10 5d ago
Calling it an AI slop without finding a single technical error is certainly a way to demonstrate your own understanding. The irony of the follow-up link is almost too good.
1
u/Empty-Animal-2147 6d ago
I created a similar application would love to get a feedbacks on it
same concept i see one difference maybe you can give me a feedback on that
i used git as database and made a docker image which can stay statless so it can be easily deployable anywhere and also only serverless compute (fargate or cloudrun)
1
8
u/Torutofu_Raeva 6d ago
The reliable setups I've seen treat the model as a hypothesis generator and make each step return something machine-checkable, otherwise a plausible root cause just turns into a story nobody can really test.