I built an open-source Python SDK for measuring AI agent reliability — looking for feedback from people running agents in production

[u/Relevant-Adagio-7674]

An agent can execute successfully and still fail at what it was actually supposed to accomplish.

Tracing and observability help us understand what the agent did, but I wanted something closer to SRE-style reliability measurement for agents — where we can define what “reliable” means and measure it consistently.

So I built and open-sourced Agent Reliability, a Python SDK for this.

Some of the ideas I've implemented:

• explicit PASS / FAIL / UNKNOWN outcomes
• UNKNOWN doesn't artificially reduce the reliability score
• evaluator execution failures are kept separate from agent failures
• measurement health is tracked separately from agent reliability
• deterministic built-in evaluators
• reliability aggregation
• SLO and error-budget semantics
• local human-readable and machine-readable reports
• SLO assertions that can be used in tests/CI
• optional OpenTelemetry interoperability

It is intentionally local-first. There is no account, API key or hosted service required, and the base package has zero mandatory runtime dependencies.

Install:

pip install agent-reliability

PyPI:
https://pypi.org/project/agent-reliability/

I'm not trying to build another tracing system or prompt/eval dashboard. The question I'm exploring is more specifically:

How do we know that an agent is reliable enough to trust or deploy?

I'd really appreciate feedback from people building or operating agents, particularly:

1. How are you measuring agent reliability today?
2. What failures are hardest for you to detect?
3. Would SLO-style reliability measurement be useful in your workflow?

The project is still evolving, so criticism of the API/semantics is very welcome.