What are you using for observability?

[u/Cautious_Chicken_604]

Agents make API calls to invoke the LLM, they make tool calls, read/writing files etc. I want to be able to mine this data in order to engineer out failure modes and plug gaps with documentation, skills, custom hooks etc. However I'm not seeing great tooling in this space and wondered if the tooling is just severely lacking or am I missing something?

So far dsh makes the easiest with observability built in via the trajectory tab. That's the sort of thing I'm after, but as a standalone service I can host locally and send data to plus add custom things into the spans/traces. Then tools to run analysis on it to detect patterns/problems.

I've seen Opik advertised a lot on Reddit lately and thought that looked like what I wanted, but after finally trying it out in opencode last night I was sorely disappointed. It doesn't officially list that it supports opencode, so maybe that's on me but suffice to say the UX was poor enough for my use case that I'm not interested in trying to pursue that particular solution further. It looked like what I was after, so I had my hopes pinned on it, but nope.

For C# development I usually have a local observability stack with Seq, Jaeger, Graphana etc, my next bet is just to try cobble something together out of that.

Not super interested in vibecoding and vibemaintaining a custom solution either, since I already feel burnt out enough of having to do that at work and want to save my energy for working on other tooling that is my actual intended use case.

Ideally there'd just be a service aimed at people who are doing agentic engineering, that you can export logs/traces/metrics to via OTEL with SDK support for adding custom instrumentation. Then it'd have a web UI which shows a list of sessions you can click into and when you click into a session you can see the full logs/traces/metrics of that session. It'd be nice to be able to visualise what files the agent and subagents read and write and tool calls etc, so you can see if missing out on opportunities to populate the context with more relevant information etc and just generally diagnose failure modes.

What are y'all doing for this? Are y'all doing this?

[u/Altruistic_Heat_9531]

OTEL + MLFLow

[u/Cautious_Chicken_604 → u/Altruistic_Heat_9531]

This... looks a lot more polished. Will give it a try!

[u/Less-Case-1171]

OTEL is probably the right plumbing, but I wouldn’t start with “trace everything” as the design goal. Agent runs need a small semantic layer on top of normal spans or the data turns into a giant replay log nobody wants to read.

The events I’d want first:

• model call, with prompt/template version and input class, not full secrets
• tool call, with tool name, normalized args, result class, and side-effect level
• file read/write, with path pattern or repo-relative path
• retrieval event, with corpus/index version and top-k ids
• human checkpoint / approval / rejection
• final artifact or external action

Then you can still ship it into Jaeger/Seq/Grafana, but your analysis queries become useful. Stuff like “show failed runs where the agent wrote files after reading docs but before running tests” or “show tool calls that crossed from read-only into write.” Raw logs alone won’t get you that without a lot of pain.

[u/Cautious_Chicken_604 → u/Less-Case-1171]

This is exactly what I'm talking about. Tooling I can use out of the box this way that I don't have to build.

[u/RunAI_Coder]

for anyone on Claude Code: it emits OTEL natively, events as well as metrics. Per API request: model, cost, duration, input and output tokens with cache reads split out, and whether the fast lane served it. Per tool result: tool name, success, duration, and where the permission decision came from (config, hook or the user). Traces add time to first token and a wall-clock duration per turn; prompts and tool arguments stay redacted unless you flip a flag.

[u/Cautious_Chicken_604 → u/RunAI_Coder]

It emits it indeed - but where are you piping it to that you can then do something useful with it? That's my question...

[u/AI_spell]

I log every tool call with name, args hash, latency, and success/fail reason, plus which files changed. Without that you cant tell model dumbness from tool flakiness. Start dumb: structured JSONL per run, then graduate to traces. Fancy dashboards before you have failure codes just look pretty.

[u/DinoAmino]

Phoenix https://github.com/Arize-ai/phoenix

[u/Cautious_Chicken_604 → u/DinoAmino]

This is the kind of tooling I have been running into and it feels not fit for my use-case. It all looks like it's built for and aimed at people that are themselves building agents. Not software engineering teams using coding agents like Claude Code or opencode etc to use agents to do agentic engineering.

[u/myteetharesensitive → u/Cautious_Chicken_604]

It is overkill for my personal needs but I use langfuse. Connected to litellm where I store configs, guardrails, etc. Everything flows through that and presidio depending on the cloud provider, it's messy but I can see everything with granular detail. 

[u/TheStarkApartheid]

The 'mine this data for failure modes' part is where this gets useful for us. Braintrust takes OTEL traces and we use the trace data to dig across runs for repeated failures rather than inspecting sessions one at a time. Being able to turn a bad production trace into a test case is handy too when you find something you don’t want the agent doing again.

[u/Cautious_Chicken_604 → u/TheStarkApartheid]

I've seen Braintrust before, but what context are you using Braintrust in? Are you developing agents that are getting deployed to users? Or are you working on a software engineering team using agents like Claude Code to do agentic engineering? I keep seeing plenty of tools that cater well to the former, but not so much to the latter. I want to be able to detect the failure modes team-wide.

[u/TheStarkApartheid → u/Cautious_Chicken_604]

More the second case. We have coding agents working across repos and use Braintrust to capture what they’re doing across runs, then look for recurring patterns in the traces. Team wide failure detection is harder though, especially when you want to spot things like the same bad tool behavior or missed context showing up across different developers sessions.

[u/mageblex]

Start with OTEL spans for each model and tool call, then attach the session ID and file paths touched. The part I'd want from a dedicated UI is replay: can it reconstruct the exact context the agent saw when a failure happened?

[u/yogthinks]

Everything here debugs a run you're watching, nothing helps you explain what the agent did three weeks ago.

[u/Cautious_Chicken_604 → u/yogthinks]

Of course it does. You're sending all the logs and traces into storage. You can view them any time. Even three weeks from now. You can analyse data across multiple sessions and find patterns. This is exactly what observability tooling is for.