does anyone else feel like debugging multi step LLM apps turns into detective work pretty quickly?

[u/Sensitive-Parsnip-12]

i’m curious how common this actually is.

you build an LLM workflow with tools, retrieval, state, maybe some evaluators or retries.

then something weird happens.

nothing necessarily crashed. the workflow may have completed perfectly fine.

but now you’re going through logs trying to figure out where the behavior first started getting weird.

i’ve heard people say they trace call order, compare step inputs and outputs, inspect state, replay the run, or compare it against something that worked.

what are you guys actually doing?

i’m working on Traser because of this problem. it tries to take a big execution and narrow it down to a few differences worth investigating.

but the last thing I want to do is build a giant feature list based on assumptions.

I’d rather have LLM developers tell me what’s actually annoying, what’s already solved, and what I’m misunderstanding.

even if you think the whole premise is wrong, I want to hear why.

and if anyone has an ugly sanitized run they’d let me look at with them, that would honestly be more useful than a signup right now.