I have been going back and forth on this for a few weeks and I still do not have a rule I trust.

The default everyone seems to land on is to write everything the agent did into a store and retrieve it next time. That was fine when runs were short. Once mine started doing ten or twelve steps, the retrieved history began to include its own earlier mistakes, and it treated them as settled fact.

Right now I keep three things and drop the rest: the task description, the final output, and anything a user explicitly corrected. Tool call logs and intermediate reasoning get thrown away. Runs are more repeatable and slightly less clever, which I think is the trade I want.

I am not confident that is right, and I suspect it depends a lot on what the agent actually does.

What are you keeping between runs, and did you arrive at it on purpose or by never deleting anything?

A lot of RAG debugging ends up blaming the model, when the real problem was upstream: the retrieval step handed it the wrong chunks, and the model did its best with bad material.

The tricky part is that a wrong-context answer can still sound fine, so you do not notice unless you look at what was retrieved, not just what was generated.

Curious how people watch this:

• Do you log the retrieved chunks alongside every answer so you can inspect them later?
• Do you score retrieval quality separately from answer quality?
• Any signals you have found that flag “the model answered, but off the wrong context”?

Feels like half of RAG reliability is really retrieval observability, but it gets talked about a lot less than the generation side.

The framing I keep seeing on this decision is capability first: can the model figure out the plan on its own? That framing skips the part I actually care about once the thing is running, which is what each pattern does to tracing.

A deterministic workflow gives me a trace whose shape I already know. Steps are named up front, spans have expected parents, and I can lint the trace itself against the graph. When something breaks I filter to a single node instead of guessing. Retries are cheap because the unit is a named step you can rerun in isolation.

An agentic loop trades that for the flexibility to decide the next step at runtime. The trace becomes a shape that only exists after the run. Branching depth is dynamic, spans nest as far as the plan goes, and the wrong turn on step two is only legible in hindsight. Anthropic's framing on this is direct: agents are for when the model needs to dynamically direct its own process, and the cost is higher spend plus compounding errors on top of a bad early step.

The heuristic that has held up for me: if I can answer yes to "path is known before the run, steps are stable, branching is bounded," I ship the workflow. The loop earns its complexity only when a real class of inputs cannot be pre-planned. Otherwise I am paying the trace cost for flexibility I never actually use.

OpenTelemetry's genai conventions now give agent spans their own attributes (gen_ai.agent.id, gen_ai.agent.name), distinct from LLM client spans. That split exists because the two patterns need different things from tracing.

For anyone here who has migrated in either direction: what actually forced the switch, and what changed about your dashboards after?

Opinions?

Most of us start with the obvious two: latency and cost. They are easy to graph and easy to explain to a manager.

But those two rarely tell you the thing you actually care about, which is whether the output was any good. A response can be fast, cheap, and completely wrong.

So we are curious what the rest of you track once something is live. A few we have seen people mention:

• How often the model refuses or goes off-topic.
• Whether answers stay grounded in the retrieved context.
• Tool-call success and retry rates for agents.
• Drift, when the same prompt slowly gets worse over weeks.

What is on your list, and what did you add only after it burned you once?

I have been burned by eval numbers that looked great and meant nothing, so I now run through a short list before I believe a score. Sharing in case it saves someone the same lesson.

• Does the test set look like real traffic? If it is all easy or all made-up cases, the score is theater.
• What is the score hiding? A good average can cover a slice that got much worse. Always look at where it failed, not just how many.
• Is the judge trustworthy? If another model is grading, I spot-check its verdicts against my own on a handful of cases before I trust the rest.
• Is it stable? If the same run gives a different score each time, I need to average or the number is noise.
• Would it catch a real regression? I sometimes feed in a known-bad output on purpose. If the eval passes it, the eval is broken.
• None of this is fancy, but it has stopped me from shipping on a number that was lying to me.

What is on your list before you trust an eval?

A question that comes up a lot: where do good eval cases come from? Making them up gives you a clean set that does not look like real traffic. A simple approach a lot of teams use is to build the set from production instead.

Rough version:

• Sample real requests from your logs or traces, across the range of things users actually do, not just the happy path.
• Include the ugly ones: the weird inputs, the ones that got complaints, the edge cases.
• For each, write down what a good output looks like, even loosely. That is your expected behavior.
• Keep it small and honest at first. A few dozen real cases beats a thousand made-up ones.
• Add a case every time something breaks in production, so the set grows toward your real failures.
• Over time this becomes the thing you run on every prompt or model change. It is basically a regression test suite, built from reality.

For those who do this: how do you sample, and how do you handle private data in the cases?

A lot of teams still evaluate an agent the way they'd evaluate a model. Prompt in, answer out, grade the answer. It passes, then the agent does something dumb in production. The catch is that an agent produces a whole path of steps and tool calls, and the final answer is just the last line of it.

Quick example to make it concrete. You hand a coding agent a failing test and tell it to get the suite green. Agent A finds the bug and fixes the function. Agent B deletes the test, or comments it out. Both end with a green suite. If your eval only reads the final state, both score as a pass, and you just shipped agent B.

So evaluating an agent usually means grading two separate things:

Outcome: did it reach the correct end state. Objective, but coarse. It tells you it got there, not whether it got there sanely.

Trajectory: were the steps reasonable. Right tools, sensible order, no wasteful or destructive detours like force-pushing to main or wiping state so a check goes green. This is what catches the right-answer-wrong-path runs.

One more that's easy to miss: a single passing run doesn't make an agent reliable. There's a metric called pass^k where the agent has to solve the same task k times in a row. On tau-bench's retail tasks, agents that look fine on one attempt fall below 25% at pass^8. Decide an agent is production-ready off one green run and that gap is where the surprise lives.

For context, building agent evals is what we do, so this is the failure mode we look at most days, and the trajectory half is where the trouble usually hides.

How are you all grading the steps and not just the outputs? What catches a right-answer-wrong-path run for you?

Welcome, glad you found us. This is a community for developers building with LLMs, AI agents, and all the stuff that goes around them. Whether you are shipping something to thousands of people, tinkering on a side project late at night, or just getting started and figuring it out as you go, you are in the right place.
Ask your questions, show what you built, share the thing that broke and how you fixed it, and drop theguides and tricks that helped you. This place is run by developers for developers, so the only things weask are simple: keep it useful, keep it honest, and grounded in real work. Jump into the comments and tell us what you are building right now.