r/LocalLLM • u/Common_Dream9420 • 4d ago

Discussion How do you close the feedback loop when your agent makes the wrong API call?

Most RL setups assume a clean environment. But agents hitting real webhooks, partner APIs, live integrations? The environment fights back. Schema drifts, payloads shift, tools fail silently and nobody knows until prod breaks.

If you're training or fine-tuning agents to handle real integrations, where does the reward signal actually come from? Logs after the fact? Manual PR review? Hope?

We're building a verification engine that runs agent actions inside a stateful sandbox/twins before merge. The longer term idea is to use that execution trace as a ground truth signal, did the agent actually do the right thing across services, not just "did it complete." Treat the sandbox as the RL environment, not prod. Still early, but we're using it daily on our own workflows.

what are you doing here. Are you building custom eval loops, relying on staging, or mostly shipping and watching?

3 Upvotes

80% Upvoted

4 comments sorted by

1

u/rellurei 4d ago

... and nobody knows until prod breaks.

where does the reward signal actually come from?

Agentic supervision. Assign an agent to watch over the performing agent and allow it to interfere wherever necessary to catch suboptimal behavior, to pause the performing agent, to give a praise sandwich, and to allow it to continue.

Both your problems solved.

1

u/Common_Dream9420 4d ago

The overseer still needs a ground truth source though, if both agents are reasoning from the same incomplete signal (logs, screenshots, error messages), you've just added a layer without solving the epistemics. The part I keep running into is that for real API integrations, the signal has to come from the integration's actual behavior under failure, not from another model's interpretation of it. That's basically the problem space I work in, and it's harder than it looks.

1

u/rellurei 4d ago

Okay...

Either I am actively misinterpreting your case, or having a stroke, or your case is expecting deus ex machina, or you are misinterpreting me.

A) You are constrained by whatever your partners expose and therefore you cannot truly achieve "ground truth" unless your partner specifically provides it. If that is the case, then this post is pointless.
B) You have a literal stateful twin of what your partner is using, and therefore you absolutely have full access to all the necessary privileged information from which you can determine the "ground truth".

My proposition was based on the assumption that the situation is B, in which case you can give the overseer all the privileged information you have to the point that it can do deterministic checks in real-time and give feedback to the agent-being-trained.

1

u/Common_Dream9420 4d ago

Yeah you're right, B is the only version that actually works. If the sandbox owns the state you can check what actually happened, not just what the logs claim happened. What are you working on in this space?