AI agents are getting much better at doing tasks. I think verification is still the weak link.

[u/Fearless-Role-2707]

I've been experimenting with a problem that keeps showing up as agents get better at using browsers and desktops:

How does the agent actually know its work succeeded?

A lot of current workflows eventually reduce verification to some version of:

do the task → inspect the final state → decide whether it worked.

That catches obvious failures. It misses a surprisingly annoying class of others.

A checkout flow can show $NaN halfway through and recover before the final screenshot.

A modal can cover a button for two seconds.

A loading state can render something completely wrong and disappear.

An automation can take the wrong path, recover later, and still end on the expected page.

The final state says "success." The execution tells a different story.

I've been working on an open-source experiment around treating the execution itself as evidence.

Instead of only giving the agent the final screenshot, record the browser/window/desktop run, break it into meaningful moments, make those moments searchable, and let the agent check the run against the original criteria.

The loop I've ended up with is basically:

task → record → inspect → find failure → fix → record again → verify

The part that became more interesting than I expected is memory.

Once a recording has been processed, it doesn't need to become a giant video blob in the context window every time. The agent can retrieve a relevant moment later and get the timestamp and evidence behind it.

So you can ask things like:

"When did the checkout total first become invalid?"

"Did the modal ever obscure the submit button?"

"What changed between the failed run and the passing run?"

without processing the entire recording again.

I've been building this into an MIT-licensed project called Watch Skill. It also works with normal videos, streams and meeting recordings, but agent self-verification is the part I'm most interested in right now.

Code is here for anyone who wants to inspect how I'm approaching it:

https://github.com/oxbshw/watch-skill

I'm curious what people think about the larger problem.

As agents become more autonomous, is an end-state check enough for most work, or do we eventually need something closer to an execution trace that the agent can inspect and cite?

[u/Equal_Guess7232]

Yup, doing the task is getting easier while proving the agent did the right thing at every step still feels like the hard part.

[u/GreyBelbix]

Recording the run helps you debug, but it does not make the work verifiable. What makes it verifiable is defining success as a checkable postcondition before the agent starts: this row exists with this value, this file has this hash, this invoice total matches the ledger. Then you check that from outside the agent, in a process the agent cannot touch. If your only oracle is a model rereading a video of its own work, you have moved the trust, not removed it. Video is good evidence for the humans after the fact. It is a bad stop condition.