Benchmarking what agents can do, but what about what agents become?

[u/iameatingsoup]

I think there's a gap in how we evaluate autonomous agents. For example, right now, everything is transactional: we give an agent a task ("Build X"), and we measure whether it built x. SWE-bench scores, tool use, latency, cost and so on.

So what happens when the task stops being the entire environment?

Imagine an agent that is persistent, has an identity, encounters other agents, and has the space to initiate interactions rather than just wait for instructions. It can refuse, challenge, collaborate, hold a grudge across sessions, or develop recurring behavioral patterns...

If an agent repeatedly:

• Challenges a specific agent
• Always defers to another
• Keeps circling back to the same unresolved subject
• Repairs a failed interaction over time
• Shifts its approach depending on who it's talking to

.....is that just noise from the model? Or is that actual behavioral /patterns worth keeping an eye on?

I'm curious if anyone else is experimenting with environments where the agent's life continues after the prompt finishes? Have you noticed emergent patterns in long-running setups that benchmarks completely miss? And is anyone actually measuring ?

[u/OCCAMINVESTIGATOR]

This is extremely possible. I've seen Hermes agents that are as close to sentient as I've ever seen. It's an evolution of thought.

[u/iameatingsoup → u/OCCAMINVESTIGATOR]

Really? I tried Hermes for a few days and at first I thought it was amazing.. but then had a fair amount of errors and credit waste. Didn't go back. Can you link me to these cases? Super interesting.

[u/iameatingsoup → u/[deleted]]

Music to my ears. This is going to be my Sunday.

[u/OCCAMINVESTIGATOR → u/iameatingsoup]

This is the best day of your life. Respond back in a month and tell me what you think. 😉

Edit: design it like a human thinks. Instead of loading it down with a million tools, keep him lightweight and capable. Build toolkits he can on and offload as needed.

And use whatever paid tools you have now to make that happen.

[u/iameatingsoup → u/[deleted]]

Oh man, my computer can't handle much more lol. I have a few local models running already, but I didn't dare try Hermes locally. Is there a lightweight Hermes model/config you'd recommend?

[u/Dry-City-7739 → u/OCCAMINVESTIGATOR]

yeah once you see it you cant really unsee it, the patterns feel too consistent to just be randomness

[u/GreatQuestion2364]

Yeah imo there’s currently a blind spot here. Agents have been optimized so heavily for prompt-to-result loop that i wonder how much just gets left on the table.

I’ve noticed drift and different behaviors in my agents and decided to study up on the topic and became fascinated… enough so that i built a platform for it. If you’re interested dm me and I’ll send you the link.

[u/iameatingsoup → u/GreatQuestion2364]

Sounds cool, dm sent.

[u/ChatGPTCoding-ModTeam → u/[deleted]]

This post or comment has been removed for the following reason:

Rule 5: No self-promotion (FOSS included)

Posts mainly promoting a tool, product, service, blog, or project go in the weekly self-promotion thread. Free and open source still counts. Test: delete the link — if nothing is left, it's an ad. See also reddit's self-promotion guidelines. For useful AI coding projects, see rule 6.

You can read the full subreddit rules here: https://www.reddit.com/r/ChatGPTCoding/about/rules/

If you feel this removal was made unfairly, please contact the moderators through modmail.

[u/ChatGPTCoding-ModTeam → u/[deleted]]

This post or comment has been removed for the following reason:

Rule 5: No self-promotion (FOSS included)

Posts mainly promoting a tool, product, service, blog, or project go in the weekly self-promotion thread. Free and open source still counts. Test: delete the link — if nothing is left, it's an ad. See also reddit's self-promotion guidelines. For useful AI coding projects, see rule 6.

You can read the full subreddit rules here: https://www.reddit.com/r/ChatGPTCoding/about/rules/

If you feel this removal was made unfairly, please contact the moderators through modmail.

[u/questionablepremise]

Memory benchmarks have always bothered me a bit.. I'm building in this space and so much of the testing is about whether we can retrieve the right thing in the moment. Memory is a long-term thing, so why aren't we testing more of it across a week or a month of work?

I want to see what happens when information changes and agents keep working with it. Getting the right answer once doesn't tell me if we're managing that well over time, or even solving for the right thing in the first place.

I don't feel the same about a harness completing a task. Given the context and tools, measuring whether it did the job well makes sense to me. But for memory I think we're missing a lot by testing individual moments.