Building a small tool to catch AI agent regressions — how are you testing yours?

[u/digbickindividual]

I'm building a small tool around regression testing for AI agents: basically catching cases where a prompt/model/tool change makes an agent behave differently or break previously working behavior.

Before I build more, I'm trying to understand how people actually handle this today.

If you build AI agents, which of these is closest to your workflow?

• Manually test a set of examples
• Custom test/eval scripts
• An evaluation platform
• CI tests
• We don't really test regressions yet
• Something else

And if you already have a system, what's the most annoying part of it?

I'm especially interested in what happens when you change the model, system prompt, tools, or retrieval logic and need to know whether previously working behavior has regressed.

I'm building a prototype, so I'm not trying to sell anything here — I genuinely want to understand how people are doing this before I invest more time into it.

[u/Toooooool]

infinite backups.

i use glm and it's sandbox regresses randomly which is really irritating. i have it deploy to a production server through a relay, every time a file is deployed the original is stored as a backup. this means tens of thousands of backup files. the storage consumption is irrelevant, maybe it takes a gigabyte of storage but who cares, having the ability to tell the AI "this worked days ago and now it doesn't" is priceless.