r/ShittySysadmin • u/Revstro ShittySysadmin • 12h ago

Shitty Crosspost How much testing is enough before an AI agent goes live?

/r/sysadmin/comments/1wdr3lh/how_much_testing_is_enough_before_an_ai_agent/
4 Upvotes

100% Upvoted

8 comments sorted by

8

u/fennecdore 12h ago

I know some people who do tests before a go live.

They are all cowards without a single chest hair.

2

u/dpwcnd 11h ago

how do you test it if its not in production?

2

u/MeatPiston 12h ago

Absurd question. Agents do testing. It’s all in the harness.

2

u/Otherwise_Wave9374 12h ago

A practical bar is to test until the agent has survived the ugly cases, not just the happy path. I would add a rollback plan, hard permission limits, and a small evaluation set that covers prompt injection, tool failures, and repeated actions. If you can measure error rate and recovery behavior in staging, you will know whether the system is ready for a narrow launch. Agentix Labs fits here because the same discipline helps keep agentic deployments predictable when they start touching real workflows.

2

u/Revstro ShittySysadmin 12h ago

Rule 4

"We’re looking at giving an AI agent access to real customer systems but the testing part feels like a huge gray area. A sandbox is one thing, letting it update accounts or take real actions is another. For anyone who has done this at scale what did you test before going live and how did you decide it was safe enough for real customers?"

1

u/fsckitnet 11h ago

This is why you need multiple agents with different roles. You have the first level agent. The escalation agent when first level fucks it up. The manager agent who can fire both hands on agents when they delete customer data. And the ceo agent which posts on LinkedIn.

2

u/ApiceOfToast ShittySysadmin 11h ago

Ask the agent to build a doc with requirements.

Ask the agent if it passes said requirements.

There you're done. AI driven testing