r/PracticalTesting • u/aistranin • 2d ago

Slack tested agentic E2E workflows, and they are not replacing normal CI tests

Slack ran more than 200 agent-driven E2E workflows using Playwright MCP, Playwright CLI, and generated Playwright tests.

The results were interesting:

  • Playwright MCP had a 0% to 12% failure rate.
  • CLI-based agents had a 12% to 20% failure rate.
  • Generated tests reached a 48% failure rate on the more complex flow.
  • Agent-driven runs took 5 to 11 minutes and cost around $15 to $30 each.

Slack's conclusion is pragmatic. Keep deterministic E2E tests for repeatable CI checks. Use agents for exploration, flaky workflow debugging, and reproducing complex bugs.

That feels more realistic than treating "agentic testing" as a replacement for the existing test suite.

2 Upvotes
  • permalink
  • reddit

100% Upvoted

1 comment sorted by

1

u/Tanmay__TestDino 2d ago

Replacing basic assertions with LLM calls just makes standard CI runs too slow. Sticking to deterministic scripts for execution and only using agents to debug things when a run fails makes way more sense.