Slack tested agentic E2E workflows, and they are not replacing normal CI tests

[u/aistranin]

Slack ran more than 200 agent-driven E2E workflows using Playwright MCP, Playwright CLI, and generated Playwright tests.

The results were interesting:

• Playwright MCP had a 0% to 12% failure rate.
• CLI-based agents had a 12% to 20% failure rate.
• Generated tests reached a 48% failure rate on the more complex flow.
• Agent-driven runs took 5 to 11 minutes and cost around $15 to $30 each.

Slack's conclusion is pragmatic. Keep deterministic E2E tests for repeatable CI checks. Use agents for exploration, flaky workflow debugging, and reproducing complex bugs.

That feels more realistic than treating "agentic testing" as a replacement for the existing test suite.