Justy: Okay, so Slack just ran two hundred plus agent-driven E2E tests and published the whole thing, and I think this is the clearest explanation I've read yet of where agents actually fit in testing.
Cody: Yeah?
Justy: They didn't try to crown agents as the new testing paradigm. They measured three different approaches—MCP, CLI, and generated code—on two workflows, and the results are… actually honest.
Cody: Honest how?
Justy: They show where each one breaks. MCP is the most reliable but slow. Generated tests are fast but fragile on complexity. And they're transparent about cost—fifteen to thirty bucks per run. That's real.
Cody: Mm-hm.
Justy: But the actual insight isn't about speed or cost. It's the distinction they lead with: traditional tests enforce a journey. Agents verify a goal.
Cody: That's the whole thing, yeah. You're testing whether the user can accomplish what they set out to do, not whether they took a specific path to get there. The agent tries different valid sequences and still lands on the same outcome.
Justy: Right. And they measured this—only about twenty percent of runs followed the exact same action sequence. Same final state, totally different paths.
Cody: That's actually interesting because it means the agent is exploring the UI surface, not following a script.
Justy: Exactly. So the first thing to understand is the execution models. They tested three: agent plus Playwright MCP, agent plus Playwright CLI, and then a separate path where the agent generates deterministic Playwright code upfront.
Cody: The difference being what?
Justy: MCP gives the agent a persistent view of the browser state. The agent sees DOM snapshots and logs and decides the next move. CLI is step-by-step—the agent runs a command, sees the result, then decides what's next. Generated code is pre-written, deterministic, just runs like a normal test.
Cody: Got it. And the reliability numbers?
Justy: MCP crushed it on simple flows—zero percent failures on thread reply. On the harder search discovery flow, it hit about twelve percent. CLI was rougher. Twelve to twenty percent failure rates, mostly timing and session stuff. And generated tests…
Justy: …eight percent on simple, but forty-eight percent on complex. They were dying late in the flow.
Cody: So the reliability gap widens as the flow gets harder. That tells you something about state management.
Justy: Yeah, and they actually call this out. MCP keeps a live, stable view. CLI is rebuilding state from snapshots at each step. Small timing inconsistencies compound over a longer flow.
Cody: Right. And there's the in-session context thing too—the agent in MCP is probably reusing successful interactions from earlier in the same run, while CLI feels like starting from scratch each time.
Justy: Speed numbers?
Cody: Generated tests were fastest overall, about three minutes including the upfront generation cost. But raw execution per test run was like thirty to forty-five seconds. MCP averaged five to eight minutes. CLI nine to eleven. So deterministic is way faster.
Justy: But you're paying the agent cost every run with MCP and CLI, while generated tests pay it once.
Cody: Exactly. In CI, that one-time cost becomes nothing. You run the test fifty times, the generation is amortized away. Agent-driven workflows pay the full cost every single execution.
Justy: Which brings us to cost. Fifteen to thirty bucks per run. They broke down token usage too—MCP was more efficient than CLI or code generation. The execution model mattered more than which model you picked.
Cody: So you can't just swap in a cheaper model and solve the cost problem. The architecture of how you're running the agent is the load-bearing part.
Justy: Right. And here's the actual insight—and I think this is why I liked this piece so much. They don't try to claim agents are the new testing paradigm. They say agents are a new layer. Complementary.
Cody: Mm-hm.
Justy: You still need deterministic tests for regression and CI speed. Agents are for exploratory validation. Catching UI flexibility, state variability, weird edge cases where the user can reach the goal through a path the original test writer didn't think of.
Cody: That's actually the right way to think about it. Deterministic tests are brittle by design—they enforce a specific sequence. That's a feature for regression. Agents are flexible by design. They adapt. That's a feature for exploration.
Justy: And they're expensive and slow, so you don't run them on every commit. You run them periodically or before major releases. Different layer, different purpose.
Cody: The stack is now unit tests, traditional E2E for regression, and agentic E2E for exploratory validation. That's a coherent story.
Cody: Fair. It's solid work.
Justy: Yeah. This one's the kind of piece that lands because it's not trying to sell you on agents—it's trying to help you use them right.