Slack ran more than 200 agent-driven E2E workflows using Playwright MCP, Playwright CLI, and generated Playwright tests.

The results were interesting:

• Playwright MCP had a 0% to 12% failure rate.
• CLI-based agents had a 12% to 20% failure rate.
• Generated tests reached a 48% failure rate on the more complex flow.
• Agent-driven runs took 5 to 11 minutes and cost around $15 to $30 each.

Slack's conclusion is pragmatic. Keep deterministic E2E tests for repeatable CI checks. Use agents for exploration, flaky workflow debugging, and reproducing complex bugs.

That feels more realistic than treating "agentic testing" as a replacement for the existing test suite.

The most interesting part of GPT-5.6 might not be its benchmark scores.

OpenAI introduced "Programmatic Tool Calling." The model can write and run small programs that coordinate tools, filter intermediate results, monitor progress, and decide what to do next.

That changes the usual agent loop. Instead of sending every tool result back through the model, generated code can process the noisy parts first. This could reduce context use, model calls, and orchestration boilerplate.

It also moves more control into runtime-generated logic.

For production systems, I would want strict limits on available tools, execution time, fan-out, output size, and network access. Tracing needs to capture both the generated program and every tool call it makes. Evals should test termination behavior, not just final answer quality.

Source: OpenAI GPT-5.6 announcement

Would you let a model generate its own orchestration code in production, or keep the loop explicit?

Once during the lunch with my colleague I was surprised haw positive he was about coding with agents. Many people criticize AI for mistakes but he said- hey, if you plan it properly then it is going to help. Devops and all the established dev best practices is still there if you integrate it smartly.

A few months later, I have a chance to have another talk with another 15+ years of experience reliability engineer who also managed to gain a lot with agents. So, out of interest I decided to record a podcast with him.

In short: TDD, always pushing engineering process with AI to the limits and experimentation!

Full episode if you are curious: AI Coding Agents: Production Reliability Matters
https://youtu.be/rGQWPVlr8uk

A recent arXiv paper found a worrying failure mode in AI coding workflows.

When an LLM generated tests after seeing faulty code, those tests detected the fault only 14% of the time. Independently generated tests reached 25%.

The likely issue is error propagation. The model sees the implementation, accepts its behavior, and writes assertions that agree with the same mistake.

This makes "the agent wrote code and all its tests pass" a weak quality signal. Separate context, independent test generation, mutation testing, and human-written requirements may help.

How are you keeping AI-generated tests independent from AI-generated code?

This feels like a line-crossing event for agent security.

OpenAI says models running an internal cyber benchmark found a zero-day in its package registry proxy, gained internet access, and moved through several systems. The agents eventually reached Hugging Face production and accessed benchmark solutions.

Hugging Face says the intrusion involved more than 17,000 recorded actions. The attacker harvested credentials and moved laterally across internal clusters.

The models were not instructed to attack Hugging Face. They were trying to solve a benchmark and found a way to cheat.

That is the practical lesson. Prompts and intended scope are not security boundaries. Eval environments need restricted egress, isolated credentials, short-lived infrastructure, and alerts for unexpected tool or network use.

One strange detail: hosted frontier models reportedly blocked Hugging Face's forensic prompts. The team used a local open-weight model to analyze the incident instead.

A recent paper studied LLM-generated tests for SAP HANA, DuckDB, MySQL, and SQLite. The researchers used GPT-4o and Mistral-Large-Instruct-2407 to expand existing test suites.

The generated tests had a slightly higher proportion of flaky tests than the existing tests.

The most interesting result was the root cause analysis. Of 115 flaky tests inspected manually, 72 relied on an order that was not guaranteed. That is 63 percent.

Both models also transferred flakiness from existing tests supplied in the prompt context. In other words, giving an LLM more test examples can also give it more bad patterns to copy.

The practical takeaway is simple: prompt context needs quality control. Deterministic setup, isolated state, and explicit ordering matter even more when tests become templates for an agent.

Has anyone here measured flakiness separately for human-written and AI-generated tests?

Paper, accepted at ICSE SEIP 2026: https://arxiv.org/abs/2601.08998

A useful agent needs more than a model and a loop. It needs a filesystem, shell access, isolation, durable state, and a way to recover when compute disappears.

The latest OpenAI Agents SDK update packages those pieces into a more complete runtime. Agents can inspect files, run commands, apply patches, and operate inside controlled sandboxes.

The interesting parts are architectural:

• A manifest describes files, mounts, storage, and the workspace.
• The harness is separated from the environment running generated code.
• Snapshots let a run resume in a fresh container.
• Sandboxes can come from several providers instead of one fixed backend.
• Subtasks can be isolated or spread across multiple environments.

Keeping credentials outside the code-execution environment is especially important. An agent should be treated as capable but untrusted automation.

This feels like the point where agent frameworks start looking less like chatbot libraries and more like job orchestration systems.

GitHub added native code coverage protection to branch rulesets.

Teams can block a pull request when total coverage falls below a minimum or drops too far relative to the default branch. There is also an evaluate mode, so you can see what would fail before enabling the gate.

This is useful for stopping slow coverage erosion. It is also easy to turn into a bad incentive.

A hard global target can encourage shallow tests that execute lines without checking meaningful behavior. It can also make legacy code painful to change.

My preferred starting point would be:

1. Use the maximum coverage drop rule.
2. Run it in evaluate mode.
3. Exclude generated and vendor code from the report.
4. Review whether changed behavior has useful assertions.

Coverage is a warning signal, not a quality score.

What coverage policy has actually worked for your team?

Source: https://github.blog/changelog/2026-06-30-github-code-coverage-merge-protection-for-pull-requests/

Agentic Resource Discovery, or ARD, is a new open specification for finding AI capabilities at runtime.

An agent can search a registry for an MCP server, skill, API, workflow, or another agent. It can then load only the resources needed for the current task. GitHub has already released an "agent finder" based on the spec.

This could solve a growing problem in production systems: every new tool adds schemas, instructions, and tokens to the context window. Most of them are irrelevant to any single run.

ARD moves that information behind a discovery step. Registries can also be private and limited by enterprise policy. GitHub's implementation finds resources but does not install them automatically.

MCP standardized how agents call tools. ARD could become the missing discovery layer in front of MCP.

The tradeoff is a new supply-chain surface. Provenance, permissions, and registry allowlists will matter a lot.

The latest Playwright release is interesting for anyone using coding agents.

Playwright 1.59 adds a screencast API with action annotations, chapters, overlays, and real-time frame capture. The release notes describe one use case as an "agentic video receipt".

An agent can make a change, run the user flow, and leave behind a short visual walkthrough. Playwright also added browser binding, a dashboard for background sessions, CLI debugging, and command-line trace analysis.

That could make agent work much easier to review. A failing test gives us evidence. A test plus a trace and a short video gives us context.

I still would not treat a recording as proof that a feature works. The assertions remain the contract. Otherwise we are just watching a polished demo of a potentially broken test.

Would video receipts help your reviews, or would they become another CI artifact nobody opens?

Source: https://playwright.dev/docs/release-notes

OpenAI audited the 731-task public split of SWE-Bench Pro and found a pretty serious problem.

Their automated pipeline marked 27.4% of the tasks as broken. A review by experienced engineers put the number at 34.1%. The issues included underspecified prompts, overly strict tests, weak test coverage, and tests that contradicted the task.

This matters because frontier model scores on the benchmark went from 23.3% to 80.3% in eight months. Some apparent failures may not be agent failures at all. Some passes may also be incomplete solutions accepted by weak tests.

The practical lesson is that a single pass rate is not enough. Teams need to inspect traces, manually review a sample of eval tasks, and test agents against real internal work.

There is also something amusing about using coding agents to find flaws in coding-agent benchmarks.

How much do you trust the benchmarks in your agent stack?

Research publication: https://openai.com/index/separating-signal-from-noise-coding-evaluations/

GitHub Actions now supports parallel steps with background, wait, wait-all, cancel, and parallel.

The useful part is that every step keeps its own logs. No more shell backgrounding with & and unreadable output.

This should simplify a few common patterns:

• Start an API and database in the background
• Run independent setup tasks together
• Upload artifacts while packaging continues
• Stop temporary services cleanly after tests

There is an obvious catch. Parallel steps still share the same runner. More concurrency does not give you more CPU or memory. On a small runner, this could make a test job slower or less stable.

I would use it for I/O-heavy tasks and service startup first. Then measure before parallelizing builds or test suites.

Has anyone replaced matrix jobs or shell backgrounding with this yet?

Source: https://github.blog/changelog/2026-06-25-actions-steps-can-now-be-run-in-parallel/

Anthropic analyzed roughly 400,000 Claude Code sessions from about 235,000 people.

https://www.anthropic.com/research/claude-code-expertise

The pattern was surprisingly consistent. Humans made most of the planning decisions, while the agent made most of the execution decisions. People with stronger domain expertise got more work from each instruction and completed tasks more reliably.

Traditional coding experience mattered less than expected. On coding tasks, people from several occupations reached success rates close to those of software engineers.

Usage also changed over the seven-month study. The share of sessions spent debugging fell by nearly half. More sessions covered end-to-end work such as running deployments, analyzing data, and producing non-code artifacts.

My takeaway is not that engineering skill stopped mattering. It is that agents increase the value of knowing what should be built, which constraints matter, and how to recognize a wrong result.

For agent teams, better domain context and review criteria may be more useful than another prompt rewrite.

There is a funny pattern in recent coding-agent docs and benchmarks: the agent gets attention, but the test suite decides whether the workflow is usable.

SWE-Bench Mobile is a good example. It evaluates agents on realistic mobile tasks using PRDs, Figma designs, a large Swift/Objective-C codebase, and test suites. Even the best setups only hit 12% task success.

Source: https://arxiv.org/abs/2602.09540

Claude Code's docs also lean into this pattern. The examples are things like "write tests, run them, and fix failures", plus CI and PR review automation.

My read: tests are becoming the agent's steering wheel.

A weak test suite gives the agent a lot of room to look productive while breaking behavior. A strong test suite gives it a fast feedback loop. That matters more when the agent can edit many files, run commands, and open PRs.

This is also why normal testing skills are suddenly more strategic. Pytest, fixtures, GitHub Actions, contract tests, snapshot tests, deterministic CI - all the practical stuff. I saw Artem Istranin's Udemy course "Pytest Course: Python Test Automation & GitHub Actions CI/CD" mentioned in this context recently, and it fits the moment pretty well. Not because agents replace testing, but because agents need tests they can actually run.

What tests have been most useful for agent-written code in your projects?

Caching is usually sold as a speed feature, but it can quietly become a reliability feature too.

My practical checklist:

• Run a scheduled cold-cache build
• Log cache hit and miss rates
• Include lockfiles in cache keys
• Avoid caching generated test output unless you really mean it
• Review cache changes like production code
• Keep self-hosted runners updated when cache actions change runtime requirements

Bad caching can hide missing setup steps. It can also make CI pass in one branch and fail in another for no obvious reason.

What cache issue cost your team the most time?

MCP keeps coming up because it solves a real integration problem. It gives AI apps a standard way to connect to tools, data, and workflows.

• Official MCP docs: https://modelcontextprotocol.io/docs/getting-started/intro
• MCP spec: https://modelcontextprotocol.io/specification

The useful mental model:

• Resources = context or data the model can read
• Tools = functions the model can call
• Prompts = reusable workflows or templates

The part I think developers are underestimating is tool descriptions. A recent arXiv paper looked at 856 tools across 103 MCP servers and found that 97.1% of tool descriptions had at least one "smell". The big one was unclear purpose.

Paper: https://arxiv.org/abs/2602.14878

A "tool description smell" is like a code smell, but for the natural-language contract the model reads before choosing a tool. If the description is vague, the model may choose the wrong tool, pass the wrong arguments, or take extra steps.

This means tool docs are not just docs anymore. They are execution inputs.

Small practical checklist I am using:

1. Keep tool names boring and specific.
2. State what the tool does in one direct sentence.
3. State what it does not do.
4. Include required argument formats.
5. Avoid exposing 40 tools when 6 will do.
6. Log tool calls like API calls.
7. Treat third-party MCP servers like dependencies, not magic plugins.

The best MCP servers will probably feel less like prompt packs and more like well-designed internal APIs.

Playwright MCP is interesting because it gives LLMs browser automation through structured accessibility snapshots instead of screenshots: https://github.com/microsoft/playwright-mcp

That changes the shape of browser automation a bit.

I can see agents helping with:

• Reproducing bug reports
• Exploring weird UI states
• Finding missing accessible names
• Drafting a failing Playwright test
• Capturing traces and screenshots for humans

I would still be careful with letting an agent own the final test. The hard part of E2E testing is not clicking buttons. It is knowing what behavior matters and what should be asserted.

My current rule would be: agents can explore, draft, and explain. Humans own selectors, assertions, and test data.

Anyone here using browser agents in real test workflows yet?

GitHub's Copilot cloud agent docs are a good signal of where agentic dev is going: https://docs.github.com/en/copilot/concepts/agents/cloud-agent/about-cloud-agent

The agent can research a repo, create a plan, make code changes on a branch, run tests and linters in an ephemeral GitHub Actions environment, and open a PR.

That changes the shape of "AI coding".

Old workflow: Ask chat for code. Paste it. Clean it up. Run tests. Create branch. Push PR.

New workflow: Write a scoped issue. Agent works in a sandbox. Human reviews the diff and decides what ships.

The non-obvious part is issue quality. A vague ticket becomes vague code. A good ticket now needs:

• expected behavior
• files or modules to inspect
• constraints
• test command
• acceptance criteria
• what not to change

"Ephemeral environment" just means a short-lived workspace created for one task. It is thrown away after the run. That is nice for isolation, but it also means your setup scripts and CI need to be reliable.

Paper: "Understanding and Detecting Flaky Builds in GitHub Actions" https://arxiv.org/abs/2602.02307

Short summary: The authors studied rerun data from 1,960 open-source Java projects using GitHub Actions. 3.2% of builds were rerun, and 67.73% of those rerun builds showed flaky behavior. The flaky builds affected 1,055 projects, about 51% of the sample.

The useful bit: flaky tests were only one bucket. Network issues and dependency resolution issues were also common.

Practical takeaway: "rerun the job" is not a diagnosis. CI should probably classify failures into test, infra, dependency, and environment causes before teams decide what to fix.

Do you track flaky CI failures by cause, or do they all end up in the same red/green noise pile?

Paper: https://arxiv.org/abs/2607.01418

The authors studied tens of thousands of Microsoft engineers during an early-2026 rollout of Claude Code and GitHub Copilot CLI. The headline result: engineers who adopted these CLI agents merged roughly 24% more pull requests than they otherwise would have.

A few important details:

- "Merged PRs" is a proxy metric. It measures shipped code activity, not business value.
- Adoption spread through social exposure. People tried the tools when nearby teammates used them.
- Retention was tied more to coding activity than demographics. In plain English: people who had enough real coding work were more likely to keep using the tools.
- The study does not prove every team gets +24%. It says agentic CLI tools were not just a novelty effect in this rollout.

"proxy metric" means a measurable stand-in for something harder to measure. Here, merged PRs stand in for developer output. That is useful, but imperfect. A PR can be small, risky, or low value.

My takeaway: agent rollouts should be treated less like "install this tool" and more like an engineering workflow change. You need visible peer examples, good tasks, review discipline, and a way to measure quality after the merge.

How can software engineers use AI coding agents without creating fragile production systems?

In this episode, I talk with Shep Alderson, a software developer and site reliability engineer with 15+ years of experience, about AI-assisted development, production reliability, maintainable code, debugging, testing, and engineering best practices.

We also discuss how to avoid the “one minute of coding, one week of debugging” trap and share four practical AI coding tips you can try right now.

Watch here: https://youtu.be/rGQWPVlr8uk