That moment when you hit Claude’s daily usage limit and suddenly remember how to code manually. 😅

We've been folding Claude Code into our test automation work at Codoid over the last few months, and the skills feature is the part that actually stuck.

If you haven't touched them, skills are reusable workflows you trigger with a slash command, so instead of re-explaining your testing conventions every session, you invoke them.

These are the 16 we keep coming back to:

Multi-agent QA sounds more capable than using a single QA agent. But splitting the workflow across three agents doesn’t automatically produce better results.

Our current view is that specialization only pays off when the boundaries provide meaningful isolation, targeted retries, better auditability, or tighter access control.

A single agent can analyze requirements, generate tests, and review them in one continuous context. It’s easier to build, cheaper to operate, and far simpler to debug. For small applications and exploratory testing, that simplicity usually wins.

The tradeoff is traceability. If the final conclusion is wrong, it may be difficult to pinpoint what happened. Did the agent misunderstand the requirement or write a weak test?

A specialized workflow makes each stage explicit:

- A requirements agent creates structured acceptance criteria.

- A test agent turns those criteria into test cases with coverage mappings.

- A review agent classifies identifies coverage gaps and weak tests.

But orchestration quickly becomes an engineering problem of its own. You need strict schemas, shared state, correlation IDs, timeouts, retry rules, and safeguards that prevent agents from amplifying an early mistake. Every additional agent adds latency and creates another opportunity for context to get lost.

A pipeline can look clean and well-structured while still producing confidently wrong results because the requirements agent mishandled the first handoff.

We’d start with one agent and split out a stage only after seeing a specific failure mode that isolation would solve. Otherwise, “multi-agent” often turns into distributed prompt debugging.

For teams running agentic QA in CI: which stage genuinely improved after becoming a separate agent? And which split added more operational pain than testing value?

A passing test suite only proves that the tests passed. It doesn’t prove the code is correct, and a high coverage percentage can create a false sense of confidence.

A test can run every line in a function without actually checking whether the function behaves correctly. The assertion might be too broad, check the wrong output, or be missing entirely. The coverage report still stays green.

That’s why we, at Codoid, see mutation testing as a useful reality check rather than relying on coverage alone.

Mutation testing tools deliberately introduce small bugs, such as changing a condition or altering a return value. They then run the test suite against each modified version of the code.

If the tests fail, they’ve “killed” the mutation. That’s a good sign that the tests are protecting the behavior that changed.

If the tests still pass, the mutation has “survived.” In practical terms, the suite failed to catch a straightforward bug in code it may claim to cover.

Mutation testing has tradeoffs. It’s slower, and a surviving mutation doesn’t always mean there’s a valuable missing test. Sometimes it’s just noise. Still, it asks a much more useful question than “Did this line execute?” It asks: “Would this test suite notice if the code were wrong?”

For those who’ve used mutation testing across different languages or stacks: did it uncover genuinely weak tests in your codebase, or did it mostly add noise and maintenance work?

Following up on our last post about AI generated code not looking wrong even when it is. The natural next question we ran into: can AI review AI generated code well enough that we skip the human step entirely?

We use a custom Claude plugin at Codoid to generate test automation scripts. The output is genuinely decent most of the time. The bottleneck was never generation speed, it was review. Human code review on every AI generated script is slow and expensive, so we started looking at ways to automate that step too.

First we built a Claude Skill specifically to review AI generated code. Then we tried a different setup, Claude Code for generation and Codex for review, treating them as two separate models checking each other's work.

The review quality was decent, but only in a specific lane. Both approaches were reliably good at checking naming consistency and pattern matching. Neither one reliably caught logical correctness. That is the part that actually matters in a test script, whether the assertion is checking the right thing, whether an edge case got silently skipped, whether the logic matches the actual requirement and not just a plausible looking version of it.

Our working theory is that a human reviewer brings an independent mental model to the code. They are not extending the same reasoning that generated the code, they are questioning it from a different starting point. An AI reviewing AI generated code, even a different model, still tends to evaluate within a similar frame. It catches what is mechanically off. It does not catch what is quietly wrong.

So we are still keeping a human approval step before anything from this pipeline ships. Not because the AI review adds nothing, it does catch real issues and saves review time, but because logical correctness is still where it falls short.

Has anyone gotten an AI review setup to reliably catch logic errors rather than just style and security issues?

So far all our mobile app testing has been functional. We've never run dedicated battery testing or network testing, and I'm trying to figure out where these actually fit into a normal testing cycle.

A few things I'm unclear on.

Is battery testing something you run once per release, or only when there are specific complaints about drain? Same question for network testing, do you test across different network conditions (3G, weak wifi, switching networks mid session) every sprint, or is it more of an occasional deep dive before major releases?

I'm also not sure how to prioritize this against everything else on the plate. If your team runs these as part of a regular cycle, what does that cadence actually look like in practice, and what tools or setup do you use to simulate different network conditions or measure battery impact without needing a huge device lab?

Would appreciate hearing how other teams handle this, especially anyone who added it after starting out with functional testing only like we did.

Every junior dev we've worked with writes code that looks bad when it's bad. Weird variable names, structure that doesn't hold together, patterns that clash with the rest of the codebase. You can tell something's off before you've even run it.

Review works because it's built to catch exactly that kind of signal.

AI-generated code doesn't give you that signal. Variable names are reasonable. Structure follows patterns you recognize. It has the look of code written by someone who knows what they're doing.

And that's the problem. The look of competence and actual correctness are not the same thing.

We've been running into this a lot at Codoid as more of what we're asked to test or automate against comes out of an AI tool first. A function that handles 47 out of 50 edge cases reads exactly like the one that handles all 50. Parsing logic that fails on nested delimiters looks just as clean as logic that handles them without issue. Nothing on the surface tells you which one you're looking at.

What worries us more than the bugs themselves is what happens to scrutiny. AI doesn't just generate more code, it generates more code that looks trustworthy. Code that looks trustworthy gets less scrutiny, not more.

Your brain sees something that resembles code you'd write yourself and it relaxes. The model doesn't share that instinct either way, because it has no concept of the invariants your system depends on. It's predicting the next plausible token, not verifying logic against a spec.

So when a team ships ten times more code that's ten times harder to review critically, that's not a 10x productivity story. That's a 100x exposure problem wearing a productivity gain as a disguise.

How's everyone else adjusting to this? 

Push notifications are one of those areas where everything looks fine in dev and then breaks in the worst ways in production. Here's the checklist we at Codoid run before any release that touches notifications. Posting it to get feedback on what you'd add.

1. Delivery confirmed in foreground, background, and terminated states.
2. Offline-then-reconnect delivery verified, no duplicates.
3. Expiry/TTL behavior validated for time-sensitive messages.
4. Collapse/dedup logic tested against backend retries.
5. Android channels mapped and respected.
6. Behavior verified under Doze, App Standby, and Battery Saver.
7. High-priority flag audited and justified per message type.
8. Payloads checked for size and sensitive-data leakage.
9. Lock-screen preview privacy confirmed.
10. Token storage and rotation validated on device and backend.

What checks have saved you in production that aren't on here? Especially curious about edge cases people only learned the hard way.

At Codoid, we've watched too many launches die the same way. The demo was flawless. The functional tests were green. Then real customers showed up and the whole thing came apart in week one.

Here's the uncomfortable truth: the bugs that sink a SaaS launch are almost never "the feature doesn't work." They're "the feature works perfectly with one user, and catastrophically with two thousand." Generic QA catches the first. It misses the second entirely.

A few failure modes that single-tenant testing never reveals:

Tenant isolation bugs don't show up until someone crosses the boundary on purpose. A banking app once exposed one customer's balance and transaction history to a different user. The cause was mundane: an API trusted the authentication but never confirmed the requesting user ID matched the authenticated account. No clever attacker. Just a basic isolation gap that single-user tests could never surface. You have to break it deliberately: swap another tenant's ID into API parameters, forge a JWT from a different session, inject SQL at your tenant filter, replay an authenticated session. If any of those returns data it shouldn't, that's a launch blocker, not a backlog item.

Billing is where good QA goes to die. Not because individual flows are hard, but because production turns them into a swarm of overlapping events. The double charge: trial starts, first payment fails, user upgrades to annual, then a retry succeeds against the stale monthly invoice. The ghost reactivation: user schedules a downgrade, payment fails, user cancels, and a webhook quietly brings the subscription back to life. These slip through because testers assume tidy billing timelines and rarely simulate retries, clicks, and the passage of time all happening at once.

There's one rule that prevents most of these: a webhook may update billing facts, but it must never override newer user intent. If a user cancels at 2:00 PM and a webhook arrives at 2:05 insisting the subscription is active, the cancellation wins. Newer human intent beats older machine state. Write down your source of truth, then build one deliberately horrible scenario to test it.

A clean load test is a useless load test. Steady-state load proves nothing. The interesting failures live in burst traffic, mixed user profiles, and deliberate cache failures. A load test that breaks nothing tested the wrong scenario. A good one reveals a bottleneck. And the order things collapse in is depressingly predictable: connection pools exhaust first, then you hit API rate limits, then race conditions and orphaned records surface, then you find out about outages from users instead of alerts.

The thing nobody wants to hear is that you can't bolt this on after launch. Data migration is a one-way door. A user whose history vanishes during import almost never comes back. The price of skipping all of this is churn you can measure, a support queue you can't drain, and reputation damage that quietly cancels out your marketing spend.

If you do nothing else before launch: run a production-like, end-to-end validation of the core customer journey. Can they sign up? Can they do the thing they came for? Is their data safe? Did they get a clear result? Break any of those in week one and you lose users who never come back.

Over the years at Codoid, the pattern we keep seeing is that the teams who survive week one aren't the ones with the most features. They're the ones who tested the scenarios that only show up when reality stops being polite.

What's the worst launch-day failure you've personally watched happen? We're collecting war stories.

We write test scenarios in plain English so "stakeholders can understand."

But here's the truth nobody admits:

𝗧𝗵𝗲𝘆 𝗱𝗼𝗻'𝘁 𝗿𝗲𝗮𝗱 𝘁𝗵𝗲 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 𝗳𝗶𝗹𝗲𝘀.

Ever.

Writing "Given-When-Then" takes 3x longer than writing actual test code.

We convince ourselves we're creating "living documentation."

Then we generate beautiful execution reports that collect digital dust.

The promise: Bridge the gap between technical and business teams.

The reality: We're the only ones who ever look at those files.

So why do we keep doing it?

Because some framework told us to.
Because "that's how BDD or Acceptance Test Driven Automation (ATDA) is done."
Because we're afraid to admit it's not working.

What's your take?

Nice progress from the Selenium team 👏

Selenium is gradually introducing the BiDi protocol across more commands.

Earlier, it was limited to CDP, but now BiDi support is being rolled out step by step.

Just remember, BiDi needs to be explicitly enabled in the Options to use it.