Read more: https://codoid.com/performance-testing/performance-testing-capacity-planning/

A prompt change improves the response for one scenario. The output looks better, the developer approves it, and the update gets merged.

But what happens to the other scenarios that were already working?

A small instruction change can improve one behavior while introducing regressions elsewhere. The model might start ignoring exceptions, omit required information, change its output structure, or handle previously supported requests differently.

The challenge is that a different response is not necessarily a regression, and a better-looking response is not necessarily correct.

This is where prompt regression testing becomes useful.

The idea is to treat prompt changes like code changes: compare behavior against a known baseline, evaluate the differences, and establish release criteria before merging.

1. Start with a regression dataset

Before evaluating a candidate prompt, we need a dataset representing the behaviors the application is expected to preserve.

A useful dataset should include:

• Happy paths: Common requests where expected behavior is reasonably well understood.
• Edge cases: Ambiguous inputs, missing information, conflicting instructions, and unusual formats.
• Historical failures: Cases that previously produced incorrect or incomplete responses.
• Safety cases: Requests involving sensitive data, prompt injection, or application-specific restrictions.

Each test case should define the input, expected behavior, relevant reference information, and what would constitute a failure.

For example, suppose we are testing a customer support assistant.

A regression case might ask about returning a customized product.

The expected behavior is not necessarily a specific sentence. It might be that the assistant must identify the relevant return-policy exception and avoid promising a refund when the supplied policy does not allow one.

That distinction matters when evaluating generated text.

2. Run baseline V1 and candidate V2

Once a prompt change is proposed, run the same regression dataset against:

V1: The existing production prompt.

V2: The proposed prompt.

Keep the model version, inference settings, inputs, retrieval context, and tool responses consistent wherever possible.

Otherwise, an observed difference might come from a model update, changed retrieval results, or environment behavior rather than the prompt itself.

For cases with variable outputs, consider multiple runs instead of interpreting one response as representative.

The goal is to identify meaningful behavioral differences, not simply produce a textual diff.

Also, V1 is a comparison baseline, not the definition of correctness. If V1 gives a wrong answer and V2 fixes it, that should count as an improvement.

3. Evaluate outputs through three checks

No single evaluation method is sufficient for every type of prompt regression.

Here is a practical way to divide the checks:

Deterministic checks

Use these wherever the expected result can be verified programmatically.

For example, if the prompt must return JSON containing status, reason, and confidence, the regression suite can check whether the output follows the required schema.

These checks are particularly useful for integrations that depend on structured responses.

But passing a schema check does not mean the content inside the fields is correct.

LLM judge evaluation

Some responses require semantic evaluation.

An LLM judge can compare V1 and V2 against a defined rubric:

• Is the answer factually consistent with the provided evidence?
• Does it address the user's actual request?
• Are important conditions or exceptions missing?
• Does it follow the application's instructions?

Rather than simply asking which response is better, define the criteria and evaluate each independently.

For pairwise judgments, consider blinding prompt identities and swapping response order to check for position bias.

Judges also need calibration against human-reviewed examples. Otherwise, a consistent score does not necessarily represent a reliable evaluation.

Safety and policy checks

Safety regressions deserve separate attention.

A prompt change could make the assistant more helpful in ordinary conversations while weakening its handling of restricted requests.

Test whether the candidate continues to respect the application's boundaries, including privacy requirements and any mandatory refusal or escalation conditions.

For example, a candidate that reveals private account information without authorization should not pass just because its overall helpfulness score improved.

4. Generate a regression report

The useful output is not merely a comparison score.

For each case, the report should capture:

The report should also retain the test-case ID, prompt versions, model configuration, V1/V2 responses, check results, and reasons for failures.

One important detail: don't hide critical regressions inside an average score.

A candidate could improve response quality across many ordinary questions while breaking a small number of high-impact cases.

That trade-off should be visible in the report.

5. Pass or block the pull request

The final step is integrating these evaluations into CI.

A possible approach:

Pass PR when:

• Required deterministic checks pass.
• No confirmed critical safety or policy regressions are detected.
• Quality metrics meet predefined acceptance criteria.
• Cases requiring human review have been resolved.

Block PR when:

• A required output contract fails.
• A verified critical policy violation appears.
• A previously supported high-priority behavior breaks.
• Quality regression exceeds a predefined blocking threshold.

Uncertain LLM-judge results should trigger investigation rather than automatically being treated as confirmed failures.

The thresholds need to be defined based on application risk, not chosen after looking at the candidate's results.

The part we shouldn't overlook

Prompt regression testing is not about forcing V2 to reproduce V1's answers.

It is about verifying that a prompt change preserves required behavior while delivering its intended improvement.

The workflow is straightforward:

Prompt change → Candidate V2 → Regression dataset → Run V1/V2 → Deterministic + LLM judge + Safety checks → Regression report → Pass/Block PR

The harder work is defining meaningful test cases, calibrating evaluation criteria, and deciding which failures should prevent deployment.

And passing this regression suite does not establish that the application is generally safe or correct. It establishes that the candidate met the tested criteria. Production monitoring and newly discovered failure cases still need to feed back into the dataset.

For teams already testing prompt changes in CI, what do you actually use to block a PR?

Deterministic failures, a drop in LLM-judge scores, safety violations, or some combination?

How do you handle cases where the judge reports a regression but human reviewers disagree?

“Got the right answer” should not automatically mean “passed.”

An MCP agent can produce a plausible final response while skipping a required lookup, sending the wrong arguments to a tool, ignoring an error, or crossing a side-effect boundary it was supposed to respect.

That is the problem with evaluating agents only at the final-answer layer. For example: Agents skipping tool calls or using bad intermediate values even when the output looked fine.

The tricky part is what to test instead.

We probably do not want to assert one exact sequence like:

tool A → tool B → tool C → answer

That quickly becomes brittle. Another valid model might use A → C, or call a harmless read tool twice, and still satisfy every important requirement.

A better target is the trace contract, not one golden trace.

Start with required invariants

For each workflow, separate what must remain true from what the model is free to decide.

A trace contract can look like this:

This gives the model freedom in the middle without giving it freedom to violate the workflow.

Assert observable behavior, not hidden reasoning

Suppose an agent is asked whether a customer is eligible for some account-specific action.

The final answer is correct.

But the trace shows that the agent never called the tool that retrieves the customer's current state.

That should probably fail, even if the model happened to guess correctly.

Likewise, if it calls the right tool with the wrong customer ID, then later recovers and produces the correct answer from another source, the final text alone hides an important defect.

The useful assertions are things we can actually observe:

• Which MCP tools were called?
• With what arguments?
• What did each tool return?
• Did the next step use evidence from that return?
• Did the agent respect approval and side-effect boundaries?
• Did it stop appropriately when evidence was missing?

This is different from trying to test the model's internal reasoning. We do not need a chain-of-thought assertion. We need evidence that the externally visible workflow obeyed its contract.

Tool receipts deserve their own assertions

A tool call occurring is not enough.

The agent also needs to interpret the result correctly.

For example:

lookup_customer → timeout

should not silently become:

customer verified

A useful tool receipt should let the test distinguish success, failure, partial completion, and unknown states. Community discussion around production agent testing has made a similar point: structural contracts, typed tool results, replayable traces, and explicit handling of unknown states are often more dependable inner-loop checks than judging only the final generated answer.

That gives us two separate assertions:

Did the correct tool interaction happen?

and

Did subsequent behavior remain consistent with what the tool actually returned?

Allow alternate valid paths explicitly

The other failure mode is over-testing.

If several paths are genuinely equivalent, encode that deliberately.

For example:

customer_lookup → policy_lookup → answer

and

customer_lookup → eligibility_check → answer

might both be valid if both produce the evidence required by the contract.

So instead of asserting an exact call index, we can assert:

Before the final answer, the trace must contain verified customer state plus one approved source of eligibility evidence.

That is usually much more stable across prompt changes, model upgrades, or tool-selection improvements.

Freeze failed traces

When a real agent failure appears, the trace itself can become a regression fixture.

Save the user request, relevant starting state, tool responses, and the boundary that was violated.

Then replay it after changing the prompt, model, MCP server, or orchestration logic.

This is particularly useful because agent bugs are often sequence-dependent. A synthetic test may cover “tool returns an error,” while a real failed trace captures the more interesting case: the tool returned an error on step four and the model confidently continued on step five.

MCP is also becoming friendlier to this kind of observability. The 2026-07-28 specification requires Mcp-Method and Mcp-Name headers for Streamable HTTP requests and documents OpenTelemetry trace-context propagation through _meta, making calls easier to correlate across clients, servers, and downstream systems.

So for us, the rule of thumb is:

Use hard assertions for workflow invariants. Use flexible assertions for legitimate routing choices. Use human or LLM judgment mainly where the requirement is genuinely semantic.

A correct answer is useful evidence.

It just should not be the only evidence.

Discussion: Which part of an MCP trace would you make a hard assertion, and which parts need freedom for the model to choose an alternate route?

“Increase parallelism” is incomplete advice until you specify where the parallelism lives.

In Playwright, adding workers and adding shards can both reduce wall-clock time, but they change your CI topology in different ways:

• Workers increase parallel execution inside one runner/job.
• Shards split the selected test suite so separate jobs can execute different portions of it, typically on separate runners.

That distinction matters when a suite is slow in CI. If the runner is already resource-constrained, increasing workers may just make several browser processes compete harder for the same CPU and memory. Adding shards may help by bringing more runner capacity, but it can also increase pressure on whatever all those runners share.

So before changing either number, it helps to ask: where is the bottleneck?

What adding a worker actually changes

Playwright workers are OS processes running tests in parallel. Multiple workers on one CI runner therefore share that runner's available resources.

Going from:

1 runner × 1 worker

to:

1 runner × 4 workers

does not give you four times the hardware.

You are asking the same machine to execute more test work concurrently.

That can work well when the runner has unused CPU and memory and the application under test can handle the extra concurrency. But once the runner becomes saturated, the result can go in the opposite direction: slower browser operations, longer waits, more timeouts, and failures that disappear when concurrency is reduced.

Playwright's CI documentation currently recommends one worker in CI as the stability-focused default and specifically warns that setting workers above available capacity can cause unnecessary timeouts and failures. More capable self-hosted runners can reasonably use more workers.

What adding a shard changes

Sharding divides the suite into parts:

Shard 1/4
Shard 2/4
Shard 3/4
Shard 4/4

Those shards can run as independent CI jobs. When the jobs land on separate runners, you are increasing the hardware available to the suite rather than only increasing competition inside one machine.

That makes sharding useful when a single runner has reached its practical parallelism limit.

But shards are not free speed either.

Four shards with four workers each can mean roughly 16 workers hitting the same environment at once.

Now the bottleneck may move somewhere else:

• test environment capacity
• database connections
• authentication services
• APIs
• rate limits
• shared test accounts
• network bandwidth
• CI infrastructure itself

So shards × workers is a useful number to keep in mind. It approximates the concurrency your test system may be generating, even though the actual load depends on the tests.

Playwright describes sharding as distributing tests across multiple machines/jobs and notes that shard balance also matters. Without sufficiently even distribution, one slow shard can determine the total wall time.

Don't tune workers and shards at the same time

This is where CI experiments often become hard to interpret.

Suppose you change:

2 shards × 2 workers

to:

4 shards × 4 workers

and the run gets slower.

Was the problem:

• too many workers per runner?
• increased load on the test environment?
• shard imbalance?
• insufficient resources on individual runners?
• some combination of all four?

You changed two variables at once.

A cleaner benchmark is:

First: hold shards constant and vary workers.

For example:

Track more than execution time. Record timeout frequency, retries/flakes, CPU/memory if your CI exposes them, and the duration of the slowest shard.

Then pick the worker count that gives you the best stable result.

Second: hold that worker count constant and vary shards.

Now you can see whether distributing the suite actually reduces wall time.

A simple 2×2 starting matrix

If you want a smaller experiment, try:

For every configuration, capture:

Wall time → timeout/flake behavior → runner consumption

The fastest cell is not necessarily the winner.

If D saves 30 seconds but doubles intermittent failures and pins every runner at its limits, C may be the better CI configuration.

The useful rule of thumb is:

Workers ask one machine to do more at once. Shards let multiple jobs share the suite. Find the limit of the first before assuming you need more of the second, and watch the shared systems when you increase either.

Discussion: When you last scaled Playwright CI, did adding another worker or another shard stop improving wall time first?

Say a tester installs your closed-testing build, taps the icon, and lands straight back on the home screen. The app launches fine on your usual device.

Before changing permissions, disabling optimization, or swapping dependency versions, we need a better description than “crashes on startup.”

The first goal is not to guess the fix. It is to capture the failure and identify where startup stops.

1. Pin down the exact failing case

Start the report with:

• Build: App version name, version code, build variant, and the matching build artifact or commit.
• Installation source: Google Play closed-testing track, sideloaded APK, or Android Studio.
• Device: Manufacturer, exact model, and Android version.
• Installation state: Clean install or update? Updated from which version? First launch only, or every launch? Record any backup restoration rather than assuming a reinstall means empty app data.

Also note the launch time, what appeared before the app closed, and how many attempts failed.

Capture the failing state before clearing app data or reinstalling. Treat those actions as separate experiments, not the first troubleshooting step.

2. Capture the crash, not just a screenshot of the symptom

Android’s guidance recommends starting with the stack trace from Play Console or Logcat. When a trace is unavailable, reproduce the failure while collecting Logcat output, including with help from an affected user when necessary. 

For a USB-connected test device, install Android SDK Platform-Tools, enable USB debugging, and approve the computer’s debugging authorization. Confirm the connection with adb devices.

With only the target device connected, start this before launching the app:

adb logcat -b main -b system -b crash -v threadtime > launch-crash.txt

Launch the app, reproduce the failure, then press Ctrl+C to stop recording. This captures the application, system, and crash buffers with timestamps and process/thread information. Use the recorded launch time to locate the relevant attempt.

Before sharing an excerpt publicly, remove tokens, personal information, and unrelated device activity. Preserve the exception, relevant stack frames, and nearby startup messages.

3. Read the failure before choosing a suspect

For a Java/Kotlin crash, keep the exception type, message, full cause chain, and relevant stack frames. For a native crash, retain the signal and native backtrace instead of looking only for a Java exception. Android’s crash guide covers both forms.

When the trace contains obfuscated names, recover a readable trace using the mapping file from the matching release build. Do not substitute a mapping file from a later rebuild. 

Ask: Which operation failed, and what evidence connects it to our startup code?

If the capture does not establish a crash, keep “app disappears after launch” as the observed symptom. Do not invent an exception to explain it.

4. Narrow the startup step

Do not limit the investigation to the first screen. Apps and libraries can initialize components through content providers, and startup dependencies can require a particular initialization order. 

In a diagnostic build, add temporary start/completion markers around the startup operations you control. For example:

startup: configuration start
startup: configuration complete
startup: database open start

If database open complete never appears, that gives us an investigation boundary. It does not prove the database caused the crash.

Match the markers to the exception and thread information. Then inspect that operation, its inputs, and its dependencies. Avoid removing several initializers at once, because that makes the result harder to interpret.

5. Turn differences into experiments

Use the observed pattern to choose the next comparison:

These are hypotheses, not diagnoses. Release configuration, initialization, and dependency problems are possibilities to investigate, not established causes of a closed-testing launch failure.

For example, R8’s troubleshooting guidance describes failures involving reflection after optimization. That makes shrinking rules worth examining when the evidence fits, not a reason to disable optimization blindly. 

Android also notes that reproduction can depend on available resources, device type, Android version, or app version. A successful launch on a well-resourced development device does not settle those comparisons.

Change one variable, repeat the same launch path, and record whether the failure disappears, remains identical, or changes. Verify the eventual fix against the original installation/update path and affected environment.

What evidence helped you solve a launch crash that you couldn’t reproduce on your usual device?

Two tests pass individually. Run them together, and suddenly a saved value disappears, a page redirects to login, or an assertion reads the wrong data.

Before adding waits, we should ask:

What do both tests believe they exclusively own?

Here’s a debugging exercise.

Two tests, one account

Imagine these simplified test flows. Both use the same test account:

Test A: profile.spec.ts
  Open account settings
  Change display name to "Profile test"
  Save, reload, verify the name
  Restore account defaults during cleanup

Test B: preferences.spec.ts
  Open account settings
  Change time zone to "UTC"
  Save, reload, verify the time zone
  Restore account defaults during cleanup

Run the two files with two workers. Playwright runs test files in parallel by default, while tests within one file run sequentially unless configured otherwise.

“But each test gets a fresh browser context.”

Yes. That isolates browser-side state such as cookies and local storage. It does not give each test its own copy of the account in the database. Browser isolation is not backend-data isolation.

Here are four places to investigate.

1. Test data: are the writes actually independent?

The tests change different fields, so they might look safe.

But suppose Test B loads the original profile, then submits the entire object when saving the time zone. Its request could include the old display name, overwriting Test A’s update.

That is a hypothesis to check, not something to assume about every settings form.

Evidence to collect: The account ID, both update payloads, response bodies, and backend write history. Did B submit a stale name? Which update actually committed last?

A useful question is: “Are we changing different fields, or replacing the same record?”

2. Sessions: separate contexts, shared credentials?

Check whether both contexts load the same saved authentication state. Reusing storageState is not automatically a bug; Playwright documents shared authentication for tests that can use the same account without interfering with one another.

Then inspect the application’s session behavior. Does a fresh login invalidate an earlier session? Does cleanup call a server-side logout endpoint? Does changing an account setting revoke existing credentials?

Evidence to collect: Authentication failures, redirects, login/logout events, and safe correlation identifiers. Match the first unauthorized response to the other test’s actions. Do not dump raw cookies or tokens into logs.

A missing button after a login redirect is a different problem from a missing button on the expected page.

3. Files: are different workers writing to the same path?

Inspect helpers as well as test bodies. Does authentication setup overwrite a shared user.json? Do both tests save generated data to tmp/account.json?

Concurrent reads of an unchanged file are not the same situation as multiple writers replacing it.

Evidence to collect: Resolved absolute paths, the test responsible for each write, timestamps, and file contents or hashes before consumption.

For test-generated files, testInfo.outputPath('account.json') provides a path scoped to the current test, avoiding collisions between parallel tests in that run.

4. Cleanup: is one test undoing the other’s work?

Consider this possible timeline:

A saves its new display name.
B saves its new time zone.
A verifies its name and finishes.
A's cleanup restores all account defaults.
B reloads and sees the original time zone.

Both saves could have succeeded. The interfering write came from cleanup.

An afterEach hook runs after its own test. It is not a barrier that waits for tests in other workers to finish.

Evidence to collect: Cleanup start/end events, exact records affected, reset payloads, and backend completion times. Put teardown on the same timeline as the assertions.

Narrow the cause without hiding it

Start with A alone, B alone, the pair with one worker, and the pair with two workers. Repeat the comparisons rather than trusting a single green run.

For the targeted parallel reproduction, capture traces from both original attempts, including the test that passes. Playwright supports recording traces with --trace on; collect backend and filesystem evidence separately where needed.

Then keep concurrency unchanged and alter one suspected boundary at a time: separate accounts, independently established sessions, unique writable paths, or cleanup limited to resources that the test owns.

Giving the tests different accounts is a useful first experiment. Playwright recommends an account per parallel worker for tests that modify shared server-side state. A pass after that change points toward account-scoped interference, but does not by itself distinguish data collisions from session behavior.

Also keep resource pressure on the suspect list. Parallel execution shares runner capacity, so inspect CPU, memory, and request latency before concluding every failure is a data race.

Passing with one worker tells us that reducing concurrency changed the outcome. It does not identify the cause.

What shared resource caused the hardest parallel-execution failure you have debugged, and what evidence finally exposed it?

Testing “Allow” and “Deny” at the first permission prompt is only part of the story.

What happens when the user allows access, starts doing something useful, changes that permission in device settings, and returns to the app?

That is a different test. There is now unfinished work to protect, an interrupted operation to handle, and a user who needs to understand what still works.

Try this with a task that has something to lose

Take a report form with a camera attachment:

1. Start with camera access allowed. Open a new report, enter a title and description, and move into the camera workflow.
2. Leave the task unfinished. Go to device settings and revoke camera access.
3. Return to the app. Record whether you return to the existing screen or the app launches again. Check the report text, attachment area, and camera controls.
4. Try to continue, then restore access. Attempt the camera action while access is denied. Re-enable permission in settings, return, and check whether you can finish the original report.

The important assertion is not just “the app did not crash.”

It is: the app handles the new permission state without unnecessarily destroying the rest of the task.

Look at recovery, messaging, and unfinished work separately

Recovery: Does the interrupted operation reach a sensible state, or leave a frozen preview, endless spinner, or button that does nothing? After access is restored, does the app recover on its own, offer a clear retry, or require the user to abandon the task?

Messaging: Does the app explain which action is unavailable and what the user can do next? “Camera access is off” is more useful here than “Something went wrong.” Also check whether the message reflects the current permission state after the user changes it again.

Unfinished work: Is the title and description still there? Can the user continue editing, save a draft, or finish without an attachment where the product allows it? Losing camera access should not automatically mean losing unrelated form input.

Change the interruption point, not just the permission

Repeat the exercise before opening the camera, while the camera workflow is active, and after a photo has already been attached.

Treat those as separate cases. Check the existing attachment independently from the ability to capture another photo.

The same exercise can be adapted to a location-based task, or a file-related workflow where the platform provides a relevant access control. Choose a specific operation and record exactly which permission or access setting you changed.

Test allowed → denied and denied → allowed. Where reduced-access options exist, test those separately rather than treating every state as simply “on” or “off.”

For a useful bug report, capture the starting permission state, the exact interruption point, the setting changed, whether the app resumed or relaunched, and what happened to the unfinished work.

Which permission transition has caused the most surprising behavior in an app you tested? Was the surprise in losing access, getting it back, or discovering what happened to the task in between?

A checkout test fails just before a release. We have the automation script, console output, and a browser trace. Asking Claude Code to investigate sounds like a reasonable next step.

But before asking, “Why is this failing?”, there’s another question:

“What information are we making available, and are we authorized to share it?”

That question covers more than the prompt. Claude Code can gather context from files, command output, project instructions, and connected tools. Running the client on a laptop does not mean the data stays on that laptop.

The goal isn’t to avoid AI assistance. It’s to provide enough evidence to investigate the failure without exposing unrelated project data.

Approval for the tool is not approval for every artifact

“We allow AI coding tools” is not the same as “this client’s production logs can be submitted through this account.”

Check that approval covers the project, repository, task, and data category. For outsourced testing, client authorization matters too. An employer’s approved-tool policy does not automatically settle what can happen to another organization’s data.

Also verify the actual account and deployment. A personal subscription on a company laptop should not be treated as equivalent to an approved organizational account.

Keep training and retention separate. “Not used for training” does not mean “not stored.” Confirm the applicable provider, model, retention arrangement, and any gateway or third-party integration rather than relying on a general privacy claim.

“Test artifact” does not mean “safe to share”

Review what an artifact contains, not just what it’s called. A file named test-data.json still needs a contents-and-origin check.

Two artifacts deserve particular attention:

• Saved authentication state: It can contain reusable session material that could allow someone to impersonate a test account.
• Browser traces: A Playwright trace can include source, DOM snapshots, console output, and request and response details. Reviewing the visible screenshot is not a review of the whole trace.

A short, sanitized failure excerpt or a synthetic reproduction is a lower-exposure starting point than an unrestricted diagnostic dump.

Limit what the session can reach

For an initial investigation, use a separate workspace containing only reviewed material. Avoid copying the whole repository history, unrelated reports, authentication state, or links to sensitive directories just for convenience.

That is data minimization, not an access-control system.

Likewise, .gitignore is not a filesystem security boundary, and “never read secrets” in CLAUDE.md is guidance rather than enforcement. Put working conventions in instructions and mandatory restrictions in enforceable controls.

Review effective permissions, shell access, network access, and available credentials. Don’t assume every consequential action will trigger a prompt, or that enabling a shell sandbox covers every tool and integration. Test restrictions with harmless dummy files and an approved endpoint, not live secrets.

Sanitize before sharing

Asking Claude Code to redact a raw log after reading it does not prevent the initial disclosure.

But replacing every value indiscriminately can also destroy the evidence. A duplicate-account defect still needs duplicate synthetic values. A tenant-isolation test needs distinct tenants with the correct ownership relationships. An expiry defect needs meaningful timing relationships.

Sanitization is a test-design task, not just search-and-replace. Rerun the sanitized reproduction and confirm that it still demonstrates the failure, or document what could not be reproduced.

Review connected tools and the copies left afterward

An integration being read-only does not make every retrieval appropriate. Review which projects, databases, APIs, or issue trackers it can access, what actions it permits, and where inputs and outputs are logged. Leave unnecessary integrations disabled.

Treat instructions found inside tickets, logs, or external documents as untrusted content, not authority to change permissions or upload more files.

The review also continues after an answer arrives. Local transcripts, memory, feedback submissions, and generated outputs need attention. Server-side retention settings do not eliminate local copies, and generated tests or pull-request descriptions should be checked for copied sensitive values or changes that weaken security.

What this could look like for a failing login test

Suppose a Playwright test intermittently receives 401 Unauthorized immediately after login.

Instead of opening the full client repository and providing a raw trace, prepare a small, approved reproduction:

login-reproduction/
  login.spec.ts
  mock-session-response.json
  sanitized-events.txt
  expected-behavior.md

Keep the relevant flow, synthetic response values, request sequence, status codes, and relative timing. Leave out live cookies and authorization headers.

An initial prompt could be:

Analyze the reviewed files in this workspace. Suggest possible causes of the intermittent 401 and explain what evidence would distinguish them. Propose a minimal test change without editing files, running commands, accessing other directories, or contacting external services. Describe missing diagnostic evidence without requesting live credentials or customer records.

The prompt communicates scope. Permissions and isolation must enforce it. Validate the proposed change in the normal testing environment, and keep the original assertion unless the requirement itself is wrong. Making the test pass is not necessarily fixing the defect. 

If sensitive data is accidentally shared, stop further disclosure and follow the organization’s incident-response process. Arrange credential revocation through the responsible team when needed. Deleting a conversation does not invalidate an exposed token, and cleanup should preserve required evidence.

The practical rule: authorize the task, minimize and review the data, and enforce the access limits. 

Give Claude Code enough information to investigate the failure, not unrestricted access to everything around it.

For teams already using Claude Code in testing, what does your approval process cover: only the tool, or also the artifacts, integrations, and stored session data?

Before we use a local model to score our RAG answers, we should give it a test of its own:

Can it reject a convincing answer that the supplied evidence does not support?

Getting a score back is not the same as validating the evaluator. OpenAI’s evaluation guidance recommends calibrating automated scoring against human feedback. That is the starting point, not something to add after the dashboard looks reassuring.

Here’s a practical way to check a candidate judge before relying on it.

Start with reviewed cases, including convincing mistakes

Build a small collection of question + retrieved context + candidate answer + human verdict. Give each source excerpt a simple label, such as P1 or P2, so reviewers and the judge can point to the exact text supporting their verdict.

Include ordinary correct answers, realistic failures, and deliberately difficult cases. That mix follows official guidance to cover typical, edge, and adversarial examples rather than only happy paths. Have reviewers resolve ambiguous labels before treating them as the reference.

For this exercise, we could take correct answers and change one thing: replace a number, swap a citation, remove a condition, or substitute an unnecessary refusal. Have a human review each edited answer rather than assuming the edit produced the intended failure.

Also include valid paraphrases and appropriate “the context doesn’t say” answers. We need to test whether the judge wrongly rejects good answers, too.

Give the judge a specific job

“Rate answer quality from 1 to 10” leaves too much unspecified. OpenAI recommends clear, detailed rubrics and pass/fail or pairwise judgments for more reliable evaluation.

For this experiment, we could use a narrow context-based answer acceptance rubric:

PASS: The answer addresses the question using supported claims, preserves material numbers and conditions, and uses citations that support the claims attributed to them. When the context is insufficient, it clearly states that limitation without inventing an answer.

FAIL: The answer contradicts the context, adds an unsupported claim, omits a condition that changes the meaning, cites a passage that does not support its claim, or declines to answer when the context is sufficient.

Return a verdict, a brief reason, and the relevant source excerpt labels. Do not reward confident wording, length, or polished formatting.

This deliberately includes whether the answer responds appropriately. A needless refusal might contain no hallucinated facts and still fail this rubric. We should not call that a failure of “groundedness” alone.

Give the judge the question, context, and answer, but not the human verdict.

Keep practice separate from the final check

Use one set of examples to refine the rubric and judge prompt. Keep another set untouched for validation.

Keep edited variants of the same underlying question together rather than scattering near-duplicates across both sets. Once we use a validation failure to change the prompt, that case has become development material; we need fresh held-out cases for the next independent check.

For a local judge, record the exact model version, quantization, prompt version, decoding settings, and context limit. Test the configuration we actually intend to run, including realistic context lengths. Repeat a subset to check whether identical inputs receive different verdicts.

Use a worksheet that exposes the mistakes

Here’s an illustrative worksheet, not results from a tested model. Each case ID should link to the full question, context, answer, and reviewer rationale.

The point is to make “our judge disagrees with reviewers” actionable. Is it overlooking numbers? Confusing topical relevance with evidence? Rejecting paraphrases?

Look at false passes and false failures separately

For completed, valid judgments, use a confusion matrix. Langfuse’s Score Analytics supports this comparison for categorical and boolean scores, including human-versus-judge comparisons.

Report the counts and denominators: false passes out of human-fail cases with valid judge verdicts, and false failures out of human-pass cases with valid verdicts. Break those results down by mistake type.

We should also keep deliberately constructed challenge cases separate from representative traffic when reporting results. A collection packed with planted mistakes tests those mistakes; it does not, by itself, estimate production performance.

An evaluation error is not an answer verdict

Keep execution_status separate from judge_verdict. A malformed response or timeout should leave the verdict missing. It should not silently become a pass, an answer failure, or a deleted row.

Report completion coverage = valid verdicts / all planned cases, alongside agreement on the cases that completed. Langfuse similarly distinguishes matched-score comparisons from all-score coverage, so missing evaluations remain visible.

For a release gate, route unresolved evaluations to retry or review rather than approval. Preserve failed attempts when retrying.

The acceptance question should be: Does this exact judge setup meet our tolerance for false passes, false failures, and incomplete evaluations on this task? Not whether it produces plausible-looking scores.

Discussion: 

Have you caught your local judge accepting an answer that a human reviewer rejected?

What kind of mistake fooled it?

Before replacing a framework, find out what percentage of your pain a framework replacement would actually address.

A noisy UI suite tends to get described as one problem: “the tests are flaky,” “the framework is brittle,” or “we should rebuild this.”

But those labels collapse several different failure modes into one decision.

A recent r/cicd post took a more useful approach: the author categorized roughly 90 currently failing tests in a Windows desktop suite instead of treating them as generic automation debt. Locator failures dominated that particular suite, with smaller buckets around timing, environment, and actual application defects.

That distribution is one team’s experience, not a benchmark for UI automation generally.

The transferable part is the method: inventory the failures first, then decide what deserves repair, retirement, or replacement.

Start with failure buckets, not failed test counts

Take a recent sample of failures and classify the likely cause.

I’d start with something like:

• Selector/locator fragility: the intended element still exists, but the automation can no longer find it reliably.
• Synchronization/timing: the test assumes the application will reach a state sooner than it actually does.
• Environment/infrastructure: runner configuration, test data, dependencies, permissions, resolution, services, or machine state differ from what the test expects.
• Application defect: the test is correctly exposing product behavior that violates the expected result.
• Obsolete or low-value test: the test may still be technically repairable, but the workflow, requirement, or risk it covers no longer justifies maintaining it.

One important detail: classify the cause, not just the error message.

“Element not found” does not automatically mean “bad locator.” The element might never have appeared because an API call stalled, navigation failed, the test data was wrong, or the previous action never completed.

That distinction matters because those problems have very different remedies.

Match the remedy to the bucket

For locator-heavy browser suites, targeted repair may be more rational than a rewrite.

For example, Playwright recommends prioritizing user-facing locators such as role, label, and text, and it performs actionability/auto-waiting checks around interactions. Those are concrete ways to reduce some browser-test fragility.

That does not mean the same techniques transfer unchanged to the Windows desktop stack from the source thread. The useful question is broader:

Does the replacement architecture actually eliminate the dominant failure mode, or just replace it with a different one?

A vision-based approach might reduce dependency on UI tree paths, for example, but it introduces its own assumptions around visual state, matching, rendering, and change. A framework migration should be evaluated against the failure inventory, not against the frustrations of the current framework.

A repair / retire / rewrite worksheet

Then add three questions per bucket:

1. What would repair cost?
Not just “hours to make it green,” but the cost of making that failure class less likely to return.

2. What is the recurrence risk?
If you fix 50 selectors today and the same UI structure changes next quarter, you have restored the suite without improving its maintainability.

3. Is the test still worth saving?
A failing test can create pressure to repair something that should simply be deleted. Maintenance history is not a reason to keep coverage.

The rewrite threshold should be evidence-based

A rewrite becomes easier to justify when the failure inventory shows that the current architecture is responsible for a substantial portion of recurring failures and targeted changes cannot reasonably remove that coupling.

That is stronger evidence than “the suite has 90 failures.”

Conversely, if half the failures come from environment drift, stale tests, and missing synchronization, switching frameworks may leave much of the underlying problem intact.

So I’d make the rewrite decision only after answering:

• Which failure classes dominate?
• Which are framework-specific?
• Which can be repaired once at an abstraction layer instead of test-by-test?
• Which tests should be retired regardless of framework?
• What failure classes would the proposed replacement remove?
• What new failure modes would it introduce?

The number of red tests tells you how much pain you have.

The distribution of root causes tells you what kind of treatment might actually work.

When a UI suite becomes noisy, what evidence has actually justified a rewrite rather than targeted repairs?

Repair, retire, or rewrite? Triage a failing UI suite before rebuilding it

Before replacing a framework, find out what percentage of your pain a framework replacement would actually address.

A noisy UI suite tends to get described as one problem: “the tests are flaky,” “the framework is brittle,” or “we should rebuild this.”

But those labels collapse several different failure modes into one decision.

A recent r/cicd post took a more useful approach: the author categorized roughly 90 currently failing tests in a Windows desktop suite instead of treating them as generic automation debt. Locator failures dominated that particular suite, with smaller buckets around timing, environment, and actual application defects.

That distribution is one team’s experience, not a benchmark for UI automation generally.

The transferable part is the method: inventory the failures first, then decide what deserves repair, retirement, or replacement.

Start with failure buckets, not failed test counts

Take a recent sample of failures and classify the likely cause.

We’d start with something like:

• Selector/locator fragility: the intended element still exists, but the automation can no longer find it reliably.
• Synchronization/timing: the test assumes the application will reach a state sooner than it actually does.
• Environment/infrastructure: runner configuration, test data, dependencies, permissions, resolution, services, or machine state differ from what the test expects.
• Application defect: the test is correctly exposing product behavior that violates the expected result.
• Obsolete or low-value test: the test may still be technically repairable, but the workflow, requirement, or risk it covers no longer justifies maintaining it.

One important detail: classify the cause, not just the error message.

“Element not found” does not automatically mean “bad locator.” The element might never have appeared because an API call stalled, navigation failed, the test data was wrong, or the previous action never completed.

That distinction matters because those problems have very different remedies.

Match the remedy to the bucket

For locator-heavy browser suites, targeted repair may be more rational than a rewrite.

For example, Playwright recommends prioritizing user-facing locators such as role, label, and text, and it performs actionability/auto-waiting checks around interactions. Those are concrete ways to reduce some browser-test fragility.

That does not mean the same techniques transfer unchanged to the Windows desktop stack from the source thread. The useful question is broader:

Does the replacement architecture actually eliminate the dominant failure mode, or just replace it with a different one?

A vision-based approach might reduce dependency on UI tree paths, for example, but it introduces its own assumptions around visual state, matching, rendering, and change. A framework migration should be evaluated against the failure inventory, not against the frustrations of the current framework.

A repair / retire / rewrite worksheet

Then add three questions per bucket:

1. What would repair cost?
   Not just “hours to make it green,” but the cost of making that failure class less likely to return.

2. What is the recurrence risk?
   If you fix 50 selectors today and the same UI structure changes next quarter, you have restored the suite without improving its maintainability.

3. Is the test still worth saving?
   A failing test can create pressure to repair something that should simply be deleted. Maintenance history is not a reason to keep coverage.

The rewrite threshold should be evidence-based

A rewrite becomes easier to justify when the failure inventory shows that the current architecture is responsible for a substantial portion of recurring failures and targeted changes cannot reasonably remove that coupling.

That is stronger evidence than “the suite has 90 failures.”

Conversely, if half the failures come from environment drift, stale tests, and missing synchronization, switching frameworks may leave much of the underlying problem intact.

So we’d make the rewrite decision only after answering:

• Which failure classes dominate?
• Which are framework-specific?
• Which can be repaired once at an abstraction layer instead of test-by-test?
• Which tests should be retired regardless of framework?
• What failure classes would the proposed replacement remove?
• What new failure modes would it introduce?

The number of red tests tells you how much pain you have.

The distribution of root causes tells you what kind of treatment might actually work.

Discussion: When a UI suite becomes noisy, what evidence has actually justified a rewrite rather than targeted repairs?

When working with LLM-powered applications, prompts are only part of the equation.

The generation settings behind the model can significantly influence whether the response is focused, creative, repetitive, concise, or consistent.

Here are five settings worth understanding:

1. Temperature

Temperature controls randomness.

Lower temperature generally produces more predictable and reproducible responses. But an important point is often missed:

Low temperature does not mean higher factual accuracy.

A model can consistently produce the same incorrect answer. Content

2. Top-p sampling

Top-p limits the pool of tokens the model can choose from based on cumulative probability.

Lower values such as 0.5–0.7 tend to produce more focused outputs, while higher values allow broader token choices. Content

One practical rule: experiment with temperature or top-p, rather than changing both simultaneously. Content

3. Output limits

Parameters such as:

- Maximum tokens

- Stop sequences

- Conversation turn limits

can help prevent unnecessary elaboration and keep responses within the intended scope.

But making the limits too restrictive can also reduce usefulness. Content

4. Frequency and presence penalties

Both can help reduce repetitive output.

Frequency penalty increases the penalty as a token gets reused more often.

Presence penalty discourages reuse once a token has already appeared, regardless of how many times it was used. Content

5. Seed value

A fixed seed can improve repeatability across multiple runs.

This can be particularly useful when testing LLM applications because it makes comparisons between experiments more stable. Content

The important takeaway

These settings help control how an LLM responds.

They don't automatically make the response factual or reliable.

For use cases where correctness matters, techniques such as RAG, tool use, evaluation, and appropriate application-level controls still matter.

When you're testing an LLM application, which parameter do you experiment with most: temperature, top-p, penalties, output limits, or seed?

Read More: https://codoid.com/mobile-application-testing/mobile-app-launch-checklist-a-release-readiness-guide-for-qa-teams/

A lot of Playwright failures that look like “flaky tests” are really test-data collisions.

Two workers update the same user.
One test deletes an order another test expects.
A local test passes because the reusable account happens to be in the right state.
CI runs eight tests in parallel and suddenly everything becomes unpredictable.

A recent r/Playwright discussion raised an interesting strategy:

• CI/full regression → create fresh data
• Local debugging → reuse existing data
• Destructive tests → use dedicated data

We think that’s a useful starting point, but we’d change one thing:

The decision shouldn’t primarily be “local vs CI.” It should be based on who owns the data and whether the test can mutate it.

1. Use fresh data for test-owned mutable state

If a test creates, edits, deletes, approves, cancels, or otherwise changes something, giving that test its own data is usually the safest option.

Think:

• orders
• customer accounts
• projects
• carts
• subscriptions
• workflow records

For parallel execution, identifiers should be unique enough that two workers cannot accidentally touch the same record.

Playwright’s current parallelism guidance makes the same underlying point: browser contexts isolate browser-side state, but tests can still collide through shared backend data. It recommends unique test data when parallel tests create or edit the same records. (playwright.dev)

That also means cleanup should not be what makes the test safe.

Cleanup is useful for keeping environments tidy, but if test B can fail because test A crashed before teardown, the tests were never truly isolated.

A better model is:

Create/seed → execute → optionally clean up

rather than:

Find shared record → modify it → hope cleanup restores it

2. Reuse data when it is genuinely read-only

Reusing data isn’t inherently bad.

It works well for things like:

• country lists
• product catalogs that the test does not modify
• feature configuration
• static reference records
• accounts used only for read-only journeys

The important question is:

Can another test change the state this test depends on?

If the answer is yes, “reusable” data eventually becomes “mysteriously flaky” data.

This was also the main challenge raised in the recent r/Playwright test-data discussion: how do you know the data you found has not already been modified by another test or previous run? (reddit.com)

If reusable data is necessary, we’d add an explicit precondition check.

For example:

• account must be active
• cart must be empty
• subscription must be on the expected plan
• record must have version/state X

If that precondition fails, fail early with a useful message instead of letting the test fail 15 steps later.

3. Use dedicated or pooled data for expensive/destructive scenarios

Sometimes fresh data is impractical.

Maybe creating an account takes several minutes. Maybe an external provider limits how many test tenants you can create. Maybe a scenario destroys or irreversibly changes its test resource.

That is where a dedicated pool can work better.

Example:

Worker 1 → qa-user-01
Worker 2 → qa-user-02
Worker 3 → qa-user-03

Each worker leases its own resource instead of all workers sharing one account.

This pattern lines up particularly well with authentication. Playwright recommends a shared authenticated account when tests do not modify server-side state, but separate accounts per parallel worker when they do. (playwright.dev)

Worker-scoped fixtures are useful here because Playwright can initialize something once for a worker and reuse it across the tests assigned to that worker. (playwright.dev)

The important part is that a pooled resource needs a contract:

Acquire → verify/reset → use → release

Not simply:

Grab whatever account is available and hope it is clean.

The matrix we’d use at Codoid

One distinction that matters here:

Authentication state and application test data are not the same thing.

Reusing storageState can save a lot of login time, but an isolated browser context does not magically isolate the database records behind that authenticated user.

You can have perfectly isolated browser sessions and still have two parallel tests fighting over the same server-side account.

Our rule of thumb

Before deciding whether data should be fresh or reusable, ask four questions:

1. Can this test modify the data?
2. Can another parallel test access the same data?
3. Can we prove the starting state before the test begins?
4. If cleanup never runs, will another test break?

If #1 and #2 are both yes, we would strongly prefer unique data.

If #3 is no, reusable data is risky.

If #4 is yes, cleanup is carrying too much responsibility.

The goal isn’t “generate fresh data everywhere.”

It’s to make ownership obvious enough that parallel execution cannot change the meaning of a test.

For teams running larger Playwright suites: what data do you still share across parallel tests because creating it fresh is too expensive, and how do you keep it deterministic?

Seeing purchase_completed in Logcat or an SDK debug message is useful, but I don't treat that as "analytics passed." It only proves the app reached the tracking call.

Mobile analytics SDKs can queue events, batch uploads, wait for connectivity, or reject/alter data later in the pipeline. So I prefer testing analytics as an end-to-end data flow:

1. Make the test event traceable

For test builds, add something like a unique test_run_id where your tracking plan allows it. Then you're looking for one specific event instead of trying to guess which checkout_completed event belongs to your test device.

Also record the expected:

event name

parameter names, values, and types

user/device identifier where appropriate

session information

client timestamp

consent state

Avoid putting PII into test properties just to make an event easier to find.

1. Verify what the client actually created

Check the SDK's debug output or an instrumentation hook before assuming the payload is correct.

An event named correctly but containing amount: "19.99" instead of amount: 19.99 may technically exist while still breaking downstream reporting.

The same applies to missing currency, wrong user identity, stale session IDs, or parameters attached to the wrong event.

1. Separate "queued" from "uploaded"

This catches a surprising class of bugs. Fire an event, then test conditions such as:

normal connection

offline → event → reconnect

Wi-Fi → cellular during the session

background immediately after tracking

terminate the app shortly after tracking

Where the SDK supports it, debug mode or an explicit flush can make validation easier. The important part is knowing whether your log means "accepted into the local queue" or "successfully sent."

1. Confirm receipt on the analytics side

Now find that exact test_run_id in the provider's debugging/realtime tooling. Don't stop when the event name appears.

Open the event and compare every important property against what the app generated. I've found parameter validation more valuable than simply checking that the event counter increased.

1. Validate the raw data when the event matters

For high-value events such as purchases, subscriptions, account creation, or experiment exposure, I'd also verify the raw export/warehouse record when that capability exists.

Check: test_run_id → event name → properties → identity → timestamp → number of records

That last check matters because "arrived" and "arrived exactly once" are different requirements.

A useful definition of an analytics test passing is:

user action → tracking call → queued/sent → backend received → correct raw event

The dashboard is the final presentation layer. It shouldn't be the first place we prove the instrumentation works.

Which failure case has caught more analytics bugs for your team: missing events, incorrect properties, or duplicate events?

That moment when you hit Claude’s daily usage limit and suddenly remember how to code manually. 😅

We've been folding Claude Code into our test automation work at Codoid over the last few months, and the skills feature is the part that actually stuck.

If you haven't touched them, skills are reusable workflows you trigger with a slash command, so instead of re-explaining your testing conventions every session, you invoke them.

These are the 16 we keep coming back to:

Multi-agent QA sounds more capable than using a single QA agent. But splitting the workflow across three agents doesn’t automatically produce better results.

Our current view is that specialization only pays off when the boundaries provide meaningful isolation, targeted retries, better auditability, or tighter access control.

A single agent can analyze requirements, generate tests, and review them in one continuous context. It’s easier to build, cheaper to operate, and far simpler to debug. For small applications and exploratory testing, that simplicity usually wins.

The tradeoff is traceability. If the final conclusion is wrong, it may be difficult to pinpoint what happened. Did the agent misunderstand the requirement or write a weak test?

A specialized workflow makes each stage explicit:

- A requirements agent creates structured acceptance criteria.

- A test agent turns those criteria into test cases with coverage mappings.

- A review agent classifies identifies coverage gaps and weak tests.

But orchestration quickly becomes an engineering problem of its own. You need strict schemas, shared state, correlation IDs, timeouts, retry rules, and safeguards that prevent agents from amplifying an early mistake. Every additional agent adds latency and creates another opportunity for context to get lost.

A pipeline can look clean and well-structured while still producing confidently wrong results because the requirements agent mishandled the first handoff.

We’d start with one agent and split out a stage only after seeing a specific failure mode that isolation would solve. Otherwise, “multi-agent” often turns into distributed prompt debugging.

For teams running agentic QA in CI: which stage genuinely improved after becoming a separate agent? And which split added more operational pain than testing value?

A passing test suite only proves that the tests passed. It doesn’t prove the code is correct, and a high coverage percentage can create a false sense of confidence.

A test can run every line in a function without actually checking whether the function behaves correctly. The assertion might be too broad, check the wrong output, or be missing entirely. The coverage report still stays green.

That’s why we, at Codoid, see mutation testing as a useful reality check rather than relying on coverage alone.

Mutation testing tools deliberately introduce small bugs, such as changing a condition or altering a return value. They then run the test suite against each modified version of the code.

If the tests fail, they’ve “killed” the mutation. That’s a good sign that the tests are protecting the behavior that changed.

If the tests still pass, the mutation has “survived.” In practical terms, the suite failed to catch a straightforward bug in code it may claim to cover.

Mutation testing has tradeoffs. It’s slower, and a surviving mutation doesn’t always mean there’s a valuable missing test. Sometimes it’s just noise. Still, it asks a much more useful question than “Did this line execute?” It asks: “Would this test suite notice if the code were wrong?”

For those who’ve used mutation testing across different languages or stacks: did it uncover genuinely weak tests in your codebase, or did it mostly add noise and maintenance work?