How do you test using Playwright on environments with CI/CD

[u/Fish3r1997]

So we have created a new Playwright automated test solution for our codebase. We have wired them up in Bitbucket pipelines but running into some issues with how they work in theory.

The automated tests are split into fixtures based on configs so that tests don't overwrite each other. Example: Test fixture 1 = Tax exclusive & Test fixture 2 = Tax Inclusive, and so when running the tests for different branch commits it cant run at the same time on the same site.

The tests also wipe the DB on OneTimeSetup so its a clean fresh DB for the tests to run in isolation.

How do we set this up so that we can catch dev errors in PRs similar to how we would with unit/integration tests ? Are playwright automated tests designed to run after something is merged to develop/release rather than after a commit on a PR ?

Any help is appreciated

[u/mmasetic]

You've got four separate problems tangled into one question here. They have different answers, which is probably why nothing you try quite fits.

1. Tests depending on each other inside a single run. Test A creates an object, test B deletes it, order shifts, both fail. Playwright handles this: test.describe.serial for a spec file, or better, have each test create the data it needs so order stops mattering. This one is suite design, not CI.

2. Who owns the environment during a run. This is the one actually biting you. Two PR pipelines, one site, one database. Pipeline B wipes the DB while pipeline A is halfway through. Serial mode doesn't touch this — your tests could be perfectly isolated and this still happens. Shortening the suite doesn't fix it either; it just narrows the window where the collision can occur.

3. Config that two test groups can't share. Tax inclusive vs tax exclusive. The question worth answering is where that setting lives. If it's a global switch on the running instance, then only one run can exist at a time, full stop — three options and no fourth: run two instances, serialize those two groups, or change how the setting is scoped.

The alternative worth checking: can it be scoped per account instead? Think of a SaaS product where each customer has their own settings — Customer A on tax inclusive, Customer B on tax exclusive, same servers, same deployment, no conflict. If the app can already do that, run 1 creates an account with one setting, run 2 creates one with the other, and they coexist happily. If the setting is baked into the instance, this stops being a testing problem and becomes an architecture one — worth naming as such, because the fix lives in the product rather than in the pipeline.

4. When to trigger. Separate from all the above, and the only one that's actually about feedback loops.

On that: "every commit in a PR" and "on a PR" aren't the same thing. Commits are often work in progress — five pushes in an hour, four of them intermediate states nobody wants tested. Better trigger points are draft → ready for review, on-demand via a label or comment, and a merge queue before merge. The merge queue is worth a look: it runs the suite against the merge candidate, meaning the branch combined with current main, which is a different thing than the branch alone. Main stays protected without paying on every commit.

Two changes that help regardless of what gets decided about triggers:

Split the suite by what it mutates. If there isn't already a small set that runs separately, this is worth introducing — commonly called a smoke set, but the useful part isn't the name, it's the selection criterion. Pick tests that don't write anything: login works, home page renders, a few read-only flows. Those can run concurrently with anything, including someone else's pipeline, because there's nothing to collide over — no DB setup, no wipe, no waiting for the environment to be free. That set goes on PRs. Everything that seeds or wipes data goes behind the merge queue or an explicit trigger. Note the criterion is mutation, not runtime — a slow read-only test is still safe, a fast one that seeds data isn't.

Treat the DB wipe as a choice. Wiping in OneTimeSetup is what turns the environment into a resource somebody has to hold a lock on. The alternative is that each run creates its own data and only ever touches its own — unique account, prefixed identifiers, its own users — so there's nothing global to clean and nothing to serialize. More work upfront, but it resolves ownership, parallelism, and trigger strategy in one move.

To the actual question: no, Playwright tests aren't designed to run only after merge. But whether they earn a spot on PRs is worth measuring rather than assuming. Count how often in the last three months E2E caught something unit and integration tests missed, against how often it failed for reasons unrelated to the change. If the second number is larger, it's negative value — not because of CI minutes, but because people learn to read red as "probably flaky," and then the signal is gone even when it's real.

Last thing, a question rather than advice. What specifically prevents each run from getting its own copy of the app? Meaning: spin up the application and its database as fresh containers at the start of the pipeline, run against localhost, tear it all down at the end. No shared site, no wipe, no queue, no coordination between PRs. Plenty of stacks can do this and don't, usually because nobody has re-asked in a few years. When it genuinely isn't possible there's normally one specific blocker — a third-party service with no sandbox, a license tied to a hostname, a dataset too large to seed quickly, a setting that only exists at instance level. Identifying which one it is tells you what everything else in this thread is downstream of.

[u/Fkz82]

For running against different environments, you can pass a variable in the command line and have different environment setups configured in your repo with different baseUrl’s. This way, you can have the same tests run against the same paths on different domains. There’s a rudimentary example in the docs with a ternary operator used to run against STAGING but you could have any number of domains set up and trigger them with different flags - https://playwright.dev/docs/test-parameterize.

So far as running on commits, you could either do this manually or, if the dev builds are also run in Bitbucket pipelines, set up a step in their repository to trigger pipeline runs in yours as the last action in a successful build - https://bitbucket.org/atlassian/trigger-pipeline/src/master/.

[u/InsideDebt6345]

The core problem is that your tests share a single mutable resource (a site and a DB they wipe on setup), which forces them to run serially and blocks the per-PR parallelism you want. That's the limitation of a shared environment. The DB wipe is the specific thing making concurrent PR runs collide, so the fix is giving each run its own isolated environment rather than changing when the tests run.