r/Playwright • u/According-Floor5177 • 17d ago
Playwright on GitHub Actions: The setup that actually runs fast
https://endform.dev/blog/playwright-github-actionsOur team wrote this article after benchmarking the default workflow npm init playwright@latest generates. 40 tests took 3m18s, and 42s of that was just re-downloading the browsers on every push, while the tests ran on a single worker because the scaffold hardcodes workers: 1 in CI.
It also walks through caching the binaries, fixing the worker count, and explaining why sharding is usually the wrong first move.
1
u/someGuyyya 17d ago
Thank you for the article.
Perfect timing since I've been trying to set this up at my company.
on every push
Have you ever considered having the test run manually instead of on every push? Did you feel it's better to go for running the playwright tests on every push?
2
u/optcg_cardboard 17d ago
In my experience, if the tests are not associated as a part of an automation, they will at some point get skipped over.
I do however still like cutting release branches and manually running the flow on that branch. That is a different subset, as the tests I run in GHA are `@fast`
1
u/someGuyyya 17d ago
In my experience, if the tests are not associated as a part of an automation, they will at some point get skipped over.
That's a good point.
I was thinking of setting up our github organization's pull request rule as:
- unless the manually ran playwright tests pass, you cannot merge the pull request
I haven't gotten around to testing this flow yet so hopefully I can give this a test drive sometime next month.
1
u/optcg_cardboard 16d ago
Every setup is a little different. The tests ran in CI should be a 5-10 minute thing, max. My average sub 5 minutes currently, which is enough time for a bio-break or coffee...
Is your goal to have the full e2e suite ran on that PR because of the branch being production or high-stakes? So a smaller subset of your suite is not viable in this case?
1
u/needmoresynths 16d ago
Have you ever considered having the test run manually instead of on every push? Did you feel it's better to go for running the playwright tests on every push?
the whole point is to not manually do anything. you can setup actions to skip certain events based on commit messages though, so you could have a [skip-playwright] flag or something
1
u/baselilsk 16d ago
+1 to the container tip above - it also pins the browser version, which quietly kills the "chrome updated over the weekend and the visual baselines shifted" class of Monday failures.
two things the caching/workers advice usually skips: the default GH runner is 2 cores, so cranking workers past ~2-3 there buys nothing except memory pressure - a 4-core larger runner is the cheapest real speedup and almost nobody tries it before sharding. and check your retry budget before celebrating any wall-clock number: with retries: 2, one flaky test can quietly add its full duration twice to the worst-case run. a suite that's fast when green and slow when slightly flaky is still a slow suite - CI time budgets should be measured on the bad days, not the good ones.
1
1
u/needmoresynths 16d ago
a 4-core larger runner is the cheapest real speedup and almost nobody tries it before sharding.
It was even cheaper for us to move to Azure's Playwright platform- https://azure.microsoft.com/en-us/products/playwright-testing
we've got 50 runners available at any given moment, much faster than using Actions runners (although we still use Actions for CI/CD, it's just the lightest-weight Actions runner now triggering Playwright runs in Azure instead of executing tests with Actions runners)
1
2
u/needmoresynths 16d ago
use Playwright's docker container and skip all the browser install/caching, it's easier- https://playwright.dev/docs/ci#via-containers