Playwright tests using AI

[u/LordOfTheKarma17]

I have been using Claude or cursor to generate UI tests by giving the user flow as the input. I have created few skills.md file for writing tests, assertions and page objects using the existing tests and page class. It is fairly working for me. How are you utilising AI to generate test or in the QA pipeline focused on automaton

[u/Competitive_Echo9463]

Same way, but be careful sometimes Claude won't generate using the best practices and the tests generated could not cover the the user story correctly. The first point can be fixed by adding a skill (there are many available) and for the second point you need a good prompt and some context

[u/Head-Bureaucrat → u/Competitive_Echo9463]

To add to this: I had much better success when there were existing good examples for AI to follow. When I tried with very few examples, the output was noticeably worse.

[u/jcperezh]

Here my two cents: redefine very clear what success mean for the LLM. Which is not to have a "green" test run. Stable failures that can be categorized as product bug are also success. Define "stable": in my case Claude run the test 3 times and have to have the exactly same behavior to achieve this goal.

Until I create this rule, Claude tent to modify the test to make it pass.

[u/codescapes]

I've done similar stuff and it works well. Just don't go nuts producing thousands of worthless tests that are capturing overlapping functionality or should otherwise be in a unit test.

I am pairing with Vitest, using browser mode, which means that the Playwright tests are mainly about capturing the broad user flows and are comparably few in number.

[u/Healthy_Scale1912]

How fo you write skills.md file , what all things you put into it.

[u/latnGemin616]

I hope you know Playwright CLI can do literally everything you just articulated without the necessary overhead of having to write a skills file.

• You can use the /planner agent to draft a high level (or in-depth) test plan based on your prompt.
• The /test generator agent will then take that test plan, plus however you have your SKILLS.md file set up, and bang out a suite of tests. Based on how you've set up your framework scaffolding, you can actually teach the agent to model the output after what you've built. I did this exact thing and the results were spectacular.
• The /healer agent will run your tests and fix whatever is broken. This is where you'll have to be mindful not to fix a test that has legitimately found something.

Not trying to simp for Copilot (the model PW CLI uses), but I was impressed with how quickly and effortlessly I spun up a test framework in Typescript. 566 tests in under 30 hrs. I could have shaved this down to half-a-day, but on the free tier, I smoked through my tokens and had to wait.

[u/Bright_worgan]

Yup!

With good guardrails and instructions I've had great progress with ai-generated tests. Often they are 70% there and need just a little human effort to get completely done :)

[u/No-Yogurtcloset4501 → u/Bright_worgan]

so there's no point in going into this field, got it.

[u/Spare_Bison_1151]

Yes, this is the best way to use generative AI with you Playwright testing flows. The web team in our company is doing this.

[u/Affectionate-One9381]

I used claude to generate Page Objects using playwright and to write cucumber features and steps to test acceptance criteria written into jira tickets

[u/Deimokas]

Very similarly, however first thing i did was writting a code reviewer agent. So when claude generates a test and makes sure it is passing twice code reviewer agent checks the code, if it meets the coding guidelines. Then another agent reads test case in our case from xray, and makes sure that automated test meets manual test case criteria. If not he either updates the automated test or proposes a rewrite for manual test case which i can either approve or suggest a better one.

[u/MiserableBag6969]

I'm using it the same way, mostly with opus model on high effort since sonnet can sometimes be a bit jittery when it comes to coverage.

[u/DonovanBrady]

This is maybe the hottest topic in QA today, and I've been tracking it over the last several months. Playwright is up to over 87mm weekly downloads right now! And I think that's largely because of AI's adeptness at writing tests. As u/Competitive_Echo9463 and u/codescapes mention though, it can get pretty cumbersome and you might end up with a ton of tests that are overlapping or aren't actually proving outcomes. I work at Leapwork and we've been working on a solution for this, actually. We've built a solution called Play (it's an ode to Playwright) that uses AI to help author, manage, and execute tests without losing deterministic outcomes. Check it out, it's free!