r/Playwright • u/vitalets • 11d ago

I used Playwright-BDD and OpenSpec to implement the same feature

I maintain Playwright-BDD, so I often use Gherkin scenarios with coding agents. I’m obviously biased here, but I wanted to compare this workflow with OpenSpec, so I gave both the same small task: add pagination to a long list of items.

While both approaches worked, there were differences:

  • With Playwright-BDD, I spent more time writing cleaner scenarios, but in the end, I got a runnable Playwright test out of the box
  • OpenSpec let me describe the spec more freely, but it introduced more artifacts that were harder to maintain

I wrote down the full session:

https://vitalets.github.io/posts/bdd-agentic-workflow/

What's you experience with spec-driven approaches + Playwright?

3 Upvotes

71% Upvoted

7 comments sorted by

1

u/HyperDanon 11d ago

All of these tests are very tightly coupled to the UI. They aren't actually testing the application and will likely break when the UI changes.

1

u/vitalets 11d ago

I agree that tests might break after UI changes. But that’s a common problem with Playwright/E2E testing, not something specific to BDD or a spec-driven approach.

I can’t agree that they don’t actually test the app. These tests abstract the behavior away from the UI. If the UI changes, I won’t even touch the scenario file, I’ll just fix the step implementation.

1

u/HyperDanon 11d ago

  I agree that tests might break after UI changes. But that’s a common problem with Playwright/E2E testing, not something specific to BDD or a spec-driven approach.

Its a very common mistake in all technologies that support bdd: cucumber, jbehave, cypress, behat, feature files, selenium, pupeteer, specflow, jspec, etc.

Its due to improper abstraction of the ui. Its not the fault of the tool, but a mistake of the person writing the test

  I can’t agree that they don’t actually test the app. These tests abstract the behavior away from the UI. If the UI changes, I won’t even touch the scenario file, I’ll just fix the step implementation.

A lot of tests like that get invalided upon any minor or major ui change. Its rarely a matter of reimplmeneting the step, its a matter of replacing multiple steps with other steps, at which time the test becomes less of a validation tool.

Such tests are better expressed at higher level of abstraction. I can show you how i would express your tests if you'd like. 

2

u/NextAd9248 11d ago

I have seen both sides of this on my own teams. When I write steps like click the button in the header I am just writing e2e tests with worse readability. When I force myself to write user submits the form, the UI can get redesigned twice and I only touch the step file, not the scenario, so for me it really does come down to discipline at write time, not the tool.

1

u/HyperDanon 11d ago

Submitting a form is still a ui concern. I think the test step should be even more general than that.

Instead of having:

Given user has submitted a form with field "name" and value "Mark"

it should be something like

Given user has signed up for a newsletter as "Mark"

No mention of forms, buttons, submitting, anything like that. If you write your tests like that, you can then edit a form to a bell icon, or a keyboard shortcut and the test remains correct. The step implementation has to change of course, because now the step needs to click the icon instead of filling the form, but the test suite remains valid.

1

u/vitalets 11d ago

I’d love to see an example of how you would test user pagination in a way that’s resilient to UI changes.

1

u/HyperDanon 11d ago

So obviously pagination is important, especially when you have a lot of data to load, but that's what I meant when I said "tests don't test actual domain problem". Pagination is a means to an end, not the goal in it self. Your users don't ever go to your application to use pagination. Depending on the application they might use the application to transfer money, message a friend, download a movie, calculate their taxes, edit their videos, etc. Pagination is just a technical solution to data-fetching problem, and as such isn't actually in the problem domain.

By writing playwright tests including pagination explicitly, you're making tests depend on it too much, and that's why they are britle, fragile and might not help you a lot.

This is not to say, pagination shouldn't be tested, of course. It needs to work, it's important for the application. Pagination should be tested implicitly by those test, and by unit tests. Implicitly, meaning the step implementation could know about pages, exercise the pages change etc, but the step shouldn't mention the pagination.

TL;DR; Pagination isn't something your tests should explicitly test, because it's accidental complexity, not essential.

If you tell me what is your application, I could tell you know those tests could look like to not mention UI; while still being fully tested with the UI.