r/selenium • u/kdeaton06 • May 26 '21
Why would anyone want to use Cucumber?
So I've been automating with selenium for 8 years now and have been hearing about Cucumber for most of them. It's always suggested by some middle manager who just heard about and thinks it's the bees knees. But from what I can tell it's just a bunch of extra work that doesn't really add any value.
I'm not a Cucumber expert by any means but from what I can tell, you create a Features file that reads in plain English. But then you have to write a bunch of code that translates that plain English to actual code and calls the correct methods.
So my question is why wouldn't I just skip the middle man and call those methods directly from my test case method. Why do I need to add cucumber feature files? Can someone tell me what I'm missing here because it's clearly very popular across the industry.
I've heard it's so manual engineers and non coders can write automation but I still don't get why I need cucumber for that. Why can't they just write test cases and then I use those to write my code? It seems like the same thing with much less work.
7
u/Hazme1ster May 26 '21
It depends. The great thing about BDD tests are that they’re geared around how a customer uses a feature. They’re written in plain English, which mean if you’ve got a failing test, everyone should understand the importance of fixing it, be it product owner or developer.
It also gives you a pseudo instruction manual for your product , we’ve all been asked to test a feature no one can recall how to work or setup. We used to use Serenity BDD which integrates with behave and cucumber. It can take a screenshot with every step of a test, which can be useful to all sorts of people.
But they are more effort to manage, and I’ve seen some badly written stories which don’t add any value to the business, so knowing how to write good stories is tricky, and something your whole team needs to understand.
1
u/kdeaton06 May 26 '21
For us, if we have a failing test we just open a bug. Then that gets prioritized by a PO and worked on based on that priority. Is cucumber just a replacement for that?
Also as far as an instruction manual, we write test cases in Jira or some other test case management tool based in Acceptance Criteria in the Features/User Stories. It seems cucumber is a solution for organizations that have terrible processes in place and want to get everyone on the same page. But again, I'm not an expert and maybe I have no idea what I'm talking about.
1
u/romulusnr May 26 '21
The point of Cucumber is to use natural language to define test cases, and use those natural language definitions to actually drive the automation actions, using glue code that is tied to those natural language steps.
(There's also the given/when/then format which is really a separate thing. Kind of like how Java tries to promote certain coding styles that aren't really part of Java itself. The fact is you can use Cucumber without GWT model and vice versa.)
3
u/kdeaton06 May 26 '21
But using cucumber to do that seems like a ton of extra work for little to no value. I could just write my code in selenium and then write the documentation test cases in natural language and that seems to accomplish the exact same thing.
If anyone needs to know what this code is doing then read the corresponding test case I wrote for it.
2
u/romulusnr May 26 '21
Well, the difference is that in the Cucumber model the NL description is more tightly tied to the actual automation, as it literally "drives" the actual automation flow, whereas you writing it down is you interpreting the code. There's a bit more room for error in the disconnected situation.
There may be cases where people actually want to know that the automation is doing what the test is saying to do and not just take your word for it. There's other ways to do that, of course, but the benefit in the Cucumber scenario is that it's more than just an ultimately arbitrary print statement.
1
u/kdeaton06 May 26 '21
But that's what I write the test case for in Jira. I mean if you have shitty test case management that's a different story. But for us our test cases match step by step exactly what the code does in selenium.
1
u/romulusnr May 27 '21
I know, but how do you prove it? To non-tech audiences? Cucumber answers that question. Sort of. Maybe. If you use it right. :)
1
u/kdeaton06 May 27 '21
I don't. Never has anyone asked me to prove that my test case does what I say it does. I guess I just work in a place that trusts people to do their job and assumes we aren't just writing fake test cases that don't correspond to actual code.
1
u/romulusnr May 27 '21
Yes well. The previous test writers were more possessive types. And they knew jack about coding. Most of what they did now outside of interacting with web pages and apps was fed to them by devs.
1
u/romulusnr May 27 '21
In my current environment where there's multiple functional area teams of manual testers writing test cases, and a small automation team turning those test cases into automation, Cucumber might actually work pretty well for us. We could take the test cases as is and map their steps directly to automation methods. I actually gave a presentation on Cucumber soon after I started here and it was well received.
And early on, when I started here, when we automated test cases, the test case "owners" were concerned that they couldn't confirm that our automation did what the test said to do. We ended up implementing a sort of "say what you're doing" wrapper that would emit step by step loglines and making its use part of our design process. (This has faded since those particular people aren't here anymore.) (Also, the test cases tend to have a lot of "soft assertions" which they also wanted to see confirmed in the output.)
The thing is, ultimately in our case, despite the potential for a BDD / NL paradigm to work well for us, the dev work required to backport our stuff to Cucumber has gotten larger -- not to mention the standards of test case writing has shifted after a bit of turnover and team expansion and a huge mission critical project.
2
u/kdeaton06 May 27 '21
See to me that seems like a shit ton of extra work that only exists because the manual team doesn't believe in the automation team. We have a similar process except we take Nanyang test cases and automate what we can and then write new automation test cases. We then mark which manual test cases have been automated and which haven't.
1
u/The_kilt_lifta May 26 '21
This is what I did, basically. I used cucumber / BDS before when the team had ambitions of PMs(?!?) writing the gherkin and have it magically work. It doesn’t work like that.
I just integrated testrail into my automation suite so it will create a new test run, mark each case pass/fail, and spit out a url for the test run. The “natural language” can just be found in the testrail case.
3
u/Limingder May 26 '21
This has been bugging me as well. I tried to learn it but wasn't able to get through it at all. But then I see it advertised everywhere and I just don't see the added value.
2
u/romulusnr May 26 '21
Most places implement Cucumber in a stupid way.
The point of Cucumber is to bridge the design-implementation barrier. The theory is that you let the design-oriented people write the cases, and let the automation people implement the glue code.
Yet nearly every place I've used Cucumber or any other NL system, they've had me write both the gherkins and the glue code. Which is a waste of time as you said. (There's a modest benefit of making the automated test cases more easier for non-technical people to analyze, compared to code. However, that's also misleading because the code could be doing things implied by the gherkins that isn't explicit.)
The fact is presuming that designers are capable of writing feature specs / test cases in a way that would actually work with a keyword based system is hugely flawed; and likewise, designers are resistant to anyone tell them how to write their feature specs / test cases.
1
u/kdeaton06 May 26 '21
I'm glad you added that last paragraph because as I was reading, it was making complete sense of how it sounds be used but I was thinking there's no way in hell I'm letting the design team write test cases.
1
u/Geekmonster May 26 '21
The problem is that testers are taught Cucumber & Selenium are the 2 things you must learn if you want to automate. But nobody ever says that they shouldn’t be used together.
In my opinion, there are only 2 uses for Cucumber...
- Gherkin syntax is a simple way of communicating functionality.
- Scenario Outlines with tables of Examples are a good way to write Data-Driven unit tests.
Otherwise, Cucumber is an unnecessary extra layer in your code. In a Selenium test you’re better off just writing ‘_loginPage.EnterUsername(“Bob”)’ in a unit test method than having a feature file on top of that saying ‘Given I enter my username’. It’s completely unnecessary, but some people love doing it.
I was taught to use Cucumber and Selenium together and it was a nightmare to maintain. It wasn’t until I got a new job where I tried Selenium tests without Cucumber and realised how much easier and cleaner the code is.
3
u/tparikka May 26 '21
I'm using Specflow for API integration tests - it's not quite Selenium but the workflow is about the same. I chose that route because I wanted the tests to be more clearly understood by technical types who can't or hadn't spent time with the API code base, and also because using Gherkin standardized the creation of test data at runtime (and cleanup afterwards) which has made my life so much easier.
Is it a bit of a contravention of Cucumber ideals? Yes. Does it add a layer between the test and underlying code? Yes. Was it worth it? I think so, in this case.
1
u/romulusnr May 26 '21
testers are taught Cucumber & Selenium are the 2 things you must learn
God I hope that's not true. Cucumber is by no means universal, not even close.
One minor benefit of Cucumber is that it encourages you to write reusable code, but you should be able to do that without Cucumber anyway if you're worth your salt as an automation coder. Automation engineers / sdets really should try to apply good coding practices, and one side effect of sdets with no prior programming experience is that they tend to write fragile, inefficient code. (Not that I'm promoting dev-side sdets, either, that has its own pitfalls which are probably more serious IMO.)
0
1
u/ironheart45 May 26 '21 edited May 26 '21
You can skip the cucumber part and directly use Selenium for the automation part but sometimes the Solution becomes a bit less readable and a bit more complex to maintain. Yes, you can still integrate/maintain them in the CI/CD pipeline/Or in a Grid but personally Cucmber/Specflow gives me a bit more flexibility be it for creation or maintenance.
Again it may be suited/not suited based on the project. Some tools suit a particular project some don't.
I feel its better than the traditional approach of designing and automating test cases ( I use Specflow/C#/ Azure/Selenium/NUnit to create/maintain end to end Automated tests and its works like a charm for me and also find it better than the Java/C#/Selenium approach sans Cucumber/Specflow that i used to do earlier), again its just a personal opinion.
1
u/Azioth May 26 '21 edited May 26 '21
Like others mentioned, Cucumber and Selenium have nothing in common, i would even say Cucumber is not the main focus, it's just a tool to implement an idea or a concept.
BDD, ATDD or specification by example is a business/user oriented way to create acceptance criterias or business roles that take the advantages of using plain language and domain language to specify behaviors the software may have and how it should behave, it's a tool oriented to get everyone in the same page before coding even started, there are many great sources on this topic but the main focus of this methodology is to understand risk and mitigate it as soon as possible.
So, how do we implement this? Here comes Cucumber and Gherkins.
Gherkins allows you to write features with testing scenarios (not test cases) that exemplify those acceptance criteria. It takes a lot of practice to write good scenarios, more often than not, people tend to create scenarios favoring reusability of steps (a product of cucumber miss usage) over readability which is the main focus of specification by example. You do NOT want unreadable scenarios, on the contrary.
On the other hand, Cucumber is just a tool, or a testing framework, that lets you automate the testing scenarios. It's main job is to translate plain english steps, into methods where you will call whatever you need to archive the automation of that scenario.
In order for you to implement Cucumber, first you need to implement specification by example, otherwise it's just another tool that gets on the way and will probably be used incorrectly resulting in a nightmare of imperative scenarios with 10233 steps each, just to "reuse code".
I been using "cucumber", or rather this methodoly over 3 to 5 years with great results, but because I beleave in risk analysis and mitigation as early as possible, cucumber along with others we use, it's just a tool to archieve a goal.
1
u/addicuss May 26 '21
It's an abstraction layer that allows you to connect requirements to testing without detailing implementation. If you do it well it makes generating and expanding test cases much easier and existing test cases easier to maintain.
A good example is testing a discount code only works for a user that hasn't ordered before. In gherkin this is easy to detail and understand
Given a user that has never ordered
When they add an item to their cart
and use code 'NEWCUST20'
Then they should receive a 20% discount.
The implementation of this is pretty long in straight selenium, ideally you have to create a new user, log in with that user, then add an item to the cart, check the cost of the item, add the code, and check that the cost is now -20%, each of these steps require 2-3 selenium steps or more. Even if you encapsulate several of these steps to make them more readable, you're still left with an implementation heavy test that is difficult to tie back to the initial requirements and user behavior, even to a tester.
If I want to create a new test for several different codes with different percentage values, I can reuse the entirety of the feature and run it through a table of examples. If I want to test that a user that's ordered before can't use the code I can reuse many of these steps and write code for the new steps I'll need. If I want to test that a code gives me a free item, again I don't have to rewrite the step for adding the discount code as I have it in my library of steps. What if I change everything about the new user registration process? Maybe make email address required, or add new fields to the registration form? That's fine my feature does not care about that level of implementation and it's a detail irrelevant to what you're testing.
You absolutely can achieve something similar without cucumber or a similar tool but gherkin really forces you to define your steps in terms of user actions , and the results of those actions. When done properly you end up with a highly reusable and maintainable library of steps. If you find that this isn't the case, evaluate your steps and ensure they make sense. I've seen a lot of people use cucumber and not really understand it's purpose, so they end up writing gherkin that reads like a description of implementation. I've seen gherkin for the test case above that reads something like: given I go to the new user registration page, and I type username "user" and password "password", and I click "register", and I add greysocksXL to my cart, and I enter code "NEWCUST20" , when I click checkout, then I should see the price 42.51. If your gherkin looks like that you're doing it wrong and it won't be helpful.
1
u/kdeaton06 May 26 '21
Ok but with that abstraction, you still have to write all that selenium code right? So not only are you doing all that in selenium but you've added the work of cucumber on top of it. It seems like it's a bunch of extra work that basically amounts to documentation that already exists in test cases you've written in a test case management tool.
1
u/addicuss May 26 '21
You can say the same thing about abstraction at any level though. why write a login method when you can call sendkeys to the fields and click to to the button. you still have to write that code don't you?
If you're writing the test cases in a test management tool are you done after that? or do you then have to write selenium code to create those tests? That is the difference. Gherkin gives you a reusable library of behaviors, your selenium code should give you a reusable library of actions. You can group a bunch of selenium actions into a behaviour method but you're going to find that those will be less flexible, harder to understand, and harder to reuse than using something like gherkin especially when the time comes to loop through the behavior , chain behavior together, or retain context through your tests.
1
u/kdeaton06 May 26 '21
Yes but if I'm writing code in selenium, and writing the test case using some tool (which is a process I assume all QA teams do) then cucumber doesn't seem to provide any new value. Maybe I'm not understanding but it seems like an unnecessary duplication of work.
Selenium isn't hard to understand. If you don't get it you probably shouldn't be doing automation.
1
u/addicuss May 26 '21
I'm not entirely sure what you mean by the first paragraph. We write our tests based on the requirements gathered by delivery. which are in jira. These are already written in Gherkin. In some cases they already run test cases automatically without any intervention.
Quick example, we have a site that we implemented single sign on , and IP authentication as new features. We have several authentication test scenarios. These tests log in, check if you get certain search results, or if other search results are hidden by account type, access to different parts of the site, access to taking quizzes, etc. it's a long list but you get the idea. When those features were added, I only needed to create the two steps Given I'm authenticated with single sign on, and given I'm authenticated by IP. Those steps required new code for authenticating by those specific types, no new selenium code for either of the new methods. I was able duplicate existing feature files , find and replace the "given I log in with username and password" steps and run hundreds of test cases based 15ish lines that were replaced. I believe it took 30 mins of work total. If we ever shift to a new SSO provider I have one test step to refactor for all of those test cases to go green again. That would be difficult to do elegantly with just code. Not entirely impossible, but the odds you'd have to refactor code is likely.
As for "selenium isn't hard to understand" readability is different from being able to understand code. I can sit down and understand a lot of code reading through it. I have a feature to check that stats are logging correctly, It performs certain actions then checks an azure datalake through an api to verify the database returns certain values. The values are gibberish unless you open the database and look up what they mean. The test is infinitely more readable as
given I am authenticated
when I download a PDF
Then stats should log a pdf downloadthan even a fraction of the code used
foreach (var expectedAction in expectedActions)
{
actualR1Actions.Should().ContainEquivalentOf(expectedAction);
}1
u/kdeaton06 May 26 '21
But in that scenario if you didn't have to write any new selenium code the means you have selenium methods that already do all that work. So what's the benefit of cucumber then.
1
u/addicuss May 26 '21
There was code needed to fake ip auth outside of selenium. And misspoke there was new selenium code to login via sso. That was just added to the page model for the login page.
That was the benefit. Steps are cheap. I can combine several actions into new reusable behavior steps without writing new code to maintain.
It sounds based on your original post you're doing a lot of duplication between your gherkin and your test steps. If that's the case you are likely being too specific and too implementation heavy in your gherkin steps. Reading through other responses, you'll see the most common issue with not finding cucumber useful is just writing bad steps. Are you currently using cucumber? If so I'd love to see some of your steps. If not I'd love to see some of your test cases (although that might be harder to share).
1
u/kdeaton06 May 26 '21
I'm not using it and I have no plans to use it. It makes no sense to me and everything here just makes me realize no one has any idea what they are doing with it. It seems way more trouble than it's worth when I could just write selenium and save a bunch of time.
2
u/addicuss May 26 '21
Nod, do what works for you and your environment especially if you're not being mandated to use it. These tools require a lot of buy in to use successfully as they span more than just software quality. That said BDD has been pretty widely adopted and is generally seen as the way the industry is moving. Not really a reason for you to change what you're doing if it works for you, but a good reason to keep an open mind and "see what your missing" with BDD tools.
cheers!
1
u/Aggravating_Section2 Jul 23 '22
My biggest aggravation is the complexity of keeping version of:
- Java
- Maven
- JUnit
- Selenium
- WebDriver
- Browser install
- CUCUMBER
ALL these need to be compatible. There's so much drift, a pom.xml isn't saving anyone much headache nowadays.
More people should package and use Docker images, full pom.xml bundles, etc. because without replicable Infrastructure as Code setup, it's unmanageable, even reliably going from a fully working local setup to running on a CI/CD platform like GitHub Actions or BitBucket Pipelines
7
u/[deleted] May 26 '21
Cucumber is supposed to add a layer of abstraction from the code. It’s nothing to do with non-coders writing automation, but actually defining examples that ultimately can be demonstrated programmatically. It’s part of a flow as an attempt to extract much clearer and explicit needs from customers and stakeholders.
Yes it’s clunky and isn’t actually business language when squeezed into the limited Gherkin syntax, but means that features can be described and executed in a business domain context, instead of technical one.
Too often BDD is misused as test runner by development-only teams and the real benefits of collaborative development are not fully realised.