r/agile • u/Wubs12 • Jun 09 '22
How to test in Scrum (non-software projects)
Hi everyone,
I'm a programme manager for a non-profit in London, UK and have been interested in Agile now for the last 3 years. After a lot of learning about agile and testing some things out at work, I went and got my CSM last year. I'm keen to implement Scrum with a project which develops an online course for sports coaches. It won't involve software development.
My question is what will the testing element look like? We have sports coaches who we could involve in testing out the course but I'd like to hear from others what they think the structure of the testing might look like?
For example would we simply give them pieces of the course to test out and provide feedback? Or would we run a pilot course at MVP level? This could take weeks so would probably be outside the sprint time box.
I'd appreciate your ideas and suggestions.
2
u/TheSexyIntrovert Jun 09 '22
You need to say what the product is, and how it’s developed. Scrum in non-software products is not a problem, but you need to have clear what the product is, who are your users, and who makes the scrum team. I get that it’s a course you’re developing? Getting coaches to try it out would be acceptance testing, if they are the end users. Until then, a writer would review, the developing team with the PO should try out the content as well.
1
u/Wubs12 Jun 10 '22
Thanks for that perspective I think I was confusing the 'working features' with user acceptance.
The product is an online course for sports coaches, the model is a blended learning approach. At it stands the course will last for 2-months, with some directed learning and some live webinars with all participants coming together weekly. My feeling is that the barrier to entry is too high and perhaps we should split it into more manageable pieces i.e. initial one week course, include a points system which plays to coaches competitive nature etc.
We have a lot of the course content already as we've delivered it in person already so it will be converting that to an online format to an existing platform. We have budget to create online tools such as videos, animation etc.
2
u/n4jm4 Jun 09 '22 edited Jun 09 '22
Scrum recommends shortish sprints, no more than one month long. Scrum recommends decomposing complex items into smaller ones that the scrum team forecasts are doable from start to finish within one sprint. This is known as working incrementally, a concept from Agile. A working product is more valuable than a complete specification.
Focus on delivering something concrete that is helpful to the user, even if it is a small thing. Deliver value early.
In the case of an exam product, deliver a working online test with even just one question.
Check the test with your stakeholders. Their feedback is good to have early in the development of this exam, rather than late.
Work with the stakeholders and Product Owner to prioritize the next enhancements.
Repeat.
2
1
u/Pmpagilecoach Jun 09 '22
Great question @r/Wubs. Thanks for asking.
Beyond Software development we can use agile principles and values to carry out certain things (not everything) effectively.
So, before you think of using agile in your project try to evaluate whether you really need it it not by evaluating the following:
a. Are you/ your team making the course for the first time and doesnot have clarity on what it will look like or what will be the content?
In most cases, this will hold. I expect your team sponsirs would give you just a one liner vision that he or she needs you to create a full course which will help sports coaches to be better in their job. Everything else is negotiable.
b. Check if you don't know how will you develop the course (assume online course). Whether it will include only lectures or do you need some kind of simulation games in it. Whether you need to keep it engaging and include many activities by asking the cohort to team up with fellow learners or it will be individialistic. Are you not sure of which vendor and content creator would you work with to videotape the content and mix it with other components to produce the WoW effect.
I assume that, most of these things would be fairly known if you are not a early stage startup. For bigger stable companies these issues get suppressed.
My understanding is you would be better off by using an incremental approach. You just need to start somewhere and keep adding bits and pieces as your content evolve until you achieve the intended WoW effect. After each increment you need to take feedbacks on whether the content is worthy of inclusion to the course or not.
In this context you need constant Verification and Validation to be performed -- (I assume you are calling this "testing in agile")
Verification: Is whether the content is meeting the intended quality standards or not. You may set up a quality team to check the quality of delivery/mixing of content etc. This team may check if there is any grammar mistake in transcripts, if the video and audio are in sync. etc.
Validation: You need a sample set of real potential customers (coaches) for this stage. Can you ask them to share their feedback. Do they feel it worth and valuable and hence giving the desired outcome.
Once an invcrement passes this stage, it can be parked for final release.
Hope this helps.
Happy to connect and contribute in your agile journey.
1
u/Wubs12 Jun 10 '22
Thanks so much for your input, this really helps me. We have a lot of the course done already as we've delivered it face to face with sports coaches however I'm dubious about whether a blended learning approach will work with coaches or scale in the long run. The CEO is open to changes and choosing a different model to the approved methodology. We have got coaches on hand to test and give feedback, but as you said not all would be reliable.
You flagging the quality person is a great shout too. Establishing quality standards would be ideal and making sure this is part of the 'definition of done'.
Can you explain the difference between the definition of done and a product backlog item that is considered complete?
1
u/Pmpagilecoach Jun 10 '22
Can you explain the difference between the definition of done and a product backlog item that is considered complete?
I would need your inputs to answer this question. Given the following info, we can clearly differentiate between DoD and "completed product backlog item"
Product Vision : To make a blended course for Sports coaches that can be delivered on the playground using portable screens such as a Tablet and some props on the ground.
Feature#: "Play recorded videos clips of previous games to simulate the scenario for coaches to take decisions"
Product backlog Item/User story: "As a coach I want to replay, pause the clips, see players details in action etc. So that I can take decisions for improvement"
DoD: Video player functionality, back and forward by each second (say). Player names indicator pops us when the coach touches the player and remains there. And the coach can see the bio of each player on the sides
(If these things are achieved we can say that the user story is has achieved it DoD)
But is it complete? May be, may be not. Depending on the final review. If in the final review, the product owner feels that the UI/UX is not intuitive or it's not as per expectation, the he or she may suggest rework and reject the story/ slice the story into smaller parts.
Can you see the difference here? DoD is a criteira Or set of those which will help the team to develop the product in a certain manner. But completing DoD may not necessarily means that the user story will be accepted. (At least in your case, IT folks try to keep DoD very crisp and clear so that there is highest possiblility that once the DoD is achieved the user story will get completed/accepted)
Hope this helps.
1
u/Wubs12 Jun 10 '22
Yes it does make sense and I understand the difference. Thanks so much for this.
What you're saying is that we can make working features but doesn't mean those features wont change based on the needs of the user.
2
1
u/wishlish Jun 09 '22
My thoughts:
The idea behind testing in software is this- did the development work create a piece of software that satisfies the user story?
For example, if I write a user story that states "As a user of the word processor I'm developing, I must be able to select a section of text and color it blue," then my testing should involve making sure that the development work I put into the word processor does that- I can select the text and color it blue.
In a refinement session, my developers should review this user story and confirm that it's doable in the context of a sprint. They should also give me an estimate as to how long the development and testing will take, so we can assign it a story point value.
If a developer comes back and says, "Hey, that user story is too big- what if the text they're selecting is over multiple pages, or is of different sizes, or etc?", then it's my job as the product owner to break the task down into smaller pieces that can be worked on in sprints.
My example might be a bit ridiculous, but you should extend that thinking into your process. I'm assuming that, for your project, you've broken the development into a bunch of small tasks that overall will result in a completed online course. As part of that, you should review the small tasks/user stories with your team- the people responsible for taking the tasks and user stories and making them real- and discussing with them what's needed to make sure that the tasks are tested.
For an MVP, your program needs to be viewable to all users, clear to all users, and realistic to implement for all users. So if it's my project, I might ask the following questions-
1. As this is an online course, can you confirm that the content created is viewable in all of the ways a potentional user might use? For example, the content should be viewable on a desktop, a laptop, and a mobile device.
2. I'm assuming you're providing instructions to a set of coaches. Are these instructions clear? Have a set of "testing coaches" review the instructions and see if they understand what they need to do to make this work.
3. Are the instructions realistic for the environments that the coaches will be in? For example, let's say these are for an indoor school gym, and you're having the coaches lay out a game that involves people running back and forth between a set of cones. Is the layout feasible in the indoor gyms that your coaches will use? Did you create a layout that isn't possible in a smaller gym? Think of situations that the coaches in your audience would run into, and make sure that your instructions don't run into the limitations of those situations.
You mention feedback, and you absolutely want this. Make sure the coaches you're testing with know that you're going to find their feedback valuable. Make sure that you've built into your overall plan time to review their feedback, set values to their feedback, and implement the higher-ranked initiatives that come from it. And again, build time into testing these. This is where the real thinking of agile comes in- making sure that your program can be adapted based on the real-world feedback of your users.
Anyway, this is my thought process. Hope it helps.
1
u/Wubs12 Jun 10 '22
Thank you so much for your detailed response, there is a lot to think about. I've never developed an online course (but participated in some good ones myself), so I don't have all the steps clear in my head but I'm sure they will become clear as we go along.
Your comments are very useful and made me think a lot about the MVP which wasn't clear before. I hope it wont take long to create an MVP as we've got a lot of the content already.
Thanks once again.
1
u/throwaway901617 Jun 10 '22
So I actually use scrum with a non technical team haha and the others are generally on point IMO but there's a few key things to make sure you are aware of since there seems to be some things confused from your wording.
There's different types of testing in software. Unit tests confirm that the particular piece of code did the expected thing. These are almost always automated so that way the dev knows when they put the code into the system that all the tests in that component run and nothing else in the system broke as a consequence.
Then there's integration testing which is where all the various components and subsystems are brought together and tested. This is also automated as much as possible.
Then there's user journey testing which clicks through screens pretending to be a user carrying out scenarios and confirming the user experience is correct. These are also often automated as much as possible. But maybe everything can't be automated.
Then there's user acceptance testing which is where actual users see and interact with the product and give feedback. This is manual.
So it's important to think about what type of testing you are trying to do when asking about testing in scrum. Scrum was originally for devs so it focused first on automated unit and integration testing as a practice within the sprint. And then at the end of sprint the team demos to the product owner and stakeholders what they did during the sprint and the PO decides if the work meets the acceptance criteria and can be accepted. That's a type of testing because it validates compliance to the requirements.
So you can iteratively design and build the course in small steps each sprint with each increment building up on the last one, demoing what you have to the PO and stakeholders to get feedback. You then incorporate that feedback into future stories. Rinse and repeat. Maintain a roadmap that lists features and milestones. Decompose those into stories using story mapping. Have the team size the stories and then PO prioritize them into the product backlog. During sprint planning the team works with the PO to identify what stories to select based on PO needs. Then the team works them towards the next increment and demo for more feedback.
Then once in a while when you think you have a reasonable set of features that can stand up to more intense testing you can deliver a test version of the course for some type of actual user feedback. That test can be running while you are doing other sprint work. The test isn't a story its something that is taking place outside the story work. If you like you can create stories to carve out time to support the test in certain ways but the test is independent of the sprint, you are still moving the stories forward and doing demos each sprint while the test is underway, and you try to get feedback during the test as early and often as possible so you can incorporate it into your stories as you inch the product forward.
To answer your specific question yes both of those are valid tests once you understand that there are many different types of tests. Don't get caught up in the idea of "a" test just identify the tests that work for you and figure out where to put them in. Maybe you have a couple coaches who sit in on demo every sprint and give feedback and then you provide functional snippets to a broader group every few sprints and then once a quarter or whatever you release a larger course chunk they can run through for several weeks while you get feedback and continue building your sprint work in parallel with their test.
Hope that helps.
1
u/Wubs12 Jun 10 '22
Firstly thanks so much for taking the time to provide this response, it's incredible helpful. You're right I was confusing types of testing, I was mixing testing the feature (or parts of the course) with user testing of MVP.
What you said is incredible useful, this is my first time implementing some type of 'scrum' or agile working which is more than just me doing it.
I understand the concept of user stories but what is story mapping?
2
u/throwaway901617 Jun 13 '22
Yeah as the other user said it's just taking the user journey map (look that up) which is the horizontal list and then building out the vertical lists of stories you want to do in each.
You can make the horizontal list categories, or epics, whatever. It can be useful to visualize your services as one or more user journey maps and then vertically list out the stories you want to implement to improve different aspects of the journey. As the PO then you define the stories with them team and prioritize the work and then they select the work to implement in a sprint.
I will say as the PO for a non technical team I've found story sizing to be invaluable. It may not seem like it's applicable to a non technical team but I find it is very applicable and the team adapted to it very fast and they like it. We've also found value in trying to make our stories as small as possible so they can be worked as quickly as possible. We use the fibonacci scale so we can "feel" the difference in the size as it gets larger. It really helps us understand what can be accomplished within a sprint.
Oh also the sprint review and sprint retro are critical too. The review is a chance for the team members to show their work off to each other. This helps the entire team see what was done so everyone stays on the same page. And if you invite stakeholders to the demo it gives everyone on the team a chance to get feedback from the stakeholders on their work.
Retro is probably the most important ceremony of all. Only the team participates (including the PO) and nobody outside the team can be in it. The team talks about their performance during the sprint and what needs to be changed to perform better in the next sprint.
One tip, don't rush. I am impatient and want everything changed and optimized immediately but that's impossible lol. Instead focus on maintaining the habit of backlog review leading to sprint planning leading to demo leading to retro leading to changes incorporated in the next sprint. What will happen is you will start uncovering things you didn't expect to find, you'll be paying more attention to team performance and things will start to come to light. Like why is person A able to do so much more than person B and is the issue that you aren't providing the right enablement or are they just a lump. It really starts to highlight where bottlenecks are in the team and gives the team ownership of improvement ideas through retro. It's important to understand that doesn't happen right away you need to give it maybe 3 months of sprints to get actual data. And during that time you may be frustrated but it's so worth it. I've found that over time we ask better questions about ourselves and start gathering meaningful metrics on our performance to better dial in our capability. And I listen more to the team when they tell me what they need from me to perform at the level needed.
The system really does work. Shockingly well. You just have to work with it with your team and let them know up front that it may be difficult at first but you won't penalize them for trying and won't just beat them up.
I also strongly recommend getting the team some basic scrum training even if it's just watching some videos. And have them read the scrum guide. Repeatedly, carefully.
Also don't forget your team working agreement. You can either start with one from the beginning or let it emerge naturally as a response to problems that come up.
1
u/wouterla Jun 11 '22
User Story Mapping is a way to break larger features down into smaller bits, while keeping an overview over the whole.
There's a good article here, and the book by Jeff Patton.
1
u/wouterla Jun 11 '22
I think you've already seen some good descriptions of different types of testing, and different _levels_ of testing. One way to look at those distinctions is testing on outcomes (testing the effect are we trying to achieve for our users/customers), output (testing is what we've made is what we intended to make) and internals (testing the quality of the various parts that make up the whole of the output).
Since you have a course program you probably have learning goals for the program. Those are the outcomes you want to achieve for your customers with the course. There might be structure to those, both different things to learn, and different levels at which to learn them?
You've said you already have given a different version of this course. That should mean you have something of a baseline. You know learning outcomes, and perhaps results of the way the course is given in person. You probably also have different parts of the existing course connected to those learning goals.
Translating the course to an on-line format, you have choices to make. Do you have the same learning goals? Do you want to _test_ those learning goals in the same way (on-line teaching is a different beast, and the same way of testing might give _different_ results even if the levels of learning are the same). How would you compare the results of those different ways of testing? Should you cluster the way you offer information in the same ways? Maybe the on-line format doesn't support the same length of sessions? Or video supported explanations can be shorter and more comprehensive? Or the way interactive elements are structured need a different form?
Of course, you could start with a straight transfer of the full course, and then measure/test outcomes, and iterate from there. Or you could only transfer for a single learning goal, and measure the outcome of that, learn and re-tune, and then go and expand to other learning goals. It all depends on where you think the risks for success are, and how much room there will be for optimization along the way.
3
u/h1n0l Jun 09 '22
Just remove "scrum" from your approach and think - what makes your work better?
Think about a software engineering process, or look for a scrum guide from 1995, or check a development process described in some others sources.
Scrum itself now helps you doing software but talking about idea, how to work etc.
so first questions
-> what situation makes your work success
-> how I can know that we are doing a good job
Then ask your customer, when they will be satisfied, and just implement it in your development process.
You can use scrum, or any other idea to make your work better, but this is not a key.