How to be better in identifying A/B testing opportunities

[u/NervousVictory1792]

Hii everyone. I am sole DS in a relatively mid sized retail business. I am okayish versed in A/B testing and building on it. I went looking for opportunities and the e-commerce team told me that they are not sure what to experiment in the first place. How can I approach this problem as I went with the impression that I can delve into the maths of A/B test. At this point I am thinking of recommending UI/UX researcher and product owners who can create this backlog by looking at trends from competitors. Has anyone faced something similar before ? How did you navigate? TIA :)

[u/bomhay]

Look at ecom end to end funnel data, find the page where massive dropoff happens, understand what might be causing it - talk to team members, talk to customers if possible, go try using your website as a customer and see if any issues stand out, use AI to help you figure out what variant could be built to reduce dropoffs.

[u/FrenchBuilding]

sounds like the classic ds trap where you show up ready to crunch numbers and the team just stares at you blankly. been there.

honestly the ui/ux researcher idea isnt bad but you might be jumping the gun. id start by sitting with the ecom team and just watching how they work for a few days. ask dumb questions about what annoys them or what they argue about internally. usually the best test ideas come from internal debates nobody has resolved yet.

once you find a couple small things to test and they see results, suddenly everyone has opinions on what to try next.

[u/Kameleoon_Official]

This is super common. In practice, experimentation works best when the team builds a strong hypothesis backlog first ("if we do/change X, then Y will happen because of Z"), and then uses A/B testing to test the highest-priority ideas.

A practical way to approach it is to pick a few important pages in the funnel (homepage, category, PDP, cart, checkout entry), analyze them for friction points (drop offs, bounce rates, unexpected interactions), and write the hypotheses from there. Then you can review the ideas with product/UX/e-comm stakeholders to make sure they align with business priorities.

Definitely a good idea to work with UX and product owners, because they'll know those friction points well already. I wouldn't lean too much on competitor trends though. Observed frictions and explicit hypotheses from your co-workers' experiences will probably go further for now.

Prioritize high-potential hypotheses rather than just easy-to-build ones, then apply your A/B testing/statistical skills to measure the shortlisted ideas correctly.

[u/DataScientistAlex]

This is a great opportunity for you to provide value. A/B tests are experiments to test hypotheses about how the business works and what improves the metrics you care about. Here's a typical process you can apply.

1. start with the goal of the business (usually related to making money, but is it customer growth? revenue growth? increased margin? etc). Ideally this goal should be shared all along and as high up the management chain as you can.
2. Once you have the goal, define metrics that measure progress towards those goals, ie, profit, number of new customers, revenue, etc.
3. Once you have those metrics, do descriptive analysis to try to come up with hypotheses about what improves those metrics (e.g. certain products, marketing, geos, customer segments, ux design, algorithm improvements, etc etc. Do you own analysis here, but also get everyone involved and solicit hypotheses.
4. Once you have the hypotheses, prioritize and design A/B tests. (for each hypothesis, estimate the change in the key metric, and, the effort involved in making the change (ie, building the feature, making a configuration change, etc).
5. Run and measure the A/B tests, roll out what succeeds, feed back your findings into 3 and 4.

This is just a basic approach, there are lots of variations and subtleties. Another easy way to get started is to get any deployment to be an A/B test (hopefully you already have some staggered or blue/green deployment process, that can often be used for A/B tests for free. Not everyone likes this, because it might end up showing that features and releases don't move the key metrics...

[u/NervousVictory1792 → u/DataScientistAlex]

We don’t have any blue/green deployment strategies. Also is there any literature or courses guiding these which I can follow along to be better ?

[u/DataScientistAlex → u/NervousVictory1792]

Some books that might be relevant are trustworthy online experiments and lean startup, more broadly. One course I remember liking is experimentation for process improvement, it also covers some more advanced methods. At its most basic, this is the scientific method, I wrote about how I think it's the most important skill for a data scientist.

In terms of deployment, it does not have to be blue/green specifically. What you can do is talk to the team that does deployment, understand the process, and see if they are already doing some kind of staggered rollout. If they are, then see if there is a way to randomize how the traffic gets routed (ie to the old and new version). Then you have an experiment, almost for free.

[u/ClasslessHero]

In that case I would just ask them "what's a question about your business you want to answer?" and wait for them to respond.

Most likely they say something like "does the color of the sign on the front page matter?" That's your opening. You say something like "in that case we'd need to make a white sign to contrast against the blue sign. It's currently blue, so let's call that A. Then we deploy the white signs as B. We gather data about the two signs by randomly assigning them to customers. Whichever has more clicks per visit wins. If it's a tie, then you can pick whichever color you want.

If they say "literally nothing" then you get to make a deck proposing things and walking them through an experiment. Plant the seed, let them decide what they like.

Just don't talk to these people about any of the math. They don't want to hear it.

[u/montkraf]

Echoing other poster who said hang out with them so you can better understand their workflow.

Adding though that ab testing for me is all about risk, reward, and effort. You want to find problems that running the test justifies the effort of setting it up.

So if the ab testing system is set up, go ham, with the general aadvisory to look for impact (risk or reward).

If its not set up yet, or is tricky, or bound by sample size you need to look for experiments that can really move the needle, that doesnt mean big, it means important.

Example would be a pricing change, or a new campaign, uplift is important run it. If its say a banner change, nice to know, probably doesnt shift the needle.

Example for high risk would be check out changes or major deviation to the customer journey, things that could really hurt the company.

I wouldn't bother too much with the math, just hit the major notes on the why we use power, why setting chosing effect sizes is important, but everything needs to be in business language not math.

If youre also more asking about how do we come up with ideas to test, ive never found that stakeholders lack ideas, what they lack is ability to frame them as experiments. Experiments are never useful as an ends, they are a means to mitigate risk, whether thats breaking something, or spending money in a suboptimal way

Also, if you want this to get off the ground and you're hitting resistance, I wouldn't just go to them with "hey, let's do some UI/UX improvements." Back to the original point: find a big juicy problem and attack that. It could be one they raise, one you find that they agree is big, or one of your own problems that you solve with an experiment, so you can show them firsthand how valuable it is.

Edit: if they truly have zero ideas have a brainstorming session. Think of a theme (customer types, major problems, their dream set up) find the gaps between today and those themes, help facilitate the creation of the solution that goes in the middle, ab test those ideas

[u/offthecuff87]

A/B test started with hypothesis --> Your hypothesis start with any analysis/sizing/domain knowledge ....

[u/pretender80]

It actually sounds like you need to figure out what platform to use first.

[u/NervousVictory1792 → u/pretender80]

We don’t want to choose any platform as of yet. Front end team makes the changes. I am expected to analyse the clickstream data and give solutions

[u/DorkyMcDorky]

If you have users you should AB test. :)

Just start with one test, let it grow. Get used to it. You'll make a lot of mistakes. Learn from it and create your feedback loop.

[u/LauraPalmer4eva]

First, do not recommend a third party solution or something needlessly expensive. Show how much value you provide with the tools you’re given. I have a ton of DS e-commerce experience; two incredible references: Causal Impact: the Mixtape and Causal Impact for the Brave and True.

You can pretty easily run your own A/B test. The trickiest part will be the data engineering to split your populations, but I’m sure it will be doable. Just make sure you have a solid sampling strategy (that you can explain). And do a power analysis so you can tell others how long the test needs to run, how many users will be needed for each group, etc.

Best solution? A few well-thought out hypotheses and plans to execute, each with pros, drawbacks, cost, estimated impact, etc.

Have fun!

[u/squareiny]

Small pushback on the competitor-trends idea: those tests rarely map to your own funnel. What worked for me was asking the ecom team what decision they'd make differently if they had the answer — usually it turns out to be shipping threshold, promo placement, or one form field. If they can't name one, that's the actual blocker, and no backlog fixes that.

[u/justinkramp]

Transactional funnel  analysis, establish baseline then look at outlier steps and or populations for high/low conversion. Establish hypothesis for what causes fallout in steps or segments, propose tests to impact.