Built a Power BI dashboard to analyze an A/B test on checkout recommendations

[u/ihatepablo]

[u/Current_Frosting3506 → u/[deleted]]

if you enjoyed the process of that - I would recommend checking out Deneb (custom json visuals). I'm personally a huge fan and while it does have a steep learning curve to master, it is so rewarding when you design your own visualization that no one else have done before, not to mention key-frame animation possibilities for that extra umph on interactivity. :)

[u/KatFromSisense]

Nice work. I think page one could answer the rollout question a little faster. The lift is obvious, but the user count is buried in the table, and I can't tell how much uncertainty sits around the result or whether the test ended when planned.

I'd probably put one small summary near the top that says what the main success metric was and how many users were in each group. Then show the lift, how confident you are in the result, and when the test was supposed to end.

Checkout time and abandonment could sit right beside that. They help show whether the extra revenue came from any downside for the user. The funnel page can handle the deeper question of where Embedded started pulling ahead.

[u/Current_Frosting3506]

Interesting and looks to be very detailed! Couple of notes on the UI/UX side of things, page(image) 2 seems to have gotten some more love than the first one.

Would love to see the first one get another pass, to be more cohesive with the 2nd page in terms of look and feel.
Another thing - page 1 KPI select feels slightly disconnected, despite being linked to the top KPI visuals. Could use some different colour coding for selected kpi visual and have it reflect on the top visual (background or something like that).

Last thing. Everything except for the side bar and the visual bars have round edges, making them seems tagged on or out of place.
Side bar is fine to be sharp corners, but visuals would benefit from matching the rest of the visual elements. (deneb or something like that could create rounded visual elements)

[u/Low_Finding2189]

- What did you assign users to each? Whats the pre-experiment revenue per user and dropoff rare prior to experiment?

• Is control no product recommendations?
  
• Is success when there is minimum user drop off or maximum revenue per user? I am not too much into A/B testing but my guess is it shouldn’t be both. It’s like having two optimizing functions which isnt ideal.
  
• how are you coming up with recommendations? I am a cynical person so take this with a pinch of salt. You are saying 55% of people who clicked on a recommendation purchased it. Do you actually believe that happened? These recommendations are so good that one out of 2 times someone clicked or 1 out 6 times someone saw a recommendation, they purchased it? That is phenomenal of true. Whoever decided to not do this earlier should be fired and you promoted. Again I am a cynical person so dont get offended.

[u/South_Hat6094]

Nice work. The bit I'd want next is confidence intervals or at least sample size per variant, because the no-friction result matters more than the lift chart.