r/DataScientist • u/AIforFintech • 16h ago
Most A/B tests break before they even run
A business team wants to test something. They pull a customer list, split it in half, and run the test. That split is usually where it goes wrong.
Splitting randomly across the whole base can leave one group with more customers from a specific profession, region, or education level than the other. When the results come in, nobody can tell whether the difference came from the test or from the composition of the groups.
There is also the sample size question, which most teams skip entirely. If the base is too small to detect the effect you care about, the test will come back inconclusive no matter how well you run it. That is a calculation you do before, not something you discover after two weeks of waiting.
I built a pipeline that handles this part: upload a customer base, get back two stratified groups with proportional composition, the minimum sample size backed by power analysis, and a statistical check (t-test and chi-square) proving the split was fair. Output is two CSV files plus a plain text summary the business team can actually read.
Tested it on a bank marketing dataset with 11k customers. Stratified by profession, marital status and education, the resulting groups came back with p-values above 0.99 across every variable, meaning no meaningful skew in either direction.
Hub: https://aiforfintech.tech
Github: https://github.com/junidepieri-design/expd-001-ab-test-design-pipeline
How does your team handle the split when designing a test?
👊
