Cumbersome AB testing setup with long runtimes and limited insights. Rebuilt the framework end to end.
40+
AB tests set up in 1 year
+7%
conversion rate uplift, key user groups
+15%
overall user retention improvement
Product team running web and mobile experiments across multiple platforms, struggling with experiment runtime and limited insight on full impact.
The previous AB testing setup was cumbersome with very long experiment runtime and development time, and limited insights on the full impact of changes. The goal was to improve test result quality and enhance data-driven decision-making across departments.
Five stages, run as a single roadmap so departments stop overlapping and tests stop running unfit-for-purpose.
Identified seasonality and peak periods, then built an ideal testing calendar around them.
Departments synchronise their testing plans, no overlapping metrics, enough time for scoping when several tests run in parallel.
Data analysts involved from the idea phase. Hypotheses are evidence-backed before they leave the room.
Analysts work with developers on event specifications, QA every implementation in sandbox, ship via Google Tag Manager.
Analysis automated in Python. Monetisation summaries and strategic recommendations go straight to C-level.
InteractiveTry it
A/B test simulator - size a test before you run it
Two-sided test, 95% confidence, 80% power. The framework sized every experiment like this before launch - so every test was properly powered before it ran.
Absolute lift
+0.50pp
Sample / variant
31,000
Outcomes after 1 year
Each metric, against the baseline it replaced
Send me the rough shape of it. I'll figure out scope on a 30-min call.
Book a call