JDS Academy - A/B testing

 A/B testing is a powerful method used to compare two versions of a webpage, ad, or product feature to determine which one performs better. In the context of the JDS course containing online teaching andpractical parts about the basics of A/B testing, I could explore how to design experiments, split an audience into groups, and measure key performance metrics like click-through rates or conversions. The course covered critical concepts such as formulating hypotheses, ensuring statistical significance, and interpreting results to make data-driven decisions. Practical examples and hands-on exercises were provided to build confidence (above 95% 😁 ) in applying A/B testing to real-world scenarios. 

Here is a not-complete list of the wide range of topics:

• What is A/B testing exactly? What can you A/B test?
• The correlation vs. causation issue
• A/B testing: when/why/what/how/how long?
• The limitations of A/B testing
• The four steps of executing an A/B test
• 80%? 95%? 99%? What's the right confidence level?
• An intuitive way of interpreting the importance of statistical significance
• Calculating the required sample size before the test
• Segmentation, filtering
• Important A/B Testing mindset: Conversion Rate Optimization or Research?
• Implementation and typical mistakes
• Typical mistakes while running an A/B test
• Setting a hypothesis
• Avoiding website flickering in an A/B test
• The change-one-thing-at-a-time myth, multivariate tests, A/B/n tests and unusual audience splits
• Tool demo (Google Optimize)
• What to do after evaluating your experiment
• Statistical significance and the certainty of your results
• 1) Traffic allocation: sending visitors to the old/new landing pages (with JavaScript)
• 2) Data collection and visualization (Bash + SQL + Google Data Studio)
• 3) Evaluation: significance calculations (using Python)
• ACTION: Filter out the bad A/B test ideas from your backlog!
• ACTION: Review your A/B testing backlog!
• ACTION: The A/B tests you've run so far -- and the A/B tests you want to run in the future
• EXTRA: Hand-picked A/B Testing case studies
• EXTRA: online calculators (statistical significance and sample size)
• ACTION: Run your first research round!
• ACTION: Try out the simplest ever A/B test!
• ACTION: your own knowledge base!
• EXTRA: A/B Testing Hypothesis Form
• Summary & The right mindset to be successful with A/B testing

updated with photo: Dec.2024