# Optimization

This is a work in progress, and not quite ready for primetime

## What is A/B Testing

A/B testing is about taking an unlimited number of confounding variables and narrowing them down to two:

the change you made (A vs B)

"luck" (of the draw)

### Common issues using A/B test:

Not understanding everything your testing

Not fully understanding your entire audience

Not considering the impact of luck

Overestimating the scope of the impact

Not understanding the rules of the game

## Great Online Tools Links

Great, full featured calculators that are generally easy to use for data exploration.

Search Discovery: Sample Size Calculator

Booking.com: Power Calculator

CXL: AB Test Calculators

AB Testguide: Frequentist A/B-test Results Calculator

AB Testguide: Bayesian A/B-test Calculator

### Noteworthy Tools

Requires a little more understanding, but great for specific use cases

Thumbtack Abba A/B testing statistics: Great Confidence Interval graphs

Matt Mazur's AB Test Calculator: Great rate distribution graphs

Evan Miller's Chi-Squared Test: The granddaddy of website optimization calculators

### How A/B testing works

A/B testing (also known as split testing or bucket testing) is a method of comparing two versions of a webpage or app against each other to determine which one performs better. AB testing is essentially an experiment where two or more variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal. (https://www.optimizely.com/optimization-glossary/ab-testing/ )

A/B testing is about taking an unlimited number of confounding variables and narrowing them down to two:

the change you made (A vs B)

luck (of the draw)

In addition, the various possible statistical frameworks provide methods to help you understand how likely the impact you are seeing are due to your change or to luck

Common issues in using A/B test results...

### Not understanding everything your testing

what your change eg tech error and page load 2

### Not fully understanding your entire audience

80/20 (plats at Marriott), robots in digital

### Not considering the impact of luck

and accounting for it

### Overestimating the scope of the impact

eg it worked here so it will work here

### Not understanding the rules of the game

and accidentally cheating eg erroneous data collection, including the garden of forking paths