Most winning A/B test results are illusory

This academic paper shows that badly performed A/B tests can produce winning results which are more likely to be false than true. At best, this leads to the needless modification of websites; at worst, to modification which damages profits.

We introduce three simple concepts which come as second nature to statisticians but have been forgotten by many web A/B testing specialists: ‘statistical power’, ‘multiple testing’ and ‘regression to the mean’. Armed with them, you will be able to cut through the misinformation and confusion that plague this industry.