A/B testing: why would you only test with 20% of your traffic?
It’s important to do A/B testing on your website to ensure that the changes you make to your site are increasing conversions. But it’s imperative to A/B test those parts of your sites which are considered to be ‘risky’, such as the checkout, where the downside of a test is potentially significant, and if pushed live would have a negative impact on your site’s performance.
To help in this situation, we’re excited to introduce the ability to run a ‘Pilot’ test in Deliver. Pilot enables you to run a version of your personalization experiment in which your variant is only shown to 20% of your segment’s traffic. In other words, only 1 in 5 users will be exposed to the new variation if you are running a standard A/B test with a single variant. In addition, we have adapted our statistical model to look specifically for large negative impacts.
Why did we pick 20%?
We found that 20% is the most efficient way to determine whether the experiment is having an impact. As you know, it’s important to remember sample size. Reducing the number of visitors to whom you expose your experiment to ultimately means that your experiment just needs to run longer. Overall, it is impossible to reduce the total number of visitors seeing a variant by more than a factor of 2 (e.g. from 10,000 to 5,000 visitors), while still getting meaningful results. Think of it like driving a car at your most fuel efficient speed to get to your destination, you can drive slower than 55mph, but you will end up taking a lot longer to get to your destination. 20% enables you to get to determine a result, whilst minimising the length of time required to establish a meaningful result.
For guidance, the chart below shows the total number of visitors required to be present on your site, and subsequently the number of visitors which would be exposed to the experiment to establish if an effect on performance is present using different traffic allocations.
Note, that across the traffic allocations of 95/5, 90/10 and 80/20 the number of visitors exposed to the variant does not change. Instead, all that happens is that the test would be required to run for a longer time period as the number of visitors required in total increases as the proportion of traffic exposed to the variation reduces.