Bayesian testing
The Bayesian approach to A/B testing asks questions about the world that generated the data. for example, “What are the most probable values of the uplift?” or “What is the probability that the uplift is positive?” This differs from the Frequentist approach which instead asks questions about the data, for example, “What is the probability I’d see something more surprising if I ran my test again 1000 times?”
How is Bayesian testing different from other kinds of A/B testing?
The practical difference in the Bayesian approach to A/B Testing is that you can start looking at your test results on day one, maybe even after the first hour, just by reading the probability of whether A and B is better than the other. Doing so does not affect the false discovery rate, which is automatically controlled by Bayesian techniques.
That’s not to say only the Bayesian questions matter; Qubit also guarantees the frequentist properties of its Bayesian A/B tests.
