The Base Rate Fallacy

March 12, 2019 by Matthew Tamsett

The original, and full article, is on Qubit’s Medium profile

In a Medium blog, Qubit Data Science Lead Matthew Tamsett discusses how the Base Rate Fallacy can affect whether customer experiences are considered successful, and how Qubit Bayesian testing avoids this.

What is Base Rate Fallacy?

No statistical test is perfect.

Tests can fail in different ways, and they come with guarantees about the rates at which they fail. But, when humans fail to understand these failure rates, what they think is a failure rate of 5% might actually be a failure rate of 30%. One such misunderstanding is known as the base rate fallacy.

In A/B testing, you separate the wheat from the chaff. But, sometimes random and unrelated events in the world conspire to make a winning experience look like an insignificant or losing experience (a false negative), or make an insignificant or losing experience look like a winner (a false positive).

How Qubit protects you from this fallacy

The Bayesian A/B tests performed by Qubit measure the probability distribution of an uplift. Therefore, we can make a stronger guarantee: a winning experience has a less than 5% probability of being a fluke. 

We do this by defining a “winner” as an experience where the probability of an uplift is 95% or more. Exactly 5% of these winning experiences are illusory, no matter the base rate of winning experiences. In this way, the Qubit platform preemptively prevents its users from committing the base rate fallacy.

Conclusion

The distinction between false positive rate and false discovery rate is very important one, but classical statistical testing makes it far too easy to forget about it and commit the base rate fallacy. This can, in theory, lead to A/B tests that are impractically stringent.

But, in practice, modern websites are already optimised to some degree and winning experiences are rare, leading to tests that are too lax and declare illusory winners too often. The Bayesian messaging in the Qubit platform doesn’t let you make this mistake.

 

Read the full article on Medium.

Author Matthew Tamsett Read more
Tags:

Subscribe to stay up to date.

Receive the personalization newsletter directly to your inbox.

Please enter a valid e-mail address.

We will treat your data with respect and you can find the details in our privacy and cookie policy, and our website terms of use.

We use cookies and other forms of website navigational information to offer you a better browsing experience, analyze site traffic, personalize content, and serve targeted advertisements.
Read about how we use cookies and how you can control them in our Privacy and Cookie Policy. If you continue to use this site, you consent to our use of cookies.

Accept Cookies