Though A/B (or Split URL) testing appears simple—comparing one page against another, to see which performs better—figuring out whether your test results actually mean anything specific is complicated. And for marketers and business owners, understanding visitor problems and finding the right solution, just by looking at the numbers on a report, can be equally challenging. That's why Zoho PageSense uses both Bayesian and Frequentist statistical approaches to compute and interpret your test reports—to determine which variant is performing better and more reliably, and how long the tests should run before you can draw conclusions from it. 

Note that it's important to choose the right statistical technique based on your business requirement, before starting your A/B and Split URL experiment. It will help you declare a clear winner or loser, and inform your decision-making process going forward.

What is Bayesian statistics?

Bayesian statistics in A/B testing calculates the likelihood of variation A performing better than variation B. It takes a bottom-up approach to data analysis that allows you to use previous experiences, or evidence you have on hand (commonly known as "prior"), to calculate the probability of new posterior beliefs ("outcomes") of your experiment.

The Bayesian approach is a fancy way of saying that we use data we already have to make better assumptions about new data received from your website. As we get new data, we refine our “model” of the website, producing more accurate results. It's considered an advanced method of statistical analysis—one that helps you continually refine your results as more data is gathered.

The immediate advantage of this method is that you can understand the test results intuitively within a short period of time, and this will make it easier to communicate with your stakeholders.

Practical scenario
Imagine that you've lost your phone in your house, and hear it ringing in one of the rooms. You know from past experience that you often leave your phone in your bedroom.

Using the Bayesian approach, since you know you often leave your phone in your bedroom, you have an increased chance of finding it there, and you let yourself use that knowledge to locate your phone. Then, each time the phone rings, you're allowed to walk a bit closer to where you think the phone is, making your chances of finding your phone better.

Here's the Bayesian equation (or Bayesian theorem) that was developed by mathematician Thomas Bayes:

• A and B are events,
  

• P(A|B) is the conditional probability that event A occurs given that event B has already occurred, (P(B|A) has the same meaning, but with the roles of A and B reversed), and
  

• P(A) and P(B) are the marginal probabilities of event A and event B occurring.
  

The equation says that, while making a decision, you should use all useful information available, and not just the facts (metrics) you've collected in your current reports. That means, when you're examining the occurrence of an event on a web page, you not only have to look at what’s in front of you, but think about what is likely to be true, as well (the probability of any event to occur).

Benefits of the Bayesian method of A/B (or Split URL) testing

• Provides more intuitive test results more quickly. This can be very helpful for marketers and business owners who need immediate access to information so they can make faster decisions while ensuring the reliability of the results.
  

• No need to worry about making the wrong decisions based on insufficient data. Your results are valid whenever you look at them.
  

• Never have to determine the size of your sample beforehand. Start your test and, as soon as a significant result is identified on the report, it can be relied upon.
  

• Calculates the potential dangers of ending the test (the loss value) at any point, and gives a constantly updated probability of either variant being better, and by how much.
  

• Bayesian results are better at detecting even small changes and are easier to understand for people without a statistical background.
  

What is Frequentist statistics?

The Frequentist approach to A/B (or Split URL) testing tells us whether variation A will beat B, but not the probability of A performing better than B. It calculates the success of an event with relation to how frequently a particular event occurs in a large number of trials. When applied to the world of A/B testing, you can see that anyone going with the Frequentist approach would need more data (a function of more visitors tested, over longer durations) to come to the right conclusions.

Practical scenario
Let's take the same Bayesian example above: you've lost your phone in your 5-room house, and you often leave your phone in your bedroom.

A Frequentist approach would require you to stand still and listen to your phone ringing, hoping that you can tell with enough certainty which room your phone is left, from where you are standing (without moving!). Moreover, with this approach, you wouldn't be allowed to use that prior knowledge about where you usually leave your phone.

PageSense provides three different modes of Frequentist statistics:

• Quick Trends: This mode is suitable for experiments that don't affect your revenue directly. Pick this mode if you're looking to find a winner in a short span of time.
  

1. Statistical significance level >90%
   
2. Ideal for testing non-revenue-related goals, such as quick headline tests or CTA clicks. Provides quick results in a shorter testing time.
   

• Optimal: This is the default mode, and is an ideal choice for almost all kinds of experiments. Pick this mode if you're looking for fairly accurate results in a reasonable amount of time.
  

1. Statistical significance level >95%
   
2. Ideal for all kinds of experiments. Provides fairly accurate results in a moderate testing time.

1. High Accuracy: This is the best mode for critical experiments that affect your revenue directly. Pick this mode if you want to have highly accurate results without worrying about the time taken to achieve it.
   

1. Statistical significance level >99%
   
2. Ideal for all kinds of experiments. Provides very accurate results in a longer testing time.
   

Statistical significance is the probability that the conversion rate between the baseline (original) and a given variation is significant, and not based on some random action or choice. For example, if you run an A/B testing experiment with a significance level of 95%, this means that if you determine a winner, you can be 95% confident that the observed results are real and not an error caused by randomness. It also means that there is a 5% chance that you could be wrong. 

Comparison of Bayesian and Frequentist statistics

Following the Frequentist approach calls for a lot of attention to detail for every test that you run, because for the same set of visitors, you’ll be forced to run longer duration tests compared to the Bayesian approach. Hence, each test needs to be treated with extreme care because there are only a few tests that you can run in a given time frame. Unlike Bayesian statistics, it is less intuitive and often proves difficult to understand. However, it's important to understand that the Frequentist method of testing doesn’t attach probabilities to hypotheses, or to any fixed but unknown values in general. Ignoring this fact is what often leads to misinterpretations of Frequentist analyses.