Remember, A/B testing is not just about running tests and finding the winning variation for your web page, it is also about learning how different factors contributed to the win. Before implementing the highest winning design on your website, it's important to take a close look at any outcome (winning, leading, or inconclusive) you obtained during your experiment. This will help you understand your audience's behavior towards a particular variation and make appropriate decisions in the future. 

In Zoho PageSense, we consider several criteria that should be met to conclude your A/B test results and declare a winner. This includes the number of visitors, the number of conversions, conversion rate, improvement, significance, and other related metrics for each goal configured in your variation page. These metrics can provide you with the level of accuracy with which you can trust the results in your A/B test reports run using PageSense.

In this article, we will see the major parameters that contribute to a winning variation, as well as the factors you should consider while analyzing your A/B (or Split URL) experiment results in PageSense.

Declaring results using the Frequentist method

1. Have reached the adequate visitor count (or sample size)
   
2. Have a higher conversion rate on the primary goal
   
3. Have achieved the set statistical significance level.
   

Visitor count/Sample size

Calculating the required sample size in advance and running your experiment on a set visitor count can help you reach the required significance level, without wasting much of your time and resources.

Info: How can sample size help you achieve the desired significance level?

In PageSense, the final results of an A/B test is declared based on the Significance level achieved. However, the calculation of Significance percentage is subject to a certain degree of error based on the visitor traffic and conversion rate obtained.

For instance, when there are fewer visitors and conversions, the significance level fluctuates in a wide margin. You may see that at one point the significance can be 95%. After a few more visitors, it may drop to 70%. In this case, considering a sample size can help the statistical significance to stabilize after a good number of visitors are reached on your variation page. These metrics will remain stable from there on.  

Conversion rate

1. Original version: 42,000 unique visitors and 10,000 conversions, with a 23.8% conversion rate.
   
2. Variation 1: 86, 000 unique visitors and 22,000 conversions, with a 25.5% conversion rate.

Info: Deciding a variation to be a winner or loser just by looking at the highest conversion rate before the end of the test might prove wrong and inefficient, as it would not have been tested on the majority of visitor numbers.   

Significance level

1. Quick trends: Statistical significance level > 90%
   
2. Optimal: Statistical significance level > 95%
   
3. High accuracy: Statistical significance level > 99%
   

1. Original version: 42,000 unique visitors and 10,000 conversions, with a 23.8% conversion rate.
   
2. Variation 1: 86,000 unique visitors and 22,000 conversions, with a 25.5% conversion rate.
   

By calculating the statistical significance, we notice that 'Variation 1' is more efficient than the original variation, and that the significance percentage is 96. In this case, your A/B test is a success, and you can now use 'Variation 1' on your web page.

Statistical significance depends on two variables: The number of visitors (or sample size) and the conversion rates for both the original and any variations. To ensure that your A/B tests reaches the desired statistical significance level at the earliest, you need to plan your testing hypothesis by keeping both these variables in mind.

Below are a few general cases for declaring your A/B test results in PageSense:

• If visitor count and significant is reached
  If the visitor count is attained for both original and variation, then the variation with the highest conversion rate and significance level reached is declared the winner.

• If visitor count and significant is not reached
  PageSense will show you the remaining count of visitors required for each variation, until which you need to run your A/B test to get a conclusive result.

• If visitor count is reached but significance is not reached
  If neither of the variations are statistically better, even after reaching the visitor count, then the test is marked as inconclusive. In this case, you can stick with the original variation or run another test. You can use the failed data to help you identify a new hypothesis for your next test.

Declaring results using the Bayesian method

Number of visitors and conversions metric

The Bayesian method of A/B testing is about calculating the probability percentage for both the original and the variation pages. This probability percentage can be quickly achieved based on two major factors: the number of visitors and the number of conversions.

• The number of visitors indicates the total number of unique visitors to the corresponding variation. Ideally, your visitor count should be big enough for accurate interpretation of the A/B test results. In other words, the bigger the visitor count, the higher the probability of declaring a winner.
  

• On the other hand, the number of conversions indicate the number of unique instances of the visitor fulfilling the desired action for a given goal. It can refer to any desired action that you want the user to perform on your web page, from clicking on a button to making a purchase and becoming a customer.

Note: In comparison to the Frequentist method of testing, the Bayesian tests do not require fixing the sample to give valid results.

In PageSense, we recommend following two business rules before making a decision using the Bayesian approach:

• Wait until you have recorded at least 100 unique visitors per variation
  

• Wait until you have reached at least 50 conversions on the primary goal.
  

• Check if it has a lesser than negligible loss value.
  

• Check if the primary goal value meets the required criteria to declare an Experiment as the winner.
  

• Check if all the mentioned values are collected and satisfied to declare a Goal as the winner.

Expected loss