Different experiment methods used in Zoho PageSense | Help Guide

Different experiment methods used in PageSense

Though A/B (or Split URL) testing appears simple—comparing one page against another, to see which performs better—figuring out whether your test results actually mean anything specific is complicated. And for marketers and business owners, understanding visitor problems and finding the right solution, just by looking at the numbers on a report, can be equally challenging. That's why Zoho PageSense uses both Bayesian and Frequentist statistical approaches to compute and interpret your test reports—to determine which variant is performing better and more reliably, and how long the tests should run before you can draw conclusions from it. 



Note that it's important to choose the right statistical technique based on your business requirement, before starting your A/B and Split URL experiment. It will help you declare a clear winner or loser, and inform your decision-making process going forward.

What is Bayesian statistics?

Bayesian statistics in A/B testing calculates the likelihood of variation A performing better than variation B. It takes a bottom-up approach to data analysis that allows you to use previous experiences, or evidence you have on hand (commonly known as "prior"), to calculate the probability of new posterior beliefs ("outcomes") of your experiment.

The Bayesian approach is a fancy way of saying that we use data we already have to make better assumptions about new data received from your website. As we get new data, we refine our “model” of the website, producing more accurate results. It's considered an advanced method of statistical analysis—one that helps you continually refine your results as more data is gathered.

The immediate advantage of this method is that you can understand the test results intuitively within a short period of time, and this will make it easier to communicate with your stakeholders.

Practical scenario
Imagine that you've lost your phone in your house, and hear it ringing in one of the rooms. You know from past experience that you often leave your phone in your bedroom.

Using the Bayesian approach, since you know you often leave your phone in your bedroom, you have an increased chance of finding it there, and you let yourself use that knowledge to locate your phone. Then, each time the phone rings, you're allowed to walk a bit closer to where you think the phone is, making your chances of finding your phone better.

Here's the Bayesian equation (or Bayesian theorem) that was developed by mathematician Thomas Bayes:




where,
  • A and B are events,

  • P(A|B) is the conditional probability that event A occurs given that event B has already occurred, (P(B|A) has the same meaning, but with the roles of A and B reversed), and

  • P(A) and P(B) are the marginal probabilities of event A and event B occurring.

The equation says that, while making a decision, you should use all useful information available, and not just the facts (metrics) you've collected in your current reports. That means, when you're examining the occurrence of an event on a web page, you not only have to look at what’s in front of you, but think about what is likely to be true, as well (the probability of any event to occur).

Benefits of the Bayesian method of A/B (or Split URL) testing

  • Provides more intuitive test results more quickly. This can be very helpful for marketers and business owners who need immediate access to information so they can make faster decisions while ensuring the reliability of the results.

  • No need to worry about making the wrong decisions based on insufficient data. Your results are valid whenever you look at them.

  • Never have to determine the size of your sample beforehand. Start your test and, as soon as a significant result is identified on the report, it can be relied upon.

  • Calculates the potential dangers of ending the test (the loss value) at any point, and gives a constantly updated probability of either variant being better, and by how much.

  • Bayesian results are better at detecting even small changes and are easier to understand for people without a statistical background.

What is Frequentist statistics?

The Frequentist approach to A/B (or Split URL) testing tells us whether variation A will beat B, but not the probability of A performing better than B. It calculates the success of an event with relation to how frequently a particular event occurs in a large number of trials. When applied to the world of A/B testing, you can see that anyone going with the Frequentist approach would need more data (a function of more visitors tested, over longer durations) to come to the right conclusions.

Practical scenario
Let's take the same Bayesian example above: you've lost your phone in your 5-room house, and you often leave your phone in your bedroom.

A Frequentist approach would require you to stand still and listen to your phone ringing, hoping that you can tell with enough certainty which room your phone is left, from where you are standing (without moving!). Moreover, with this approach, you wouldn't be allowed to use that prior knowledge about where you usually leave your phone.

PageSense provides three different modes of Frequentist statistics:

  • Quick Trends: This mode is suitable for experiments that don't affect your revenue directly. Pick this mode if you're looking to find a winner in a short span of time.

  1. Statistical significance level >90%
  2. Ideal for testing non-revenue-related goals, such as quick headline tests or CTA clicks. Provides quick results in a shorter testing time.
  • Optimal: This is the default mode, and is an ideal choice for almost all kinds of experiments. Pick this mode if you're looking for fairly accurate results in a reasonable amount of time.

  1. Statistical significance level >95%
  2. Ideal for all kinds of experiments. Provides fairly accurate results in a moderate testing time.
  1. High Accuracy: This is the best mode for critical experiments that affect your revenue directly. Pick this mode if you want to have highly accurate results without worrying about the time taken to achieve it.
  1. Statistical significance level >99%
  2. Ideal for all kinds of experiments. Provides very accurate results in a longer testing time.
Statistical significance is the probability that the conversion rate between the baseline (original) and a given variation is significant, and not based on some random action or choice. For example, if you run an A/B testing experiment with a significance level of 95%, this means that if you determine a winner, you can be 95% confident that the observed results are real and not an error caused by randomness. It also means that there is a 5% chance that you could be wrong. 

Comparison of Bayesian and Frequentist statistics

 

Frequentist statistics

Bayesian statistics

 

 01.

Frequentist statistics follow the "probability as long-term frequency" definition of probability.

Bayesian statistics follow the notions of "probability as degree of belief" and "logical probability."

 02.

In this approach you only use data from your current experiment. The Frequentist solution is to conduct tests and draw conclusions.

In this approach, you use prior knowledge from previous experiments, and try to incorporate that information into your current data. The Bayesian solution is to use existing data along with current data to draw conclusions.

 03.

Requires the test to run for a set period of time to get correct data from it, and can’t figure out how close or far A and B actually are. It fails to tell you the probability of A beating B.

Gives you more control over testing. You can now plan better, have a more accurate reason to end tests, and learn how close or far apart A and B are.


Choosing your preferred statistical method 

Deciding which statistical approach to use for your testing will purely depend on the usage criteria for each test. 

If you need results that you can interpret from day one, pick Bayesian. However, these results can vary over time. So, if you choose Bayesian testing, only make decisions if the probable outcome of a winning variation is greater than 90 percent.

Bayesian statistics is a theory that interprets results based on the probability expressed as a degree of belief in an event. Hence, the more you know about an event, the better and faster you can predict the end outcomes. Rather than being a fixed value, probability under Bayesian statistics can change as new information is gathered. This belief may be past information, such as the results of previous tests, or on other information about the event.

On the other hand, pick Frequentist if you're willing to let the test run for at least a month and are already getting significant traffic to your website. This is good if you wish to understand the impact of an isolated hypothesis without considering prior visitor behavior.

Following the Frequentist approach calls for a lot of attention to detail for every test that you run, because for the same set of visitors, you’ll be forced to run longer duration tests compared to the Bayesian approach. Hence, each test needs to be treated with extreme care because there are only a few tests that you can run in a given time frame. Unlike Bayesian statistics, it is less intuitive and often proves difficult to understand. However, it's important to understand that the Frequentist method of testing doesn’t attach probabilities to hypotheses, or to any fixed but unknown values in general. Ignoring this fact is what often leads to misinterpretations of Frequentist analyses.


Once you've locked on either one of these approaches, just start the test and wait for the statistically significant results to roll in.

    Zoho CRM Training Programs

    Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.

    Zoho CRM Training
      Redefine the way you work
      with Zoho Workplace

        Zoho DataPrep Personalized Demo

        If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.

        Zoho CRM Training

          Create, share, and deliver

          beautiful slides from anywhere.

          Get Started Now


            Zoho Sign now offers specialized one-on-one training for both administrators and developers.

            BOOK A SESSION










                                    You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.




                                        Manage your brands on social media

                                          Zoho Desk Resources

                                          • Desk Community Learning Series


                                          • Digest


                                          • Functions


                                          • Meetups


                                          • Kbase


                                          • Resources


                                          • Glossary


                                          • Desk Marketplace


                                          • MVP Corner


                                          • Word of the Day


                                            Zoho Marketing Automation

                                              Zoho Sheet Resources

                                               

                                                  Zoho Forms Resources


                                                    Secure your business
                                                    communication with Zoho Mail


                                                    Mail on the move with
                                                    Zoho Mail mobile application

                                                      Stay on top of your schedule
                                                      at all times


                                                      Carry your calendar with you
                                                      Anytime, anywhere




                                                            Zoho Sign Resources

                                                              Sign, Paperless!

                                                              Sign and send business documents on the go!

                                                              Get Started Now




                                                                      Zoho TeamInbox Resources



                                                                              Zoho DataPrep Resources



                                                                                Zoho DataPrep Demo

                                                                                Get a personalized demo or POC

                                                                                REGISTER NOW


                                                                                  Design. Discuss. Deliver.

                                                                                  Create visually engaging stories with Zoho Show.

                                                                                  Get Started Now









                                                                                                      • Related Articles

                                                                                                      • Declaring a winner in A/B (or Split URL) test in PageSense

                                                                                                        Remember, A/B testing is not just about running tests and finding the winning variation for your web page, it is also about learning how different factors contributed to the win. Before implementing the highest winning design on your website, it's ...
                                                                                                      • Create and launch an A/B test in PageSense

                                                                                                        A/B testing helps you analyze and observe how one version of a web page performs alongside another in front of your audience. On Zoho PageSense, you can quickly create, edit, and launch different versions of your web page, and test which one version ...
                                                                                                      • Setup triggers for your experiment in PageSense

                                                                                                        In PageSense, you can create and launch different types of polls and pop-ups to grab the attention of visitors coming to your site. Even though these on-site messages are a powerful way to kindle your audience's interest towards your brand, if they ...
                                                                                                      • Create and launch a Split URL test in PageSense

                                                                                                        Use Split URL testing when you want to test multiple versions of your web page hosted on different URLs. This type of test is considered best when you notice several deficiencies in your existing page that might affect conversion. In such cases, it's ...
                                                                                                      • Targeting specific audience segments in your experiment

                                                                                                        Audience segmentation in PageSense typically means dividing or grouping website visitors into subgroups and subcategories based on a common set of characteristics, behavior, needs, and values. For example, you can segment users based on the device ...
                                                                                                        Wherever you are is as good as
                                                                                                        your workplace

                                                                                                          Resources

                                                                                                          Videos

                                                                                                          Watch comprehensive videos on features and other important topics that will help you master Zoho CRM.



                                                                                                          eBooks

                                                                                                          Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho CRM.



                                                                                                          Webinars

                                                                                                          Sign up for our webinars and learn the Zoho CRM basics, from customization to sales force automation and more.



                                                                                                          CRM Tips

                                                                                                          Make the most of Zoho CRM with these useful tips.



                                                                                                            Zoho Show Resources