# Different experiment methods used in PageSense

Though A/B (or Split URL) testing appears simple—comparing one page against another, to see which performs better—figuring out whether your test results actually mean anything specific is complicated. And for marketers and business owners, understanding visitor problems and finding the right solution, just by looking at the numbers on a report, can be equally challenging. That's why Zoho PageSense uses both Bayesian and Frequentist statistical approaches to compute and interpret your test reports—to determine which variant is performing better and more reliably, and how long the tests should run before you can draw conclusions from it.

Note that it's important to choose the right statistical technique based on your business requirement, before starting your A/B and Split URL experiment. It will help you declare a clear winner or loser, and inform your decision-making process going forward.

### What is Bayesian statistics?

Bayesian statistics in A/B testing calculates the likelihood of variation A performing better than variation B. It takes a bottom-up approach to data analysis that allows you to use previous experiences, or evidence you have on hand (commonly known as "prior"), to calculate the probability of new posterior beliefs ("outcomes") of your experiment.

The Bayesian approach is a fancy way of saying that we use data we already have to make better assumptions about new data received from your website. As we get new data, we refine our “model” of the website, producing more accurate results. It's considered an advanced method of statistical analysis—one that helps you continually refine your results as more data is gathered.

The immediate advantage of this method is that you can understand the test results intuitively within a short period of time, and this will make it easier to communicate with your stakeholders.

Practical scenario
Imagine that you've lost your phone in your house, and hear it ringing in one of the rooms. You know from past experience that you often leave your phone in your bedroom.

Using the Bayesian approach, since you know you often leave your phone in your bedroom, you have an increased chance of finding it there, and you let yourself use that knowledge to locate your phone. Then, each time the phone rings, you're allowed to walk a bit closer to where you think the phone is, making your chances of finding your phone better.

Here's the Bayesian equation (or Bayesian theorem) that was developed by mathematician Thomas Bayes:

where,
• A and B are events,

• P(A|B) is the conditional probability that event A occurs given that event B has already occurred, (P(B|A) has the same meaning, but with the roles of A and B reversed), and

• P(A) and P(B) are the marginal probabilities of event A and event B occurring.

The equation says that, while making a decision, you should use all useful information available, and not just the facts (metrics) you've collected in your current reports. That means, when you're examining the occurrence of an event on a web page, you not only have to look at what’s in front of you, but think about what is likely to be true, as well (the probability of any event to occur).

### Benefits of the Bayesian method of A/B (or Split URL) testing

• Provides more intuitive test results more quickly. This can be very helpful for marketers and business owners who need immediate access to information so they can make faster decisions while ensuring the reliability of the results.

• No need to worry about making the wrong decisions based on insufficient data. Your results are valid whenever you look at them.

• Never have to determine the size of your sample beforehand. Start your test and, as soon as a significant result is identified on the report, it can be relied upon.

• Calculates the potential dangers of ending the test (the loss value) at any point, and gives a constantly updated probability of either variant being better, and by how much.

• Bayesian results are better at detecting even small changes and are easier to understand for people without a statistical background.

### What is Frequentist statistics?

The Frequentist approach to A/B (or Split URL) testing tells us whether variation A will beat B, but not the probability of A performing better than B. It calculates the success of an event with relation to how frequently a particular event occurs in a large number of trials. When applied to the world of A/B testing, you can see that anyone going with the Frequentist approach would need more data (a function of more visitors tested, over longer durations) to come to the right conclusions.

Practical scenario
Let's take the same Bayesian example above: you've lost your phone in your 5-room house, and you often leave your phone in your bedroom.

A Frequentist approach would require you to stand still and listen to your phone ringing, hoping that you can tell with enough certainty which room your phone is left, from where you are standing (without moving!). Moreover, with this approach, you wouldn't be allowed to use that prior knowledge about where you usually leave your phone.

PageSense provides three different modes of Frequentist statistics:

• Quick Trends: This mode is suitable for experiments that don't affect your revenue directly. Pick this mode if you're looking to find a winner in a short span of time.

1. Statistical significance level >90%
2. Ideal for testing non-revenue-related goals, such as quick headline tests or CTA clicks. Provides quick results in a shorter testing time.
• Optimal: This is the default mode, and is an ideal choice for almost all kinds of experiments. Pick this mode if you're looking for fairly accurate results in a reasonable amount of time.

1. Statistical significance level >95%
2. Ideal for all kinds of experiments. Provides fairly accurate results in a moderate testing time.
1. High Accuracy: This is the best mode for critical experiments that affect your revenue directly. Pick this mode if you want to have highly accurate results without worrying about the time taken to achieve it.
1. Statistical significance level >99%
2. Ideal for all kinds of experiments. Provides very accurate results in a longer testing time.
Statistical significance is the probability that the conversion rate between the baseline (original) and a given variation is significant, and not based on some random action or choice. For example, if you run an A/B testing experiment with a significance level of 95%, this means that if you determine a winner, you can be 95% confident that the observed results are real and not an error caused by randomness. It also means that there is a 5% chance that you could be wrong.

### Comparison of Bayesian and Frequentist statistics

 Frequentist statistics Bayesian statistics 01. Frequentist statistics follow the "probability as long-term frequency" definition of probability. Bayesian statistics follow the notions of "probability as degree of belief" and "logical probability." 02. In this approach you only use data from your current experiment. The Frequentist solution is to conduct tests and draw conclusions. In this approach, you use prior knowledge from previous experiments, and try to incorporate that information into your current data. The Bayesian solution is to use existing data along with current data to draw conclusions. 03. Requires the test to run for a set period of time to get correct data from it, and can’t figure out how close or far A and B actually are. It fails to tell you the probability of A beating B. Gives you more control over testing. You can now plan better, have a more accurate reason to end tests, and learn how close or far apart A and B are.

### Choosing your preferred statistical method

Deciding which statistical approach to use for your testing will purely depend on the usage criteria for each test.

If you need results that you can interpret from day one, pick Bayesian. However, these results can vary over time. So, if you choose Bayesian testing, only make decisions if the probable outcome of a winning variation is greater than 90 percent.

Bayesian statistics is a theory that interprets results based on the probability expressed as a degree of belief in an event. Hence, the more you know about an event, the better and faster you can predict the end outcomes. Rather than being a fixed value, probability under Bayesian statistics can change as new information is gathered. This belief may be past information, such as the results of previous tests, or on other information about the event.

On the other hand, pick Frequentist if you're willing to let the test run for at least a month and are already getting significant traffic to your website. This is good if you wish to understand the impact of an isolated hypothesis without considering prior visitor behavior.

Following the Frequentist approach calls for a lot of attention to detail for every test that you run, because for the same set of visitors, you’ll be forced to run longer duration tests compared to the Bayesian approach. Hence, each test needs to be treated with extreme care because there are only a few tests that you can run in a given time frame. Unlike Bayesian statistics, it is less intuitive and often proves difficult to understand. However, it's important to understand that the Frequentist method of testing doesn’t attach probabilities to hypotheses, or to any fixed but unknown values in general. Ignoring this fact is what often leads to misinterpretations of Frequentist analyses.

Once you've locked on either one of these approaches, just start the test and wait for the statistically significant results to roll in.

### Zoho CRM Training Programs

Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.

Redefine the way you work
with Zoho Workplace

### Zoho DataPrep Personalized Demo

If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.

Create, share, and deliver

beautiful slides from anywhere.

Zoho Sign now offers specialized one-on-one training for both administrators and developers.

 Quick Links Workflow Automation Data Collection Web Forms Enterprise Begin Data Collection Interactive Forms Workplace Data Collection App Offline Forms Customer Service Accessible Forms Digital Forms Marketing Forms for Small Business HTML Forms Education Forms for Enterprise Contact Forms E-commerce Lead Generation Forms Healthcare Wordpress Forms Customer onboarding No Code Forms Construction Free Forms Travel Prefill Forms Non-Profit Intake Forms Legal Form Designer HR Card Forms Assign Forms Translate Forms Electronic Forms Notification Emails for Forms Holiday Forms Form to PDF HIPAA Forms

Zoho Pagesense Resources

New to Zoho Survey?

Manage your brands on social media

Zoho Desk Resources

• Desk Community Learning Series

• Digest

• Functions

• Meetups

• Kbase

• Resources

• Glossary

• Desk Marketplace

• MVP Corner

• Word of the Day

# Zoho Sheet Resources

New to Zoho Forms?

# Zoho Forms Resources

communication with Zoho Mail

Mail on the move with
Zoho Mail mobile application

Stay on top of your schedule
at all times

Anytime, anywhere

# Zoho Sign Resources

Sign, Paperless!

Sign and send business documents on the go!

Zoho SalesIQ Resources

# Zoho DataPrep Resources

Zoho DataPrep Demo

Get a personalized demo or POC

Design. Discuss. Deliver.

Create visually engaging stories with Zoho Show.

Wherever you are is as good as

# Resources

### Videos

Watch comprehensive videos on features and other important topics that will help you master Zoho CRM.

### eBooks

Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho CRM.

### Webinars

Sign up for our webinars and learn the Zoho CRM basics, from customization to sales force automation and more.

### CRM Tips

Make the most of Zoho CRM with these useful tips.