Understand your A/B and Split URL test results

The A/B (and Split URL) test reports in PageSense give you both a quick glance at the overall success measure of your experiment, as well as in-depth visitor metrics for each variation, including the conversion rate, difference interval, significance, and others. This data-rich information report helps you instantly analyze the performance of different elements on your web page, and determine which variation page performs better among your audiences.

To access your A/B (or Split URL) test reports in PageSense:
Open the A/B (or Split URL) experiment and click the REPORTS tab on the top bar. You will see three different tabs under the hood: Overview, Detailed analysis, and Heatmap.

Overview tab

Here you can see a summary and a leaderboard table for each goal metric attached to your experiment. You can use these numbers to check which conversion goal has performed the best or is leading with respect to individual variations in your A/B test

Summary

This sections presents you with all the major stats of your A/B (or Split URL test) such as the total visitors count, number of variants, number of goals, the experiment start day, the day it was paused, and the active duration of all the variants, including the original.

Leaderboard

A/B test result is always based around a goal. The Leaderboard section shows you the list of all goals attached to your experiment, with a star marking the primary goal. Each goal has a table listing the following information:

Goal result: Indicates the winning and leading variant of a goal based on the conversion rates calculated for your original and variation pages of your A/B test.

Winning variant: The variation that performs the best with the maximum conversion rate, and reaches the significance level and sample size for a given conversion goal in your A/B test.
Leading variant: The variation that is performing better than its competitor variations for a given conversion goal in your A/B test.
No variant is leading currently: The variation performs similar to the original version for a given conversion goal.
{variant names} are leading the variants: Appears when more than one variation is performing better or leading the competitors for a given conversion goal at the same time.

Rank: Indicates the rank of the variants based on their performance to a corresponding goal.
Variant: Indicates the list of all the variants available in your experiment page, including the original.
Visitors: Indicates the total number of unique visitors to the corresponding variation.
Conversions: Indicates the number of unique instances of the visitor fulfilling the desired action for a given goal. A conversion can refer to any action that you want the user to take. This can include any action performed on your web page, from clicking on a button to making a purchase and becoming a customer.
Conversion rate: The conversion rate is the number of conversions divided by the total number of visitors. For example, if your variation page receives 200 visitors in a month and has 50 purchases, the conversion rate would be 50 divided by 200, or 25%.
The conversion rate is represented by percentage value and can have any positive.

Warnings and notification messages

The notification banner on the top displays the status of your experiment and any warnings you must be aware of related to the issues with data collection in a variation or with conversion tracking.

The following is a list of possible errors or warning messages that you could see in your Reports tab:

"We recommend you to run this experiment till it reaches a conclusive result."
The cause: This may happen when the desired visitor count has not been reached to conclude the results. In this case, you need to wait longer to reach the desired visitor count, have a better conversion rate, and reach the required significance level you chose.
"The result is inconclusive as the performance of the original and variation is similar."
The cause: This could be because both the original and the variation version of your webpage perform similarly and hence there is no better advantage of publishing a new variation over the original. In this case, we recommend that you end the test and create a new A/B experiment for your page.

Detailed analysis tab

This section shows the performance trend of each goal on a graph.

Understanding the graph

This graph compares the performance and trends of each variation against a chosen baseline over time. By default, the original is set as the baseline for comparing your goal's data. You can change this baseline to any other variation using the Baseline and Compare with dropdown.

You can view your graph in either of the following ways:

Time-based graph: This graph illustrates multiple spikes of data distributed across a chosen time frame (along the x-axis). It is time based because the data values or points are calculated over regular time intervals, and will be considered independent of each other.
For example, let's say you introduced a Black Friday Sale (12th of April) on your website. You see 100 new visitors coming to your variation page on this specific day, out of which 40 got converted, with a conversion rate of 40% (which is 20% higher than the previous day, 20th of April). In this case, using the time-based view can help you compare the visitor counts and conversion rates of your data on the days before and after the sale. This could later be used to interpret if the introduced change on your web page is influencing your conversion rate over time.
In this type of graph pattern (shown below), you can also occasionally observe conversions drop to a zero value on any given day within the time range. This might be quite normal, due to lower traffic or a lower conversion rate. One possible implication of such a pattern on your test with a high traffic scope may be a signal to double-check the quality of the test setup, as something might have broken. This can in turn be a QA check during a live test.

Cumulative graph: This graph illustrates flattened data distributed across a chosen time frame (along the x-axis). It is cumulative because the data values or points are calculated as a sum over time, like computing the sum of the first point, then the first two points, then the first three, and so on.
For example, let's say you started an A/B experiment on the 1st of April and you want to check the conversion rate on the 15th of April to see how your experiment is performing every two weeks. In this case, using the cumulative graph provides you with a quick inference on the average performance of your variation's data, summed up until mid-April.
In this type of graph pattern (shown below), you can also see that the effects of the original and he variations will fluctuate more in the beginning than later in a test. One possible implication of such a pattern might be to delay a test from being stopped prematurely until greater horizontal stability is first visualized.

Further, the graph allows you to filter your goal metrics between different variations against individual performance indicators such as Conversion Rate, Conversions, Visitors, Improvement, and Significance from the dropdown.
View Forecast: Enabling the Forecast button helps you to view predictions or changes in the conversion rate (and improvement rate) that each variation might take over the chosen time range. Forecast is an estimated value of change over a future time horizon. This can be used to examine the performance of your A/B results, and predict your success in the long term.

Understanding the table

This section consists of tables for each goal, with in-depth and detailed statistics that contain metrics such as Conversion rate, Conversions, Difference interval, Improvement, and Significance.

Variant: Indicates the list of all the variants available in your experiment page, including the original.
Visitors: Indicates the total number of unique visitors to the corresponding variation.
Conversions: Indicates the number of unique instances of the visitor fulfilling the desired action for a given goal. A conversion can refer to any desired action that you want the user to take. This can include any action performed on your web page, from clicking on a button to making a purchase and becoming a customer.
Conversion rate: The conversion rate is the number of conversions divided by the total number of visitors.

For example, if your variation page receives 200 visitors in a month and has 50 purchases, the conversion rate would be 50 divided by 200, or 25%. The conversion rate is represented by percentage value, and can have any positive or negative value.

Difference Interval: When communicating the results, it's not only a good idea to present the observed difference in the conversion rate value for the original and variation, but also the range between which the conversion rate of your original and the variation page can actually lie. This possible range of values is called the difference interval, and is plotted on a number line scale.

On the number scale, the upper limit is marked by the maximum possible range of conversion rate, and the lower limit is marked by the minimum possible range of conversion rate between all the variations. The different shades on the scale indicate the following:

Grey shade area: Indicates the experiment is inconclusive, or needs more visitors to declare a valid result.
Green shade area: Indicates a winning variation.
Red shade area: Indicates a losing variation.

For example, let's say you test your variation page with 7626 visitors, and get 1722 conversions with a conversion rate of 22.58%. For this, you see a difference interval between 21.64 % - 23.52 %, which means that your conversion rate value could lie anywhere between this difference based on the new visitors and conversions obtained on your variation page.
Note:
One thing to watch out for in difference interval is the overlap of the conversion rate from the Original and Variation 1 of your A/B test. For example, suppose that Original has a confidence interval of 10-20% for conversion rates, and Variation 1 has a confidence interval of 15-25%. Notice that the overlap of the two confidence intervals is 5%, and it is located in the range between 15-20%. In this case, it's very difficult to be sure that the variation tested in B is actually a significant improvement. This is why we will not declare winner if there is overlap.

Improvement: Improvement denotes the relative difference between the conversion rate of the original and variation version of your web page. The improvement percentage can be a positive or negative value.
For example, if the conversion rate of variation is 30%, and the conversion rate of the original is 15%, then the improvement is (30- 15)/15*100 = 100 % points. In this case, there is a 100% increase in the conversion rate for the variation page.
Significance (for the Frequentist approach): For any A/B test, the most important concept for interpreting your results is statistical significance. Statistical significance tells you whether a variation is outperforming or under performing the baseline, at whatever level of significance level you have chosen. This significance level is decided based on the type of frequentist method you have applied to evaluate your results.
Significance is defined as the probability that the difference between your variation A and B page's conversion rate is due to real changes in your visitor's behavior and not based on some random action or choice. Simply put, if a statistics has a higher significance, then the results are considered more reliable.
For example, if your A/B test shows a significance level of 95% this means that if you determine a winner, you can be 95% confident that the observed results are real, and not an error caused by randomness. It also means that there is a 5% chance that you could be wrong. Learn the importance of statistical significance in declaring a winning variation in your A/B experiment.
Probability to win (the Bayesian approach): This is defined as the probability of a variation to outperform the original version in your A/B test. Probability to win tells you which variation is better and by how much in percentage. This is the most actionable metric used in the Bayesian method to declare the winner of your A/B tests. The winner of this approach will be declared when these three primary rules are met:
1. Has achieved a unique visitor count of 100 or above for the variations.
2. Has gathered a unique conversion of 50 or above for the goals setup.
3. Has an average of minimal loss value.

Once the primary rules are satisfied, in the Bayesian approach, we declare winners based on two other parameters:
1. To declare a variation from the Experiment as the winner, the primary goal value should meet the visitor count, conversion counts, and expected loss value.
2. To declare a variation from the Goal as the winner, we will take in visitor count, conversion count and expected loss value.

Heatmap tab

By activating a heatmap in your A/B (or Split URL) experiment, you can visualize and identify where your visitors are clicking, scrolling, and looking in your original and variation pages. This report also provides you additional insights on why a particular variation did not perform as expected, and what needs to be changed to address issues.

What heatmap can tell you about your A/B (or Split URL) test

Click the HEATMAP tab to compare the visitor data between the original and variation to discover how elements perform differently on each page. From important CTA elements, links, or videos that attract less clicks, to other unimportant elements distracting visitors from getting converted on each page, heatmap reveals where visitors are clicking on your web pages, and what opportunities are available to turn your A/B (or Split URL) test into the winner you were hoping for.

These reports use a colored hotspot to show which element on the page gets more clicks, or vice versa. Blue means fewer clicks, warmer reds indicate more clicks, and the most clicks are bright yellow spots. Based on these heatmap metrics, you can quickly see which areas of the page get a lot of action, and which do not. Based on this, you can make the desired position change in your variation page and run your next A/B test.

What scroll maps can tell you about your A/B (or Split URL) test

Click SCROLLMAP to compare and discover which part of the page (either original or variations) is most effective at keeping users engaged and interested. This is especially important to provide insight into what percentage of your visitors are scrolling far enough down the page to see your call to action, and to analyze why visitors are bouncing back even before they reach half of your web page length. If a high proportion of your visitors are not scrolling down far enough to see it, they certainly will not be clicking on it.

These reports use a color-coded gradient to show where users spend the most time. The warmer an area appears, the more attention it gets from your visitors; it fades out gradually with reduced visitor activity. Based on these scroll map metrics, you can try creating variation pages of varying lengths, place the most critical elements of the web page on sections where visitors spend the most time, and see how it improves the conversion.

What attention map can tell you about your A/B (or Split URL) test

Click ATTENTIONMAP to compare and see how long your users spend time on your original and variation pages. This metrics is more useful to see if the key pieces of information, such as text and visuals, are visible to most users, and which section of the page your visitors are more enraged with, by moving their mouse trying to read information on your page.

These reports use a colored overlay to reflect the level of interaction on your pages. A red area is where there is the most activity, while blue indicates less amount of activity. Based on these attention map metrics, you can track which section attracts visitors’ gaze the most on your variations, then place all important content and images in the most attention-grabbing sections of the page for testing.

By combining the knowledge from heatmaps with A/B testing reports, you can increase the conversions on your variation pages, and achieve a higher statistical significance for declaring a winner.

Access your files securely from anywhere

Zoho CRM Training Programs

Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.

Redefine the way you work

with Zoho Workplace

Start your free trial

Zoho DataPrep Personalized Demo

If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.

Create, share, and deliver

beautiful slides from anywhere.

Get Started Now

Zoho Sign now offers specialized one-on-one training for both administrators and developers.

BOOK A SESSION

Zoho Workerly Home
Forums
Connect With Us:

Zoho Recruit Home
Forums
Connect With Us:

Start tracking your website metrics today

Use PageSense to understand what your visitors are looking for, how they are behaving, and where they are facing problems on your website.

Ready to improve your website's performance?

Install the PageSense code snippet on your site in a matter of minutes and start collecting in-depth data about the website visitors to grow your business.

Track and measure every business goal

Set up goals in PageSense to measure every single action performed by visitors on your website like button or link clicks, form submissions, and page engagements.

Understand the customer journey on your website

Create funnels in PageSense to quickly see which pages visitors use to enter your website, where they navigate to next, and which pages they decide to leave without converting.

Visualize your visitor's behavior with color codes

Set up heatmaps in PageSense to see where users have clicked more, how far they've scrolled, and on which parts of a page they've spent the most time using color-coded patterns in reports.

Measure customer engagement with your web forms

Use form analytics in PageSense to see how people interact with different fields in your form, whether they complete the form successfully or not, and where exactly they drop out on your form.

Record real visitor interactions on your website

Use session recordings in PageSense to watch a video of all the visitor actions performed on your website including the pages they navigate, the buttons they click, the UX issues they face, and more.

Test and optimize webpages for better conversions

Run A/B or Split URL tests in PageSense to figure out which version of your web page works best for your business and results in the best conversion rate.

Give each customer a unique website experience

Use personalization in PageSense to deliver customized versions of your website for every individual customer based on their demographics, local weather, browsing history, and more.

Grow your business through customer feedback

Run polls on your website using PageSense to understand what your customers think about your products/services and what needs improvement on your site.

Send timely updates that customers love

Use web push notifications in PageSense to schedule and notify your customers about an upcoming flash sale, product releases, promotional coupons, and a lot more that can spark conversions on your website.

Advertise your business to website visitors

Use pop-ups in PageSense to instantly grab the attention of visitors by showing attractive signup offers, coupon code discounts, or email newsletters that can eventually convert them into subscribers.

Access more advanced settings in PageSense

Use PageSense's advanced features like creating mutually exclusive groups, enabling cross-domain tracking, configuring customized project JS, and more to get deeper insights about your website.

Try easy-to-use extensions

Download the PageSense extension app available for your web browser with a few clicks and start collecting all of your required website metrics in real time.

Discover your favorite integrations with PageSense

Get a deeper look at your website's data by seamlessly integrating PageSense with a host of popular third-party apps like Google Analytics, Mixpanel, Intercom, and more.

Quick Links	Workflow Automation	Data Collection
Web Forms	Enterprise	Begin Data Collection
Interactive Forms	Workplace	Data Collection App
Offline Forms	Customer Service	Accessible Forms
Digital Forms	Marketing	Forms for Small Business
HTML Forms	Education	Forms for Enterprise
Contact Forms	E-commerce	Forms for any business
Lead Generation Forms	Healthcare	Forms for Startups
Wordpress Forms	Customer onboarding	Order Forms for Small Business
No Code Forms	Construction	RSVP tool for holidays
Free Forms	Travel
Prefill Forms	Non-Profit
Intake Forms	Legal
Form Designer	HR
Card Forms	Food
Assign Forms	Photography
Translate Forms	Real Estate
Electronic Forms
Notification Emails for Forms	Alternatives
Holiday Forms	Google Forms alternative
Form to PDF	Jotform alternative
Email Forms CRM Forms Embeddable Forms Drag and Drop form builder HIPAA Forms Encrypted Forms

New to Zoho CRM Plus?

Deliver unforgettable customer experiences

ACCESS ZOHO CRM PLUS

New to Zoho CRM Plus?

Deliver unforgettable customer experiences

ACCESS ZOHO CRM PLUS

New to Zoho Marketing Plus?

Everything you need to run your marketing

ACCESS ZOHO MARKETING PLUS

New to Zoho Marketing Plus?

Everything you need to run your marketing

ACCESS ZOHO MARKETING PLUS

Zoho Pagesense Resources

You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.

New to Zoho Survey?

Access Zoho Survey

Latest Feature Updates

Explore our Pricing & Plans

Zoho Survey Resources

Android Mobile Surveys

Insurvey Blogs

Insight Blogs

Tips & Tricks

New to Zoho Social?

Manage your brands on social media

Access Zoho Social

Zoho Desk Resources

Desk Community Learning Series
Digest
Functions
Meetups
Kbase
Resources
Glossary
Desk Marketplace
MVP Corner
Word of the Day

Zoho Sheet Resources

New to Zoho Forms?

Latest Feature Updates

Explore our Pricing & Plans

Zoho Forms Resources

Secure your business

communication with Zoho Mail

Get started

Mail on the move with

Zoho Mail mobile application

Go Mobile

Stay on top of your schedule

at all times

Get started

Carry your calendar with you

Anytime, anywhere

Get started

Latest Announcements

New to Zoho Sign?

Zoho Sign Resources

Sign, Paperless!

Sign and send business documents on the go!

Get Started Now

Zoho SalesIQ Resources

New to Zoho TeamInbox?

Zoho TeamInbox Resources

Connect with us

New to Zoho ZeptoMail?

LEARN MORE

Latest Feature Updates

Zoho DataPrep Resources

New to Zoho DataPrep?

Access Zoho DataPrep

Zoho DataPrep Demo

Get a personalized demo or POC

Design. Discuss. Deliver.

Create visually engaging stories with Zoho Show.

Get Started Now

New to Zoho Workerly?

Access Zoho Workerly

New to Zoho Recruit?

Access Zoho Recruit

New to Zoho CRM?

Signup Now

Access Zoho CRM

Latest Feature Updates

New to Zoho Projects?

Access Zoho Projects

New to Zoho Sprints?

Access Zoho Sprints

New to Zoho Assist?

Access Zoho Assist

New to Bigin?

Signup Now

Access Bigin

Latest Feature Updates

Related Articles
Create and launch a Split URL test in PageSense
Use Split URL testing when you want to test multiple versions of your web page hosted on different URLs. This type of test is considered best when you notice several deficiencies in your existing page that might affect conversion. In such cases, it's ...
Filter and segment your A/B and Split URL test reports
All your A/B test and/or Split URL test reports will provide consolidated insights into your visitors' behavior. Dig deeper to find segment-specific test reports that can help you make tailored marketing decisions for specific audience segments and ...
A/B and Split URL test
1. Why does the original version load very briefly before the variation? This "flicker" happens if you have installed the asynchronous code snippet on your page. To avoid this, install the synchronous code snippet on your page. Doing so might have a ...
Create and launch an A/B test in PageSense
A/B testing helps you analyze and observe how one version of a web page performs alongside another in front of your audience. On Zoho PageSense, you can quickly create, edit, and launch different versions of your web page, and test which one version ...
Understand your funnel reports
Funnel Metrics Tab The metrics monitored in the funnel analysis experiment are: Visitors: Number of unique users coming to your web page. Conversion: Number of unique instances of the visitor fulfilling the condition to achieve the goal. Conversion ...