Analyze and interpret A/B and Split URL test report in Zoho PageSense | Help Guide

Understand your A/B and Split URL test results

The A/B (and Split URL) test reports in PageSense give you both a quick glance at the overall success measure of your experiment, as well as in-depth visitor metrics for each variation, including the conversion rate, difference interval, significance, and others. This data-rich information report helps you instantly analyze the performance of different elements on your web page, and determine which variation page performs better among your audiences.

To access your A/B (or Split URL) test reports in PageSense:
Open the A/B (or Split URL) experiment and click the REPORTS tab on the top bar. You will see three different tabs under the hood: Overview, Detailed analysis, and Heatmap.


Overview tab

Here you can see a summary and a leaderboard table for each goal metric attached to your experiment. You can use these numbers to check which conversion goal has performed the best or is leading with respect to  individual variations in your A/B test

Summary

This sections presents you with all the major stats of your A/B (or Split URL test) such as the total visitors count, number of variants, number of goals, the experiment start day, the day it was paused, and the active duration of all the variants, including the original.

Leaderboard

A/B test result is always based around a goal. The Leaderboard section shows you the list of all goals attached to your experiment, with a star marking the primary goal. Each goal has a table listing the following information:

  • Goal result: Indicates the winning and leading variant of a goal based on the conversion rates calculated for your original and variation pages of your A/B test.

    • Winning variant: The variation that performs the best with the maximum conversion rate, and reaches the significance level and sample size for a given conversion goal in your A/B test.

    • Leading variant: The variation that is performing better than its competitor variations for a given conversion goal in your A/B test.  

    • No variant is leading currently: The variation performs similar to the original version for a given conversion goal.

    • {variant names} are leading the variants: Appears when more than one variation is performing better or leading the competitors for a given conversion goal at the same time.   

  • Rank: Indicates the rank of the variants based on their performance to a corresponding goal.

  • Variant: Indicates the list of all the variants available in your experiment page, including the original.

  • Visitors: Indicates the total number of unique visitors to the corresponding variation.

  • Conversions: Indicates the number of unique instances of the visitor fulfilling the desired action for a given goal. A conversion can refer to any action that you want the user to take. This can include any action performed on your web page, from clicking on a button to making a purchase and becoming a customer.

  • Conversion rate: The conversion rate is the number of conversions divided by the total number of visitors. For example, if your variation page receives 200 visitors in a month and has 50 purchases, the conversion rate would be 50 divided by 200, or 25%.

     The conversion rate is represented by percentage value and can have any positive.


Warnings and notification messages

The notification banner on the top displays the status of your experiment and any warnings you must be aware of related to the issues with data collection in a variation or with conversion tracking. 

The following is a list of possible errors or warning messages that you could see in your Reports tab:
  • "We recommend you to run this experiment till it reaches a conclusive result."
    The cause: This may happen when the desired visitor count has not been reached to conclude the results. In this case, you need to wait longer to reach the desired visitor count, have a better conversion rate, and reach the required significance level you chose. 

  • "The result is inconclusive as the performance of the original and variation is similar."
    The cause: This could be because both the original and the variation version of your webpage perform similarly and hence there is no better advantage of publishing a new variation over the original. In this case, we recommend that you end the test and create a new A/B experiment for your page.

Detailed analysis tab

This section shows the performance trend of each goal on a graph. 

Understanding the graph

This graph compares the performance and trends of each variation against a chosen baseline over time. By default, the original is set as the baseline for comparing your goal's data. You can change this baseline to any other variation using the Baseline and Compare with dropdown.

You can view your graph in either of the following ways:
  • Time-based graph: This graph illustrates multiple spikes of data distributed across a chosen time frame (along the x-axis). It is time based because the data values or points are calculated over regular time intervals, and will be considered independent of each other.

    For example, let's say you introduced a Black Friday Sale (12th of April) on your website. You see 100 new visitors coming to your variation page on this specific day, out of which 40 got converted, with a conversion rate of 40% (which is 20% higher than the previous day, 20th of April). In this case, using the time-based view can help you compare the visitor counts and conversion rates of your data on the days before and after the sale. This could later be used to interpret if the introduced change on your web page is influencing your conversion rate over time.
    In this type of graph pattern (shown below), you can also occasionally observe conversions drop to a zero value on any given day within the time range. This might be quite normal, due to lower traffic or a lower conversion rate. One possible implication of such a pattern on your test with a high traffic scope may be a signal to double-check the quality of the test setup, as something might have broken. This can in turn be a QA check during a live test. 


  • Cumulative graph: This graph illustrates flattened data distributed across a chosen time frame (along the x-axis). It is cumulative because the data values or points are calculated as a sum over time, like computing the sum of the first point, then the first two points, then the first three, and so on.  

    For example, let's say you started an A/B experiment on the 1st of April and you want to check the conversion rate on the 15th of April to see how your experiment is performing every two weeks. In this case, using the cumulative graph provides you with a quick inference on the average performance of your variation's data, summed up until mid-April.
    In this type of graph pattern (shown below), you can also see that the effects of the originaland he variations will fluctuate more in the beginning than later in a test. One possible implication of such a pattern might be to delay a test from being stopped prematurely until greater horizontal stability is first visualized. 

    Further, the graph allows you to filter your goal metrics between different variations against individual performance indicators such as Conversion Rate, Conversions, Visitors, Improvement, and Significance from the dropdown.

  • View Forecast: Enabling the Forecast button helps you to view predictions or changes in the conversion rate (and improvement rate) that each variation might take over the chosen time range. Forecast is an estimated value of change over a future time horizon. This can be used to examine the performance of your A/B results, and predict your success in the long term.


Understanding the table

This section consists of tables for each goal, with in-depth and detailed statistics that contain metrics such as Conversion rate, Conversions, Difference interval, Improvement, and Significance.
  • Variant: Indicates the list of all the variants available in your experiment page, including the original.

  • Visitors: Indicates the total number of unique visitors to the corresponding variation.

  • Conversions: Indicates the number of unique instances of the visitor fulfilling the desired action for a given goal. A conversion can refer to any desired action that you want the user to take. This can include any action performed on your web page, from clicking on a button to making a purchase and becoming a customer.

  • Conversion rate: The conversion rate is the number of conversions divided by the total number of visitors.

For example, if your variation page receives 200 visitors in a month and has 50 purchases, the conversion rate would be 50 divided by 200, or 25%. The conversion rate is represented by percentage value, and can have any positive or negative value.
  • Difference Interval: When communicating the results, it's not only a good idea to present the observed difference in the conversion rate value for the original and variation, but also the range between which the conversion rate of your original and the variation page can actually lie. This possible range of values is called the difference interval, and is plotted on a number line scale. 

    On the number scale, the upper limit is marked by the maximum possible range of conversion rate, and the lower limit is marked by the minimum possible range of conversion rate between all the variations. The different shades on the scale indicate the following:  

    • Grey shade area: Indicates the experiment is inconclusive, or needs more visitors to declare a valid result.

    • Green shade area: Indicates a winning variation.

    • Red shade area: Indicates a losing variation.

For example, let's say you test your variation page with 7626 visitors, and get 1722 conversions with a conversion rate of 22.58%.  For this, you see a difference interval between 21.64 % - 23.52 %, which means that your conversion rate value could lie anywhere between this difference based on the new visitors and conversions obtained on your variation page.
Note:
One thing to watch out for in difference interval is the overlap of the conversion rate from the Original and Variation 1 of your A/B test. For example, suppose that Original has a confidence interval of 10-20% for conversion rates, and Variation 1 has a confidence interval of 15-25%. Notice that the overlap of the two confidence intervals is 5%, and it is located in the range between 15-20%. In this case, it's very difficult to be sure that the variation tested in B is actually a significant improvement. This is why we will not declare winner if there is overlap. 
  1. Improvement: Improvement denotes the relative difference between the conversion rate of the original and variation version of your web page. The improvement percentage can be a positive or negative value. 
    For example, if the conversion rate of variation is 30%, and the conversion rate of the original is 15%, then the improvement is (30- 15)/15*100 = 100 % points. In this case, there is a 100% increase in the conversion rate for the variation page.
  2. Significance (for the Frequentist approach): For any A/B test, the most important concept for interpreting your results is statistical significance. Statistical significance tells you whether a variation is outperforming or under performing the baseline, at whatever level of significance level you have chosen. This significance level is decided based on the type of frequentist method you have applied to evaluate your results.
    Significance is defined as the probability that the difference between your variation A and B page's conversion rate is due to real changes in your visitor's behavior and not based on some random action or choice. Simply put, if a statistics has a higher significance, then the results are considered more reliable.
    For example, if your A/B test shows a significance level of 95% this means that if you determine a winner, you can be 95% confident that the observed results are real, and not an error caused by randomness. It also means that there is a 5% chance that you could be wrong. Learn the importance of statistical significance in declaring a winning variation in your A/B experiment.

  3. Probability to win (the Bayesian approach): This is defined as the probability of a variation to outperform the original version in your A/B test. Probability to win tells you which variation is better and by how much in percentage. This is the most actionable metric used in the Bayesian method to declare the winner of your A/B tests. The winner of this approach will be declared when these three primary rules are met:
    1. Has achieved a unique visitor count of 100 or above for the variations.
    2. Has gathered a unique conversion of 50 or above for the goals setup.
    3. Has an average of minimal loss value.

    Once the primary rules are satisfied, in the Bayesian approach, we declare winners based on two other parameters:
    1. To declare a variation from the Experiment as the winner, the primary goal value should meet the visitor count, conversion counts, and expected loss value.
    2. To declare a variation from the Goal as the winner, we will take in visitor count, conversion count and expected loss value.

Heatmap tab

By activating a heatmap in your A/B (or Split URL) experiment, you can visualize and identify where your visitors are clicking, scrolling, and looking in your original and variation pages. This report also provides you additional insights on why a particular variation did not perform as expected, and what needs to be changed to address issues.

What heatmap can tell you about your A/B (or Split URL) test

Click the HEATMAP tab to compare the visitor data between the original and variation to discover how elements perform differently on each page. From important CTA elements, links, or videos that attract less clicks, to other unimportant elements distracting visitors from getting converted on each page, heatmap reveals where visitors are clicking on your web pages, and what opportunities are available to turn your A/B (or Split URL) test into the winner you were hoping for. 

These reports use a colored hotspot to show which element on the page gets more clicks, or vice versa. Blue means fewer clicks, warmer reds indicate more clicks, and the most clicks are bright yellow spots. Based on these heatmap metrics, you can quickly see which areas of the page get a lot of action, and which do not. Based on this, you can make the desired position change in your variation page and run your next A/B test.


What scroll maps can tell you about your A/B (or Split URL) test

Click SCROLLMAP to compare and discover which part of the page (either original or variations) is most effective at keeping users engaged and interested. This is especially important to provide insight into what percentage of your visitors are scrolling far enough down the page to see your call to action, and to analyze why visitors are bouncing back even before they reach half of your web page length. If a high proportion of your visitors are not scrolling down far enough to see it, they certainly will not be clicking on it. 

These reports use a color-coded gradient to show where users spend the most time. The warmer an area appears, the more attention it gets from your visitors; it fades out gradually with reduced visitor activity. Based on these scroll map metrics, you can try creating variation pages of varying lengths, place the most critical elements of the web page on sections where visitors spend the most time, and see how it improves the conversion.


What attention map can tell you about your A/B (or Split URL) test

Click ATTENTIONMAP to compare and see how long your users spend time on your original and variation pages. This metrics is more useful to see if the key pieces of information, such as text and visuals, are visible to most users, and which section of the page your visitors are more enraged with, by moving their mouse trying to read information on your page.

These reports use a colored overlay to reflect the level of interaction on your pages. A red area is where there is the most activity, while blue indicates less amount of activity. Based on these attention map metrics, you can track which section attracts visitors’ gaze the most on your variations, then place all important content and images in the most attention-grabbing sections of the page for testing.



By combining the knowledge from heatmaps with A/B testing reports, you can increase the conversions on your variation pages, and achieve a higher statistical significance for declaring a winner.





    Zoho CRM Training Programs

    Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.

    Zoho CRM Training
      Redefine the way you work
      with Zoho Workplace

        Zoho DataPrep Personalized Demo

        If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.

        Zoho CRM Training

          Create, share, and deliver

          beautiful slides from anywhere.

          Get Started Now


            Zoho Sign now offers specialized one-on-one training for both administrators and developers.

            BOOK A SESSION









                                      You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.




                                          Manage your brands on social media

                                            Zoho Desk Resources

                                            • Desk Community Learning Series


                                            • Digest


                                            • Functions


                                            • Meetups


                                            • Kbase


                                            • Resources


                                            • Glossary


                                            • Desk Marketplace


                                            • MVP Corner


                                            • Word of the Day


                                              Zoho Marketing Automation

                                                Zoho Sheet Resources

                                                 

                                                    Zoho Forms Resources


                                                      Secure your business
                                                      communication with Zoho Mail


                                                      Mail on the move with
                                                      Zoho Mail mobile application

                                                        Stay on top of your schedule
                                                        at all times


                                                        Carry your calendar with you
                                                        Anytime, anywhere




                                                              Zoho Sign Resources

                                                                Sign, Paperless!

                                                                Sign and send business documents on the go!

                                                                Get Started Now




                                                                        Zoho TeamInbox Resources



                                                                                Zoho DataPrep Resources



                                                                                  Zoho DataPrep Demo

                                                                                  Get a personalized demo or POC

                                                                                  REGISTER NOW


                                                                                    Design. Discuss. Deliver.

                                                                                    Create visually engaging stories with Zoho Show.

                                                                                    Get Started Now









                                                                                                        • Related Articles

                                                                                                        • Create and launch a Split URL test in PageSense

                                                                                                          Use Split URL testing when you want to test multiple versions of your web page hosted on different URLs. This type of test is considered best when you notice several deficiencies in your existing page that might affect conversion. In such cases, it's ...
                                                                                                        • Filter and segment your A/B and Split URL test reports

                                                                                                          All your A/B test and/or Split URL test reports will provide consolidated insights into your visitors' behavior. Dig deeper to find segment-specific test reports that can help you make tailored marketing decisions for specific audience segments and ...
                                                                                                        • A/B and Split URL test

                                                                                                          1. Why does the original version load very briefly before the variation? This "flicker" happens if you have installed the asynchronous code snippet on your page. To avoid this, install the synchronous code snippet on your page. Doing so might have a ...
                                                                                                        • Create and launch an A/B test in PageSense

                                                                                                          A/B testing helps you analyze and observe how one version of a web page performs alongside another in front of your audience. On Zoho PageSense, you can quickly create, edit, and launch different versions of your web page, and test which one version ...
                                                                                                        • Understand your funnel reports

                                                                                                          Funnel Metrics Tab The metrics monitored in the funnel analysis experiment are: Visitors: Number of unique users coming to your web page.  Conversion: Number of unique instances of the visitor fulfilling the condition to achieve the goal. Conversion ...
                                                                                                          Wherever you are is as good as
                                                                                                          your workplace

                                                                                                            Resources

                                                                                                            Videos

                                                                                                            Watch comprehensive videos on features and other important topics that will help you master Zoho CRM.



                                                                                                            eBooks

                                                                                                            Download free eBooks and access a range of topics to get deeper insight on successfully using Zoho CRM.



                                                                                                            Webinars

                                                                                                            Sign up for our webinars and learn the Zoho CRM basics, from customization to sales force automation and more.



                                                                                                            CRM Tips

                                                                                                            Make the most of Zoho CRM with these useful tips.



                                                                                                              Zoho Show Resources