Analyze and interpret A/B and Split URL test report in Zoho PageSense | Help Guide

Understand your A/B test results

Zoho PageSense provides reports that help you compare the performance of your original webpage and its variations. You can review overall experiment metrics, identify the leading variation for each goal, and analyze visitor interactions to understand how your changes affect conversions.

To access your A/B test reports:
  1. Open your A/B test experiment.
  2. Click the Reports tab in the top navigation bar.
You can access three sections: Overview, Detailed Analysis, and Heatmap.



Overview tab

The Overview section provides a summary of your experiment and a leaderboard for each conversion goal. Use these reports to compare the performance of your variations and identify which one is leading for each goal.


Experiment summary

The summary displays the following details about your experiment:
  • Visitors: The total number of visitors included in the experiment.
  • Variants: The total number of versions being tested, including the original webpage.
  • Goals: The total number of conversion goals configured for the experiment.
  • Started on: The date on which the experiment was launched.
  • Paused on: The date on which the experiment was paused. This is displayed when the experiment has been paused.
  • Active Duration: The total time the experiment remained active.

Leaderboard

The Leaderboard displays the performance of each conversion goal configured in your experiment. Each goal has its own leaderboard, allowing you to compare how the original and variations perform against that goal.
If you have configured a primary goal, it is marked with a star.
The leaderboard includes a leading variant summary and a table containing the performance metrics of each variant.



Leading variant summary :
The summary highlights the variant currently performing best for the selected goal. It displays:
  • Leading variant: The name of the variant currently leading for the selected goal.
  • Conversion rate: The percentage of visitors who completed the goal.
  • Visitors: The number of visitors assigned to the leading variant.
  • Conversions: The number of visitors who completed the selected goal.
Leaderboard table : 
The table displays the following metrics for each variant:
  • Rank: The position of the variant based on its performance for the selected goal.
  • Variant: The name of the original webpage or variation included in the experiment.
  • Visitors: The number of visitors assigned to the corresponding variant.
  • Impressions: The number of times the tracked goal element was displayed to visitors.
  • Visibility rate: The percentage of visitors who viewed the tracked goal element.
  • Conversions: The number of visitors who completed the selected goal.
  • Conversion rate: The percentage of visitors who completed the goal out of the total visitors assigned to the variant.

Reports toolbar

The Reports page also provides options to manage how you view and use your experiment data.


  • Segment Audience: Filter the report to analyze the performance of a specific audience segment.
  • Export: Export the experiment report for further analysis or sharing.
  • Share: Share the report with others.
  • Refresh: Refresh the report to view the latest available data.

Warnings and notification messages

The notification banner on the top displays the status of your experiment and any warnings you must be aware of related to the issues with data collection in a variation or with conversion tracking. 

Notes
The following is a list of possible errors or warning messages that you could see in your Reports tab:
  • "We recommend you to run this experiment till it reaches a conclusive result."
    The cause: This may happen when the desired visitor count has not been reached to conclude the results. In this case, you need to wait longer to reach the desired visitor count, have a better conversion rate, and reach the required significance level you chose. 

  • "The result is inconclusive as the performance of the original and variation is similar."
    The cause: This could be because both the original and the variation version of your webpage perform similarly and hence there is no better advantage of publishing a new variation over the original. In this case, we recommend that you end the test and create a new A/B experiment for your page.

Detailed analysis tab

This section shows the performance trend of each goal on a graph. 

Understanding the graph

This graph compares the performance and trends of each variation against a chosen baseline over time. By default, the original is set as the baseline for comparing your goal's data. You can change this baseline to any other variation using the Baseline and Compare with dropdown.

You can view your graph in either of the following ways:
  • Time-based graph: This graph illustrates multiple spikes of data distributed across a chosen time frame (along the x-axis). It is time based because the data values or points are calculated over regular time intervals, and will be considered independent of each other.

    IdeaFor example, let's say you introduced a Black Friday Sale (12th of April) on your website. You see 100 new visitors coming to your variation page on this specific day, out of which 40 got converted, with a conversion rate of 40% (which is 20% higher than the previous day, 20th of April). In this case, using the time-based view can help you compare the visitor counts and conversion rates of your data on the days before and after the sale. This could later be used to interpret if the introduced change on your web page is influencing your conversion rate over time.
    In this type of graph pattern (shown below), you can also occasionally observe conversions drop to a zero value on any given day within the time range. This might be quite normal, due to lower traffic or a lower conversion rate. One possible implication of such a pattern on your test with a high traffic scope may be a signal to double-check the quality of the test setup, as something might have broken. This can in turn be a QA check during a live test. 


  • Cumulative graph: This graph illustrates flattened data distributed across a chosen time frame (along the x-axis). It is cumulative because the data values or points are calculated as a sum over time, like computing the sum of the first point, then the first two points, then the first three, and so on.  

    IdeaFor example, let's say you started an A/B experiment on the 1st of April and you want to check the conversion rate on the 15th of April to see how your experiment is performing every two weeks. In this case, using the cumulative graph provides you with a quick inference on the average performance of your variation's data, summed up until mid-April.
    In this type of graph pattern (shown below), you can also see that the effects of the original and he variations will fluctuate more in the beginning than later in a test. One possible implication of such a pattern might be to delay a test from being stopped prematurely until greater horizontal stability is first visualized. 

    Further, the graph allows you to filter your goal metrics between different variations against individual performance indicators such as Conversion Rate, Conversions, Visitors, Improvement, and Significance from the dropdown.

  • View Forecast: Enabling the Forecast button helps you to view predictions or changes in the conversion rate (and improvement rate) that each variation might take over the chosen time range. Forecast is an estimated value of change over a future time horizon. This can be used to examine the performance of your A/B results, and predict your success in the long term.


Understanding the table

This section consists of tables for each goal, with in-depth and detailed statistics that contain metrics such as Conversion rate, Conversions, Difference interval, Improvement, and Significance.
  • Variant: Indicates the list of all the variants available in your experiment page, including the original.

  • Visitors: Indicates the total number of unique visitors to the corresponding variation.

  • Conversions: Indicates the number of unique instances of the visitor fulfilling the desired action for a given goal. A conversion can refer to any desired action that you want the user to take. This can include any action performed on your web page, from clicking on a button to making a purchase and becoming a customer.

  • Conversion rate: The conversion rate is the number of conversions divided by the total number of visitors.

IdeaFor example, if your variation page receives 200 visitors in a month and has 50 purchases, the conversion rate would be 50 divided by 200, or 25%. The conversion rate is represented by percentage value, and can have any positive or negative value.
  • Difference Interval: When communicating the results, it's not only a good idea to present the observed difference in the conversion rate value for the original and variation, but also the range between which the conversion rate of your original and the variation page can actually lie. This possible range of values is called the difference interval, and is plotted on a number line scale. 

    On the number scale, the upper limit is marked by the maximum possible range of conversion rate, and the lower limit is marked by the minimum possible range of conversion rate between all the variations. The different shades on the scale indicate the following:  

    • Grey shade area: Indicates the experiment is inconclusive, or needs more visitors to declare a valid result.

    • Green shade area: Indicates a winning variation.

    • Red shade area: Indicates a losing variation.

For example, let's say you test your variation page with 7626 visitors, and get 1722 conversions with a conversion rate of 22.58%.  For this, you see a difference interval between 21.64 % - 23.52 %, which means that your conversion rate value could lie anywhere between this difference based on the new visitors and conversions obtained on your variation page.
Notes
Note:
One thing to watch out for in difference interval is the overlap of the conversion rate from the Original and Variation 1 of your A/B test. For example, suppose that Original has a confidence interval of 10-20% for conversion rates, and Variation 1 has a confidence interval of 15-25%. Notice that the overlap of the two confidence intervals is 5%, and it is located in the range between 15-20%. In this case, it's very difficult to be sure that the variation tested in B is actually a significant improvement. This is why we will not declare winner if there is overlap. 
  1. Improvement: Improvement denotes the relative difference between the conversion rate of the original and variation version of your web page. The improvement percentage can be a positive or negative value. 
    IdeaFor example, if the conversion rate of variation is 30%, and the conversion rate of the original is 15%, then the improvement is (30- 15)/15*100 = 100 % points. In this case, there is a 100% increase in the conversion rate for the variation page.
  2. Significance (for the Frequentist approach): For any A/B test, the most important concept for interpreting your results is statistical significance. Statistical significance tells you whether a variation is outperforming or under performing the baseline, at whatever level of significance level you have chosen. This significance level is decided based on the type of frequentist method you have applied to evaluate your results.
    Significance is defined as the probability that the difference between your variation A and B page's conversion rate is due to real changes in your visitor's behavior and not based on some random action or choice. Simply put, if a statistics has a higher significance, then the results are considered more reliable.
    Idea
    For example, if your A/B test shows a significance level of 95% this means that if you determine a winner, you can be 95% confident that the observed results are real, and not an error caused by randomness. It also means that there is a 5% chance that you could be wrong. Learn the importance of statistical significance in declaring a winning variation in your A/B experiment.

  3. Probability to win (the Bayesian approach): This is defined as the probability of a variation to outperform the original version in your A/B test. Probability to win tells you which variation is better and by how much in percentage. This is the most actionable metric used in the Bayesian method to declare the winner of your A/B tests. The winner of this approach will be declared when these three primary rules are met:
    1. Has achieved a unique visitor count of 100 or above for the variations.
    2. Has gathered a unique conversion of 50 or above for the goals setup.
    3. Has an average of minimal loss value.

    Once the primary rules are satisfied, in the Bayesian approach, we declare winners based on two other parameters:
    1. To declare a variation from the Experiment as the winner, the primary goal value should meet the visitor count, conversion counts, and expected loss value.
    2. To declare a variation from the Goal as the winner, we will take in visitor count, conversion count and expected loss value.

Heatmap tab

By activating a heatmap in your A/B test experiment, you can visualize and identify where your visitors are clicking, scrolling, and looking in your original and variation pages. This report also provides you additional insights on why a particular variation did not perform as expected, and what needs to be changed to address issues.

What heatmap can tell you about your A/B test

Click the HEATMAP tab to compare the visitor data between the original and variation to discover how elements perform differently on each page. From important CTA elements, links, or videos that attract less clicks, to other unimportant elements distracting visitors from getting converted on each page, heatmap reveals where visitors are clicking on your web pages, and what opportunities are available to turn your A/B test into the winner you were hoping for. 

These reports use a colored hotspot to show which element on the page gets more clicks, or vice versa. Blue means fewer clicks, warmer reds indicate more clicks, and the most clicks are bright yellow spots. Based on these heatmap metrics, you can quickly see which areas of the page get a lot of action, and which do not. Based on this, you can make the desired position change in your variation page and run your next A/B test.


What scroll maps can tell you about your A/B  test

Click SCROLLMAP to compare and discover which part of the page (either original or variations) is most effective at keeping users engaged and interested. This is especially important to provide insight into what percentage of your visitors are scrolling far enough down the page to see your call to action, and to analyze why visitors are bouncing back even before they reach half of your web page length. If a high proportion of your visitors are not scrolling down far enough to see it, they certainly will not be clicking on it. 

These reports use a color-coded gradient to show where users spend the most time. The warmer an area appears, the more attention it gets from your visitors; it fades out gradually with reduced visitor activity. Based on these scroll map metrics, you can try creating variation pages of varying lengths, place the most critical elements of the web page on sections where visitors spend the most time, and see how it improves the conversion.



What attention map can tell you about your A/B test

Click ATTENTIONMAP to compare and see how long your users spend time on your original and variation pages. This metrics is more useful to see if the key pieces of information, such as text and visuals, are visible to most users, and which section of the page your visitors are more enraged with, by moving their mouse trying to read information on your page.

These reports use a colored overlay to reflect the level of interaction on your pages. A red area is where there is the most activity, while blue indicates less amount of activity. Based on these attention map metrics, you can track which section attracts visitors’ gaze the most on your variations, then place all important content and images in the most attention-grabbing sections of the page for testing.



By combining the knowledge from heatmaps with A/B testing reports, you can increase the conversions on your variation pages, and achieve a higher statistical significance for declaring a winner.






We’ve designed this documentation to guide you every step of the way. If you need further assistance or have any questions, don’t hesitate to contact us at support@zohopagesense.com - we’re always here to help!