Understand your A/B and Split URL test results
The A/B (and Split URL) test reports in PageSense give you both a quick glance at the overall success measure of your experiment, as well as in-depth visitor metrics for each variation, including the conversion rate, difference interval, significance, and others. This data-rich information report helps you instantly analyze the performance of different elements on your web page, and determine which variation page performs better among your audiences.
To access your A/B (or Split URL) test reports in PageSense:
Open the A/B (or Split URL) experiment and click the REPORTS tab on the top bar. You will see three different tabs under the hood: Overview, Detailed analysis, and Heatmap.
Overview tab
Summary
Leaderboard
A/B test result is always based around a goal. The Leaderboard section shows you the list of all goals attached to your experiment, with a star marking the primary goal. Each goal has a table listing the following information:
Goal result: Indicates the winning and leading variant of a goal based on the conversion rates calculated for your original and variation pages of your A/B test.
Winning variant: The variation that performs the best with the maximum conversion rate, and reaches the significance level and sample size for a given conversion goal in your A/B test.
Leading variant: The variation that is performing better than its competitor variations for a given conversion goal in your A/B test.
No variant is leading currently: The variation performs similar to the original version for a given conversion goal.
{variant names} are leading the variants: Appears when more than one variation is performing better or leading the competitors for a given conversion goal at the same time.
Rank: Indicates the rank of the variants based on their performance to a corresponding goal.
Variant: Indicates the list of all the variants available in your experiment page, including the original.
Visitors: Indicates the total number of unique visitors to the corresponding variation.
Conversions: Indicates the number of unique instances of the visitor fulfilling the desired action for a given goal. A conversion can refer to any action that you want the user to take. This can include any action performed on your web page, from clicking on a button to making a purchase and becoming a customer.
Conversion rate: The conversion rate is the number of conversions divided by the total number of visitors. For example, if your variation page receives 200 visitors in a month and has 50 purchases, the conversion rate would be 50 divided by 200, or 25%.
The conversion rate is represented by percentage value and can have any positive.
Warnings and notification messages
The notification banner on the top displays the status of your experiment and any warnings you must be aware of related to the issues with data collection in a variation or with conversion tracking.
"We recommend you to run this experiment till it reaches a conclusive result."
The cause: This may happen when the desired visitor count has not been reached to conclude the results. In this case, you need to wait longer to reach the desired visitor count, have a better conversion rate, and reach the required significance level you chose."The result is inconclusive as the performance of the original and variation is similar."
The cause: This could be because both the original and the variation version of your webpage perform similarly and hence there is no better advantage of publishing a new variation over the original. In this case, we recommend that you end the test and create a new A/B experiment for your page.
Detailed analysis tab
Understanding the graph
Time-based graph: This graph illustrates multiple spikes of data distributed across a chosen time frame (along the x-axis). It is time based because the data values or points are calculated over regular time intervals, and will be considered independent of each other.
For example, let's say you introduced a Black Friday Sale (12th of April) on your website. You see 100 new visitors coming to your variation page on this specific day, out of which 40 got converted, with a conversion rate of 40% (which is 20% higher than the previous day, 20th of April). In this case, using the time-based view can help you compare the visitor counts and conversion rates of your data on the days before and after the sale. This could later be used to interpret if the introduced change on your web page is influencing your conversion rate over time.In this type of graph pattern (shown below), you can also occasionally observe conversions drop to a zero value on any given day within the time range. This might be quite normal, due to lower traffic or a lower conversion rate. One possible implication of such a pattern on your test with a high traffic scope may be a signal to double-check the quality of the test setup, as something might have broken. This can in turn be a QA check during a live test.
Cumulative graph: This graph illustrates flattened data distributed across a chosen time frame (along the x-axis). It is cumulative because the data values or points are calculated as a sum over time, like computing the sum of the first point, then the first two points, then the first three, and so on.
For example, let's say you started an A/B experiment on the 1st of April and you want to check the conversion rate on the 15th of April to see how your experiment is performing every two weeks. In this case, using the cumulative graph provides you with a quick inference on the average performance of your variation's data, summed up until mid-April.In this type of graph pattern (shown below), you can also see that the effects of the originaland he variations will fluctuate more in the beginning than later in a test. One possible implication of such a pattern might be to delay a test from being stopped prematurely until greater horizontal stability is first visualized.
Further, the graph allows you to filter your goal metrics between different variations against individual performance indicators such as Conversion Rate, Conversions, Visitors, Improvement, and Significance from the dropdown.View Forecast: Enabling the Forecast button helps you to view predictions or changes in the conversion rate (and improvement rate) that each variation might take over the chosen time range. Forecast is an estimated value of change over a future time horizon. This can be used to examine the performance of your A/B results, and predict your success in the long term.
Understanding the table
Variant: Indicates the list of all the variants available in your experiment page, including the original.
Visitors: Indicates the total number of unique visitors to the corresponding variation.
Conversions: Indicates the number of unique instances of the visitor fulfilling the desired action for a given goal. A conversion can refer to any desired action that you want the user to take. This can include any action performed on your web page, from clicking on a button to making a purchase and becoming a customer.
Conversion rate: The conversion rate is the number of conversions divided by the total number of visitors.
Difference Interval: When communicating the results, it's not only a good idea to present the observed difference in the conversion rate value for the original and variation, but also the range between which the conversion rate of your original and the variation page can actually lie. This possible range of values is called the difference interval, and is plotted on a number line scale.
On the number scale, the upper limit is marked by the maximum possible range of conversion rate, and the lower limit is marked by the minimum possible range of conversion rate between all the variations. The different shades on the scale indicate the following:Grey shade area: Indicates the experiment is inconclusive, or needs more visitors to declare a valid result.
Green shade area: Indicates a winning variation.
Red shade area: Indicates a losing variation.
For example, let's say you test your variation page with 7626 visitors, and get 1722 conversions with a conversion rate of 22.58%. For this, you see a difference interval between 21.64 % - 23.52 %, which means that your conversion rate value could lie anywhere between this difference based on the new visitors and conversions obtained on your variation page.Note:One thing to watch out for in difference interval is the overlap of the conversion rate from the Original and Variation 1 of your A/B test. For example, suppose that Original has a confidence interval of 10-20% for conversion rates, and Variation 1 has a confidence interval of 15-25%. Notice that the overlap of the two confidence intervals is 5%, and it is located in the range between 15-20%. In this case, it's very difficult to be sure that the variation tested in B is actually a significant improvement. This is why we will not declare winner if there is overlap.
- Improvement: Improvement denotes the relative difference between the conversion rate of the original and variation version of your web page. The improvement percentage can be a positive or negative value.For example, if the conversion rate of variation is 30%, and the conversion rate of the original is 15%, then the improvement is (30- 15)/15*100 = 100 % points. In this case, there is a 100% increase in the conversion rate for the variation page.
- Significance (for the Frequentist approach): For any A/B test, the most important concept for interpreting your results is statistical significance. Statistical significance tells you whether a variation is outperforming or under performing the baseline, at whatever level of significance level you have chosen. This significance level is decided based on the type of frequentist method you have applied to evaluate your results.Significance is defined as the probability that the difference between your variation A and B page's conversion rate is due to real changes in your visitor's behavior and not based on some random action or choice. Simply put, if a statistics has a higher significance, then the results are considered more reliable.For example, if your A/B test shows a significance level of 95% this means that if you determine a winner, you can be 95% confident that the observed results are real, and not an error caused by randomness. It also means that there is a 5% chance that you could be wrong. Learn the importance of statistical significance in declaring a winning variation in your A/B experiment.
- Probability to win (the Bayesian approach): This is defined as the probability of a variation to outperform the original version in your A/B test. Probability to win tells you which variation is better and by how much in percentage. This is the most actionable metric used in the Bayesian method to declare the winner of your A/B tests. The winner of this approach will be declared when these three primary rules are met:
1. Has achieved a unique visitor count of 100 or above for the variations.2. Has gathered a unique conversion of 50 or above for the goals setup.3. Has an average of minimal loss value.
Once the primary rules are satisfied, in the Bayesian approach, we declare winners based on two other parameters:
1. To declare a variation from the Experiment as the winner, the primary goal value should meet the visitor count, conversion counts, and expected loss value.
2. To declare a variation from the Goal as the winner, we will take in visitor count, conversion count and expected loss value.
Heatmap tab
By activating a heatmap in your A/B (or Split URL) experiment, you can visualize and identify where your visitors are clicking, scrolling, and looking in your original and variation pages. This report also provides you additional insights on why a particular variation did not perform as expected, and what needs to be changed to address issues.
What heatmap can tell you about your A/B (or Split URL) test
What scroll maps can tell you about your A/B (or Split URL) test
What attention map can tell you about your A/B (or Split URL) test
These reports use a colored overlay to reflect the level of interaction on your pages. A red area is where there is the most activity, while blue indicates less amount of activity. Based on these attention map metrics, you can track which section attracts visitors’ gaze the most on your variations, then place all important content and images in the most attention-grabbing sections of the page for testing.
By combining the knowledge from heatmaps with A/B testing reports, you can increase the conversions on your variation pages, and achieve a higher statistical significance for declaring a winner.
Zoho CRM Training Programs
Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.
Zoho DataPrep Personalized Demo
If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.
- Zoho Workerly Home
- Forums
- Connect With Us:
- Zoho Recruit Home
- Forums
- Connect With Us:
You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.
Zoho Sheet Resources
Zoho Forms Resources
Zoho Sign Resources
Sign, Paperless!
Zoho TeamInbox Resources
Zoho DataPrep Resources
Design. Discuss. Deliver.
Related Articles
Filter and segment your A/B and Split URL test reports
All your A/B test and/or Split URL test reports will provide consolidated insights into your visitors' behavior. Dig deeper to find segment-specific test reports that can help you make tailored marketing decisions for specific audience segments and ...Create and launch a Split URL test in PageSense
Use Split URL testing when you want to test multiple versions of your web page hosted on different URLs. This type of test is considered best when you notice several deficiencies in your existing page that might affect conversion. In such cases, it's ...A/B and Split URL test
1. Why does the original version load very briefly before the variation? This "flicker" happens if you have installed the asynchronous code snippet on your page. To avoid this, install the synchronous code snippet on your page. Doing so might have a ...Create and launch an A/B test in PageSense
A/B testing helps you analyze and observe how one version of a web page performs alongside another in front of your audience. On Zoho PageSense, you can quickly create, edit, and launch different versions of your web page, and test which one version ...Understand your funnel reports
Funnel Metrics Tab The metrics monitored in the funnel analysis experiment are: Visitors: Number of unique users coming to your web page. Conversion: Number of unique instances of the visitor fulfilling the condition to achieve the goal. Conversion ...