The A/B (and Split URL) test reports in PageSense give you both a quick glance at the overall success measure of your experiment, as well as in-depth visitor metrics for each variation, including the conversion rate, difference interval, significance, and others. This data-rich information report helps you instantly analyze the performance of different elements on your web page, and determine which variation page performs better among your audiences.
To access your A/B (or Split URL) test reports in PageSense:
Open the A/B (or Split URL) experiment and click the REPORTS tab on the top bar. You will see three different tabs under the hood: Overview, Detailed analysis, and Heatmap.
A/B test result is always based around a goal. The Leaderboard section shows you the list of all goals attached to your experiment, with a star marking the primary goal. Each goal has a table listing the following information:
Goal result: Indicates the winning and leading variant of a goal based on the conversion rates calculated for your original and variation pages of your A/B test.
Winning variant: The variation that performs the best with the maximum conversion rate, and reaches the significance level and sample size for a given conversion goal in your A/B test.
Leading variant: The variation that is performing better than its competitor variations for a given conversion goal in your A/B test.
No variant is leading currently: The variation performs similar to the original version for a given conversion goal.
{variant names} are leading the variants: Appears when more than one variation is performing better or leading the competitors for a given conversion goal at the same time.
Rank: Indicates the rank of the variants based on their performance to a corresponding goal.
Variant: Indicates the list of all the variants available in your experiment page, including the original.
Visitors: Indicates the total number of unique visitors to the corresponding variation.
Conversions: Indicates the number of unique instances of the visitor fulfilling the desired action for a given goal. A conversion can refer to any action that you want the user to take. This can include any action performed on your web page, from clicking on a button to making a purchase and becoming a customer.
Conversion rate: The conversion rate is the number of conversions divided by the total number of visitors. For example, if your variation page receives 200 visitors in a month and has 50 purchases, the conversion rate would be 50 divided by 200, or 25%.
The notification banner on the top displays the status of your experiment and any warnings you must be aware of related to the issues with data collection in a variation or with conversion tracking.
"We recommend you to run this experiment till it reaches a conclusive result."
The cause: This may happen when the desired visitor count has not been reached to conclude the results. In this case, you need to wait longer to reach the desired visitor count, have a better conversion rate, and reach the required significance level you chose.
"The result is inconclusive as the performance of the original and variation is similar."
The cause: This could be because both the original and the variation version of your webpage perform similarly and hence there is no better advantage of publishing a new variation over the original. In this case, we recommend that you end the test and create a new A/B experiment for your page.
Time-based graph: This graph illustrates multiple spikes of data distributed across a chosen time frame (along the x-axis). It is time based because the data values or points are calculated over regular time intervals, and will be considered independent of each other.
Cumulative graph: This graph illustrates flattened data distributed across a chosen time frame (along the x-axis). It is cumulative because the data values or points are calculated as a sum over time, like computing the sum of the first point, then the first two points, then the first three, and so on.
View Forecast: Enabling the Forecast button helps you to view predictions or changes in the conversion rate (and improvement rate) that each variation might take over the chosen time range. Forecast is an estimated value of change over a future time horizon. This can be used to examine the performance of your A/B results, and predict your success in the long term.
Variant: Indicates the list of all the variants available in your experiment page, including the original.
Visitors: Indicates the total number of unique visitors to the corresponding variation.
Conversions: Indicates the number of unique instances of the visitor fulfilling the desired action for a given goal. A conversion can refer to any desired action that you want the user to take. This can include any action performed on your web page, from clicking on a button to making a purchase and becoming a customer.
Conversion rate: The conversion rate is the number of conversions divided by the total number of visitors.
For example, if your variation page receives 200 visitors in a month and has 50 purchases, the conversion rate would be 50 divided by 200, or 25%. The conversion rate is represented by percentage value, and can have any positive or negative value.
Difference Interval: When communicating the results, it's not only a good idea to present the observed difference in the conversion rate value for the original and variation, but also the range between which the conversion rate of your original and the variation page can actually lie. This possible range of values is called the difference interval, and is plotted on a number line scale.
On the number scale, the upper limit is marked by the maximum possible range of conversion rate, and the lower limit is marked by the minimum possible range of conversion rate between all the variations. The different shades on the scale indicate the following:
Grey shade area: Indicates the experiment is inconclusive, or needs more visitors to declare a valid result.
Green shade area: Indicates a winning variation.
Red shade area: Indicates a losing variation.
For example, let's say you test your variation page with 7626 visitors, and get 1722 conversions with a conversion rate of 22.58%. For this, you see a difference interval between 21.64 % - 23.52 %, which means that your conversion rate value could lie anywhere between this difference based on the new visitors and conversions obtained on your variation page.Note:One thing to watch out for in difference interval is the overlap of the conversion rate from the Original and Variation 1 of your A/B test. For example, suppose that Original has a confidence interval of 10-20% for conversion rates, and Variation 1 has a confidence interval of 15-25%. Notice that the overlap of the two confidence intervals is 5%, and it is located in the range between 15-20%. In this case, it's very difficult to be sure that the variation tested in B is actually a significant improvement. This is why we will not declare winner if there is overlap.
By activating a heatmap in your A/B (or Split URL) experiment, you can visualize and identify where your visitors are clicking, scrolling, and looking in your original and variation pages. This report also provides you additional insights on why a particular variation did not perform as expected, and what needs to be changed to address issues.
By combining the knowledge from heatmaps with A/B testing reports, you can increase the conversions on your variation pages, and achieve a higher statistical significance for declaring a winner.
Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.
If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.
You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.