A/B testing is a great method to compare two or more versions of your web page to figure out which one performs better in front of your audience in terms of conversion. Sometimes, in order to speed up your process and maximize your marketing efforts, you might be tempted to run multiple A/B experiments at the same time. By default, when you are running several experiments in PageSense, it may be possible that a single visitor be exposed to more than one experiment simultaneously. This method of exposing visitors to multiple test pages works well if the pages are completely unrelated, or if you are tracking different goals..
However, in a few cases, as given below, exposing a single visitor to different experiments at the same time can actually interfere and skew your A/B test results:
Running two or more experiments on the same page
Tracking the same goal on multiple experiments
Having visitors interacting between test pages on the same flow, such as in funnel analysis (for example buyer's journey funnel or newsletter sign up funnel).
To avoid this interference of data collected for one test by another, and help ensure that a single visitor who is part of one test does not become part of another one, you can simply create mutually exclusive groups in PageSense. This advanced setting in the tool helps you to group multiple tests together that are mutually exclusive and comes with the following benefits:
Your visitors are not exposed to multiple tests running on the same page
You can keep the reports clean by avoiding overlapping of visitors across different tests
Remove bias in test data and relate the change in conversion rate to the correct test.
Eventually, the visitor ends up buying something from your website and registers a conversion.
Based on the conversions obtained in each test, you will be able to figure out:
Which variation of the tests has highly contributed to the success of your Christmas campaign
If any of the variations in those tests interfere with the data and results of another
Perhaps the homepage test convinced the visitor to look at what offer you exactly provide, which later directed them to the relevant product pages.It may have been the product page test where the visitor saw new attractive arrivals and made them to add items to the cart. Perhaps it was the checkout page test that provided extra coupon discounts that had a greater impact compared to all other tests and made the visitor end up buying your item.
Unfortunately, you can't easily tell which A/B experiment got higher conversions and revenue on your site, as there can always be hidden, unrelated interactions between different test pages. The easiest way to handle this is by creating mutually exclusive groups.
Creating a mutually-exclusive group lets you to split your visitor traffic into the A/B test that you're running, and makes sure that each group of visitors participate in one test only. This way, there’s no possibility that a visitor can pollute the results of one experiment with the variant they. This is because, from a visitors' point of view, there’s only one experiment being runn and shown to them whenever they land on your page.
To add new experiments in an exclusive group:
Select the group you want to add the experiment, click the plus icon then from the dropdown choose the new experiment and percentage of visitors to be included.
To remove finished experiments from the exclusive group:
Select the group from which you want to remove a finished experiment, then click the Close icon next to the experiment name.
Info: If you delete an experiment from an exclusion group, visitors who were previously included in that experiment will be removed from the exclusion group. This means the visitors won't see any other experiments or campaigns in that exclusion group. However, if you have any experiment or campaign running outside this group, and if the deleted visitor meets the targeting condition, they could see this experiment or campaign.
To delete a mutually exclusive group:
Select the group you want to remove from the list, click the More icon, then select Delete from the menu options, as shown in the figure.
To archive experiments in the exclusive group:
Select the group you want to archive from the list and select the Archive button as shown in the figure.
To find and unarchive experiments back to the exclusive group:
In PageSense, when experiments are not mutually exclusive, visitors are directly made part of every experiment that you are running on the website. For example, if a visitor sees Experiment A then they can also see Experiment B. This means the results for A and B can be skewed by the overlapping visitors.
However, in the case of mutually exclusive experiments, PageSense chooses a random value to bucket visitors to one of the experiments in the group and then stores this value in the cookie. For example, if experiments A and B are mutually exclusive, PageSense chooses a random value to bucket visitors into either Experiment A or Experiment B. This method ensures that visitors won't see more than one experiment from each exclusion group.
Experiment A: 50% of traffic allocated
Experiment B: 50% of traffic allocated
In this case, PageSense randomly distributes about half of your visitors to Experiment A and half to Experiment B. This ensures that experiments don't overlap for the same users.
Now, if you add two more experiments to the exclusion group and reallocate traffic as follows:
Experiment A: 25% traffic allocation
Experiment B: 25% traffic allocation
Experiment C: 25% traffic allocation
Experiment D: 25% traffic allocation
Based on your updated traffic allocation, returning visitors will continue to see the same experiment they saw the first time: Experiment A or B. Returning visitors will not see Experiment C or D. On the other hand, your new visitors might see any of the four experiments. About 25% of your visitors will see each of Experiments A, B, C, or D.
Example 2: When one experiment is overlapping and two experiments are in mutually exclusive group
Now imagine you're running three experiments in PageSense but one has overlapping visitors and two are in an exclusion group 'ABC' with the traffic allocation like below:
100% of traffic falls in Experiment A
50% of traffic falls in Experiment B in group ABC
50% of traffic falls in Experiment C in group ABC.
In this case, all visitors will see Experiment A. Some visitors will see Experiment B, and some will see Experiment C. However, no visitors will see both Experiment B and Experiment C. This is because Experiments B and C are both in the same exclusion group, which makes them mutually exclusive. This eventually means all visitors will see Experiment A along with Experiment B or C.
Learn how to use the best tools for sales force automation and better customer engagement from Zoho's implementation specialists.
If you'd like a personalized walk-through of our data preparation tool, please request a demo and we'll be happy to show you how to get the best out of Zoho DataPrep.
You are currently viewing the help pages of Qntrl’s earlier version. Click here to view our latest version—Qntrl 3.0's help articles.