Configure experiment settings for an A/B test

Configure experiment settings for an A/B test



The Configuration step allows you to define how traffic is distributed across variations, select the statistical model used to evaluate results, configure experiment activation, enable heatmaps, and schedule when the experiment should run.
These settings help ensure that your experiment is executed correctly and that the collected data is analyzed using the methodology that best suits your testing objectives.

Configure experiment settings

To configure your experiment:

1. Navigate to the Configuration step in the experiment workflow.
2. Configure traffic distribution settings.



3. Select an experiment evaluation method.



4. Configure the activation mode.



5. Enable heatmaps if required.



6. Schedule the experiment start and end dates.


7. Click Next to proceed to the Summary step.

Segment traffic

Segment Traffic determines how visitors are distributed across the original page and experiment variations. Choose a traffic allocation strategy based on your testing goals and risk tolerance.  
Manual distribution
Auto distribution (Multi-Armed Bandit)
Manual distribution
Manual Distribution allows you to control exactly how much traffic is allocated to each variation. You can distribute traffic equally across all variations or assign custom percentages based on your experiment requirements. This approach provides complete control over visitor allocation and is commonly used in traditional A/B testing.
Example:
You are testing one original page and one variation.
You allocate:
  • Original: 50%
  • Variation 1: 50%
This ensures that both experiences receive an equal share of traffic, allowing for a balanced comparison.
You can also allocate traffic unevenly.
For example:
  • Original: 70%
  • Variation 1: 30%
This approach is often used when testing a new design on a smaller audience before exposing it to a larger percentage of visitors.



Auto distribution (Multi-Armed Bandit)
Auto Distribution uses a Multi-Armed Bandit approach to dynamically adjust traffic allocation during the experiment.
As the experiment progresses, PageSense automatically sends more visitors to variations that are performing better while reducing traffic to lower-performing versions. This helps maximize conversions during the experiment rather than waiting until the test is complete.
Example:
You are testing three landing page variations.
Initially, traffic is distributed evenly across all versions.
As results become available, PageSense identifies that Variation 2 is generating significantly more conversions and gradually allocates more visitors to that variation while reducing traffic to the others.


When to use auto distribution

Use Auto Distribution when:
  • Maximizing conversions is more important than conducting a perfectly balanced experiment.
  • You want PageSense to automatically optimize traffic allocation.
  • You want to reduce visitor exposure to poorly performing variations.

Set experiment method

The experiment method determines how PageSense evaluates performance and calculates statistical confidence.
Bayesian method
Frequentist method
Bayesian method
The Bayesian method provides a probability-based approach to experiment analysis.
Instead of simply determining whether one variation outperforms another, Bayesian analysis calculates the probability that a variation is likely to perform better.
This method often produces meaningful insights faster and is easier to interpret for most business users.
Example :
If Variation B shows a 92% probability of outperforming Variation A, you can confidently understand the likelihood of improvement without waiting for a fixed testing duration.
Benefits :
  • Faster decision making
  • Probability-based insights
  • Easier interpretation of results
  • Suitable for most optimization experiments


Frequentist method
The Frequentist method evaluates experiment results using statistical significance and confidence thresholds.
Unlike Bayesian analysis, this method requires a sufficient sample size and testing duration before reliable conclusions can be drawn.
Example :
A variation may achieve a 95% statistical significance level, indicating that the observed improvement is unlikely to have occurred by chance.
Benefits :
  • Widely accepted statistical approach
  • Suitable for formal experimentation programs
  • Strong confidence-based validation

Statistical significance level

When using the Frequentist method, you can define the confidence threshold required before PageSense identifies a winning variation.


Quick trends
Optimal
High accuracy
Quick trends
Quick Trends provides faster directional insights using lower confidence requirements.
This option is useful when you want to identify early performance patterns.
Best for:
  • Exploratory experiments
  • Initial design validation
  • Rapid testing cycles

Optimal
Optimal provides a balance between testing speed and result accuracy.
This option is suitable for most business experiments and is generally recommended.
Best for:
  • Landing page optimization
  • Conversion rate optimization
  • User experience testing


High accuracy
High Accuracy requires stronger statistical confidence before declaring a winner.
This setting reduces the likelihood of false positives but may require longer testing durations.
Best for:
  • High-impact business decisions
  • Revenue-critical experiments
  • Large-scale optimization initiatives

Experiment activation mode

Activation mode determines when the experiment is applied to visitors.
Activate when the page loads
Activate manually
Activate on satisfying a condition
Activate when the page loads
The experiment is triggered immediately when the targeted page loads. This is the most commonly used activation method.
Example :
A visitor lands on the homepage and immediately receives the assigned variation.

Activate manually
The experiment is triggered only when manually activated through custom implementation. This option provides greater control over when variations are displayed.
Example :
A developer activates the experiment after a specific application event occurs.

Activate on satisfying a condition
The experiment is triggered only after predefined conditions are met.This allows experiences to be displayed based on visitor behavior or page state.
Example :
Launch a variation only after a visitor scrolls halfway through the page or interacts with a specific element.

Enable heatmap

Enable Heatmap to automatically collect visitor interaction data across all experiment variations.
Heatmaps help you understand how visitors interact with each variation and provide additional context when analyzing experiment results.
Heatmap data can reveal:
  • Click behavior
  • Scroll behavior
  • Visitor engagement patterns
  • Areas of interest
  • Areas receiving little interaction
Example : 
Variation B generates more conversions than Variation A.
By reviewing the heatmap, you discover that visitors interact more frequently with the repositioned call-to-action button, helping explain the performance improvement. 

Schedule

Scheduling allows you to control when an experiment starts and ends.
Use scheduling when experiments need to run during specific campaigns, promotions, seasonal events, or controlled testing periods.
  1. Launch this experiment later
Enable this option if you want the experiment to start automatically at a future date and time.
Example : Configure a Black Friday experiment several days in advance and schedule it to launch automatically when the promotion begins.
  1. Start date
Defines when the experiment becomes active. Visitors will not be included in the experiment before the configured start date and time.
  1. End date
Defines when the experiment automatically stops running.
After the end date is reached, visitor participation ends and no new experiment data is collected.
Example :  Schedule an experiment to run from March 24, 2026, at 01:51 PM until March 31, 2026, at 01:51 PM to evaluate performance during a one-week campaign.

Best practices

  • Use Manual Distribution when equal traffic allocation is required.
  • Use Auto Distribution when maximizing conversions during the experiment is a priority.
  • Select the statistical method that aligns with your organization's experimentation practices.
  • Use Optimal significance settings for most experiments.
  • Enable heatmaps when additional behavioral insights are required.
  • Schedule experiments around major campaigns or product launches.
  • Avoid making configuration changes after the experiment has started.




We’ve designed this documentation to guide you every step of the way. If you need further assistance or have any questions, don’t hesitate to contact us at support@zohopagesense.com - we’re always here to help!