Skip to main content
menu_icon.png

Everything you need to switch from Optimizely Classic to X in one place: See the Optimizely X Web Transition Guide.

x
Optimizely Knowledge Base

Mutually exclusive experiments in Optimizely X Web

There are two versions of Optimizely
What version do you have?
X
Optimizely Classic
This is what the Optimizely Classic user interface looks like.
Optimizely X
This is what the Optimizely X user interface looks like.
. If you're using Optimizely Classic, check this article out instead.
relevant products:
  • Optimizely X Web Experimentation
  • Optimizely X Web Personalization

THIS ARTICLE WILL HELP YOU:
  • Determine whether you need mutually exclusive experiments
  • Set up a mutually-exclusive experiment 

Exclusion groups are only available to customers with Enterprise Premium and Personalization plan types.

Many Optimizely customers run several different experiments at the same time. Because these simultaneous experiments are usually unrelated, it doesn't really matter if visitors are exposed to more than a single experiment when they visit your site.

But sometimes, these simultaneous experiments are related. For example, perhaps you want to run two different experiments on the same algorithm. Visitors who see both experiments when they visit your site might behave differently from visitors who see only one of these experiments in isolation. This difference in behavior has the potential to affect your results.

To avoid this, you can make your experiments mutually exclusive. Any visitor who sees Experiment A will not see Experiment B, and vice versa.

Mutually exclusive experiments help you remove noise and bias from your results, so you can make accurate inferences based on your visitors' behavior.

When to use mutually exclusive experiments

To decide whether you should create mutually exclusive experiments, first ask how likely you are to see interaction effects on the experiment. You're more likely to see these effects if:

  • You're running two experiments on the same page.

  • You're running two experiments on the same flow where there's likely going to be strong overlap from one page to the next (for example, a checkout funnel or multi-step form).

  • You're running two experiments that are actually better to run together, for instance as a multivariate experiment, to see the potential interaction

If these conditions don't apply to you, then you probably don't need to create mutually exclusive experiments. This is because both variations of the tests are exposed to the other test proportionally.

The diagram below illustrates how two tests running simultaneously (one on your homepage and one on your search results page) would interact:

  • Each visitor in variation (version) A or B has an equal chance of seeing variation C or D.

  • Each visitor in variation C or D has an equal chance of seeing variation A or B.

  • So if there are any (slight) interaction biases, they would even out, as long as both audiences are eligible to see both experiments.

In other words, your tests are less likely to have overlapping effects if they:

  • Run on different areas of your site

  • Track different goals

However, in some cases, we do recommend creating mutually exclusive experiments or running sequential tests (waiting for one to end in order for the next to start):

  • When you are testing elements on the same page with the same goal. Consider whether to run these experiments simultaneously, sequentially, or test all the element changes in the same multivariate experiment to see individual and combined performance.

  • When you are testing subsequent pages of the funnel with the same goal. Consider whether to run these experiments simultaneously, sequentially (starting from the bottom of the funnel and moving up), or as a multi-page experiment.

Some organizations choose to run simultaneous, mutually exclusive experiments, to preserve data integrity. You can accomplish this in Optimizely with the code we provide below. Note that working with mutually exclusive experiments increases the effort involved in implementation, and can significantly slow down your testing cycle.

Create an exclusion group

In Optimizely X Web, you create exclusion groups to make multiple experiments and campaigns mutually exclusive from each other. Here's how to create an exclusion group, with step-by-step instructions below.

excl-group.gif

  1. Navigate to the Experiments dashboard and select the Exclusion Groups tab.

  2. Click Create an Exclusion Group.

  3. Name and describe the exclusion group.
    In this example, the exclusion group is named "Search Algorithm Group." Customers can add experiments on their site to this exclusion group to ensure that the experiments will be mutually exclusive. No visitor will see more than one experiment that's in this exclusion group.

  4. Click Create Exclusion Group.

Congratulations! You set up an exclusion group for mutually exclusive experiments.

Mutually exclusive experiments and exclusion groups

Optimizely uses a unique value for each experiment to bucket each visitor. When experiments are not mutually exclusive, visitors are eligible for every experiment in which they're not in the holdback. In mutually exclusive experiments, Optimizely creates a unique value for the visitor to make sure that they're only bucketed into one experiment. Because the values are random and unique, some visitors enter multiple experiments. If visitors who see both Experiments A and B behave differently from visitors who see only Experiment A or B, then the results for A and B are skewed by the overlap. This is called an interaction effect.

If Experiments A and B are mutually exclusive, Optimizely chooses a random value (which is unique to the exclusion group) to bucket visitors into either Experiment A or Experiment B. This method ensures that visitors won't see more than one experiment from each exclusion group.

Let's walk through a few descriptions and diagrams to explain mutually exclusive experiments and exclusion groups in more detail.

Overlapping experiments with 100% traffic allocation

If you have two overlapping experiments, each with 100% traffic allocation, all visitors will be bucketed into both experiments (as long as they meet the experiments’ targeting conditions).

Overlapping experiments with different traffic allocation

Suppose you have two overlapping experiments, but they each have a different traffic allocation:

  • Experiment A: 100% traffic allocation

  • Experiment B: 50% traffic allocation

In this example, all visitors will see Experiment A. About half of the visitors will only see Experiment A, and about half will see both Experiments A and B. Some visitors won't see Experiment B at all.

One overlapping experiment with two mutually exclusive experiments, 100% traffic allocation

Now suppose you still have three experiments, but two are in an exclusion group:

  • Experiment A: 100% traffic allocation

  • Experiment B: 50% traffic allocation and in Exclusion Group 1

  • Experiment C: 50% traffic allocation and in Exclusion Group 1

In this case, all visitors will see Experiment A. Some visitors will see Experiment B, and some will see Experiment C. However, no visitors will see both Experiment B and Experiment C. This is because Experiments B and C are both in the same exclusion group, which makes them mutually exclusive.

One overlapping experiment with two mutually exclusive experiments, some traffic not allocated

In this example, you have three experiments again, and two are mutually exclusive, but some of your traffic is not allocated to either mutually exclusive experiment:

  • Experiment A: 100% traffic allocation

  • Experiment B: 20% traffic allocation and in Exclusion Group 1

  • Experiment C: 50% traffic allocation and in Exclusion Group 1

  • Some traffic (30%) in Exclusion Group 1 is not allocated to either Experiment B or C

Again, all visitors will see Experiment A. Experiments B and C are still mutually exclusive, so some visitors will see Experiment B, some will see Experiment C, and none will see both. However, in this case, 30% of visitors will see only Experiment A because 30% of visitors are not allocated to either experiment in the exclusion group.

Experiments in two exclusion groups

You can also use two different exclusion groups to add layers of mutually exclusive experiments. Imagine these experiments and traffic allocations:

  • Experiment A: 100% traffic allocation

  • Experiment B: 20% traffic allocation and in Exclusion Group 1

  • Experiment C: 50% traffic allocation and in Exclusion Group 1

  • Some traffic (30%) in Exclusion Group 1 is not allocated to either Experiment B or C

  • Experiment D: 50% traffic allocation and in Exclusion Group 2

  • Experiment E: 50% traffic allocation and in Exclusion Group 2

In this example, all visitors will see Experiment A, some visitors will see Experiment B, some will see Experiment C, and none will see both B and C. Again, 30% of visitors will not see either Experiment B or C.

But in this case, because of Exclusion Group 2, we have more possibilities. All visitors will see either Experiment D or E, but none will see both. It's possible for a visitor to see experiments from all three layers: Experiment A, plus one experiment from each of Exclusion Group 1 and Exclusion Group 2. It's also possible for a visitor to see experiments from only two layers, because some traffic is not allocated to either experiment in Exclusion Group 1.

Paused experiments

When you pause an experiment, new visitors will no longer be bucketed into that experiment.

Visitors who've already seen the experiment will remain bucketed into the experiment, although they will not see it again for as long as it is paused. If you resume the experiment, returning visitors who are bucketed into the experiment will begin seeing it again, and new visitors could be bucketed into it.

For example, suppose a visitor is bucketed into Experiment C when they visit your site. If you pause Experiment C, the visitor won't see it anymore. If you resume the experiment, and the visitor returns to your site, they'll see Experiment C again because they're still bucketed into it.

Personalization campaigns and exclusion groups

You can also add personalization campaigns to exclusion groups to make experiments and campaigns mutually exclusive. In the example shown below, all visitors will see Experiment A. Some visitors will also see either Experiment B or Campaign A. And some visitors will see only Experiment A.

When you add a Personalization campaign to an exclusion group, the campaign's holdback is not affected.

Two experiments that target different pages 

When two mutually exclusive experiments target different pages, the exclusion group decision about which experiment to bucket a visitor in is sticky and deterministic. In other words, if a visitor is bucketed into Experiment A on your site's homepage, and Experiment A is in the same exclusion group as Experiment B on your site's "order" page, the visitor will not see Experiment B.

Two experiments that target non-mutually exclusive and mutually exclusive audiences

When two mutually exclusive experiments target different audiences that are not mutually exclusive, the exclusion group decision about which experiment to bucket a visitor in is sticky and deterministic. For example, suppose you have two non-mutually exclusive audiences, like "visitors in California" and "all visitors." If a visitor from the "visitors in California" audience is bucketed into Experiment A on your site's homepage, and Experiment A is in the same exclusion group as Experiment B, the visitor will not see Experiment B, even though they also qualify for the the "all visitors" audience.

The same is true when the two mutually exclusive experiments target different pages and non-mutually exclusive audiences, and when you have three mutually exclusive experiments that target the same page.

Two mutually exclusive experiments that target different mutually exclusive audiences, like "Android users" and "iPhone users," are also sticky and deterministic. However, if the audiences for your experiments are mutually exclusive, you don't need to use exclusion groups for the experiments themselves.

Two experiments with different traffic allocation ratios

If an experiment is in an exclusion group, 100% of that experiment's traffic is allocated to the percentage you have allocated to the exclusion group. For example, suppose your Exclusion Group 1 includes two experiments with these traffic allocations:

  • Experiment A: 70% traffic allocation

  • Experiment B: 30% traffic allocation

In this setup, 100% of the traffic that you send to Experiment A (70% of traffic to Exclusion Group 1) will see Experiment A.

Removing experiments and campaigns from exclusion groups

If you remove an experiment or campaign from an exclusion group, visitors who were previously bucketed into that experiment or campaign are dropped from the exclusion group. This means the visitors won't see any other experiments or campaigns in that exclusion group. If the experiment or campaign is still running and the visitors meet the targeting conditions, they could still see the experiment or campaign.

Duplicating experiments and campaigns in an exclusion group

If you duplicate an experiment or campaign that's part of an exclusion group, the new experiment or campaign will not be associated with any exclusion group.

Changing traffic allocation for experiments or campaigns in an exclusion group

Optimizely uses "sticky" bucketing for the traffic allocation for experiments and campaigns in an exclusion group. This means that if you change the traffic allocation, returning visitors will continue to see the experiments and campaigns they saw before your change. New visitors will see experiments and campaigns according to your new traffic allocation.

For example, suppose you have two experiments in an exclusion group, with these traffic allocations:

  • Experiment A: 50% traffic allocation and in Exclusion Group 1

  • Experiment B: 50% traffic allocation and in Exclusion Group 1

About half of your visitors will see Experiment A and half will see Experiment B.

Now, you add two more experiments to the exclusion group and reallocate traffic like this:

 you have two experiments in an exclusion group, with these traffic allocations:

  • Experiment A: 25% traffic allocation and in Exclusion Group 1

  • Experiment B: 25% traffic allocation and in Exclusion Group 1

  • Experiment C: 25% traffic allocation and in Exclusion Group 1

  • Experiment D: 25% traffic allocation and in Exclusion Group 1

With your updated traffic allocation, returning visitors will continue to see the same experiment they saw the first time: Experiment A or B. Returning visitors will not see Experiment C or D. New visitors might see any of the four experiments: about 25% of your visitors will see each of Experiments A, B, C, or D.

Archiving exclusion groups

Before you archive an exclusion group, you must remove all experiments and campaigns in it. Archiving is disabled for exclusion groups with experiments or campaigns in them.

Visitor bucketing IDs

A given visitor will have a consistent bucketing ID each time they return to the same exclusion group, experiment, or campaign to be bucketed. This bucket ID won't be the same for a given visitor across exclusion groups, experiments, or campaigns.

Best practices

For reliable results, we recommend that you start all the experiments or campaigns in an exclusion group at the same time, after you've added them to the exclusion group. We do not recommend adding running experiments or campaigns to an exclusion group because it may have unintended effects on your results.

Debugging techniques

Use the JavaScript API and Console log for debugging:

?optimizely_log=debug

Look for “Group” in the logs.

You can also check window.optimizely.get('data').groups on a page that has a grouped experiment or campaign running on it. The weightDistributions property will contain a list of entities (and their campaign IDs).