Skip to main content
menu_icon.png

Everything you need to switch from Optimizely Classic to X in one place: See the Optimizely X Web Transition Guide.

x
Optimizely Knowledge Base

Create mutually exclusive experiments in SDK projects

Relevant products:
  • Optimizely X Full Stack
  • Optimizely X Mobile
  • Optimizely OTT

THIS ARTICLE WILL HELP YOU:
  • Create mutually exclusive experiments and prevent interaction effects
  • Ensure that a single user doesn't see overlapping experiments that test the same feature
  • Create an exclusion group to keep experiments mutually exclusive and prevent interaction effects
  • Decide when to use exclusion groups and when to allow experiment overlap

By default, your Optimizely experiments in a project may overlap with one another, so a single user could be exposed to multiple experiments at the same time. Mutually exclusive experiments help you ensure that a single user can't be exposed to two different experiments running at the same time. In other words, they ensure that users who are exposed to Experiment #1 never see Experiment #2, and vice versa.

Use mutually exclusive experiments when you'd like to control for interaction effects.

For example, imagine that you're running two experiments that test the same algorithm. You don't want any users to see both experiments because there could be interaction effects between those experiments that taint your experiment results. Certain users may see both, and they may behave differently from users who see each experiment in isolation. To clearly understand the impact of each set of changes, you make the two experiments mutually exclusive.

In SDK projects, you can make two or more experiments mutually exclusive to each other.

See additional resources
Here are all our articles about the Optimizely dashboard, which you'll use to create projects, add collaborators, manage privacy settings, and more: 

Here's our developer documentation, where you'll find code samples, full references for our SDKs, and getting started guides.

Create an exclusion group

In SDK projects, use exclusion groups to make multiple experiments mutually exclusive to each other.

  1. Navigate to the Experiments dashboard and select the Exclusion Groups tab. Click Create an Exclusion Group.

     

  2. Name and describe the exclusion group.


    For example, the exclusion group above is named the "Search Algorithm Group." All experiments on the search algorithm that are added to this exclusion group will be mutually exclusive. No user will see more than one experiment in this exclusion group.

  3. Click Create Exclusion Group.

Congratulations! You've set up an exclusion group of mutually exclusive experiments.

How exclusion groups work

For experiments that are not mutually exclusive, Optimizely uses a seed, or unique value, for each experiment to bucket the user. The seed determines whether a user enters a particular experiment. Because the seeds are random, unique, and not mutually exclusive across experiments, some users enter multiple experiments. For example, imagine two experiments: A and B, each receives 20% traffic allocation (the percentage of total traffic that is eligible for the experiment). Here's the expected traffic allocation:

  • 16% of traffic falls in experiment A only

  • 16% of traffic falls in experiment B only

  • 4% of traffic falls in both experiment A and experiment B

  • 64% of traffic is not in any experiment

In the example above, results from experiment A and experiment B may be skewed. If users who see both A and B behave differently from users who see just A or just B, then the results for A and B are skewed by the overlap. This is called an interaction effect.

If experiments A and B are mutually exclusive, Optimizely chooses the same random seed (which is unique to the exclusion group) to bucket users in experiments A and B -- and distribute traffic to one or the other. This method ensures that experiments can't overlap for the same users. If experiments A and B are mutually exclusive, the traffic allocation looks something like this:

  • 20% of traffic falls in experiment A only

  • 20% of traffic falls in experiment B only

  • 60% of traffic is not in any experiment

Optimizely also ensures mutual exclusivity between experiments in an exclusion group that run at different times. How? By assigning bucket ranges to experiments using a stratified sample of available buckets, with strata that consist of all current and previously allocated bucket ranges.

Add an experiment to an exclusion group

Here's how to add an experiment to an exclusion group, with step-by-step instructions below:

  1. Select the experiment.

  2. In the Experiment Traffic Allocation section, select the exclusion group from the drop-down menu under Add this experiment to the following exclusion group.

  3. Add the percentage of the exclusion group traffic that should be allocated to the experiment.

  4. Click Save Experiment.

Remove an experiment from an exclusion group

After an experiment is finished, remove it from the exclusion group to free up space in the group for other experiments. Here's how, with step-by-step instructions below:

  1. Select the experiment.

  2. In the Experiment Traffic Allocation section, select None from the drop-down menu under Add this experiment to the following exclusion group.

  3. Click Save Experiment.

Best practices

To guard against any possibility of interaction effects, you might consider making all your experiments mutually exclusive. But sometimes making all experiments in the project mutually exclusive requires more traffic than is available. As a best practice, we recommend that some experiments should overlap and some experiments should be mutually exclusive, depending on the traffic levels you need to reach significance and which parts of your code base are being tested.

Below are a few tips for when to create mutually exclusive experiments.

You're more likely to see interaction effects if:

  • You're running two experiments on the same area of an application

  • You're running two experiments on the same flow where there's likely going to be strong overlap

  • You're running an experiment that may have a significant impact on a conversion metric that is shared with other experiments

If the above points don't apply, then it's usually unnecessary to create mutually exclusive experiments. Both variations of the experiments are exposed to the other experiment proportionally.

However, there are a few scenarios when creating mutually exclusive experiments, or running sequential tests (waiting for one to end in order for the next to start) is recommended. Even if you ensure that experiments are mutually exclusive, it's still possible to see interaction effects from experiments running at different times. After you've experimented on some population of users, it's never possible to get a truly unbiased population for future experiments. If you’re concerned about interaction effects between experiments running at different times, finish all experiments in an exclusion group before creating new experiments.

For example, suppose you created an exclusion group with four experiments (A, B, C, and D), running at 25% traffic allocation each. If you stop experiment D and start another experiment E, the experiment results of E could be biased because all users in E were previously given the treatment from D. Wait for experiments A, B, and C to finish before starting experiment E to make sure the traffic is evenly sampled across all previous experiments.

When making important decisions for your business, evaluate your risk tolerance for experiment overlap. Evaluate your prioritized roadmap to ensure that you are planning your variation designs, goals, and execution schedule to best meet your business needs.

Archive and unarchive an exclusion group

When you don't use an exclusion group anymore, you can archive it. Here's how:

  1. Navigate to Experiments > Exclusion Groups.

  2. Click the ... icon to the right of the exclusion group you want to archive.

  3. Select Archive.

To find your archived exclusion groups, navigate to Experiments > Exclusion Groups and click the drop-down menu next to the Browse field. Select Archived to see a filtered list of your archived exclusion groups.

To unarchive an exclusion group so you can use it again:

  1. Navigate to Experiments > Exclusion Groups.

  2. Filter the list to show your archived exclusion groups.

  3. Click the ... icon to the right of the exclusion group you want to unarchive.

  4. Select Unarchive.