Skip to main content
menu_icon.png

Everything you need to switch from Optimizely Classic to X in one place: See the Optimizely X Web Transition Guide.

x
Optimizely Knowledge Base

Mutually exclusive experiments in Optimizely X Web

This article is about Optimizely X. If you're using Optimizely Classic, check this article out instead.
 
relevant products:
  • Optimizely X Web Experimentation
  • Optimizely X Web Personalization

THIS ARTICLE WILL HELP YOU:
  • Determine whether you need mutually exclusive experiments
  • Set up a mutually-exclusive experiment 

Many Optimizely customers run several different experiments at the same time. Because these simultaneous experiments are usually unrelated, it doesn't really matter if visitors are exposed to more than a single experiment when they visit your site.

But sometimes, these simultaneous experiments are related. For example, perhaps you want to run two different experiments on the same algorithm. Visitors who see both experiments when they visit your site might behave differently from visitors who see only one of these experiments in isolation. This difference in behavior has the potential to affect your results.

To avoid this, you can make your experiments mutually exclusive. Any visitor who sees Experiment A will not see Experiment B, and vice versa.

Mutually exclusive experiments help you remove noise and bias from your results, so you can make accurate inferences based on your visitors' behavior.

In Optimizely X, mutually exclusive experiments are now available in beta. If you'd like early access, please contact your Customer Success Manager.

When to use mutually exclusive experiments

To decide whether you should create mutually exclusive experiments, first ask how likely you are to see interaction effects on the experiment. You're more likely to see these effects if:

  • You're running two experiments on the same page.

  • You're running two experiments on the same flow where there's likely going to be strong overlap from one page to the next (for example, a checkout funnel or multi-step form).

  • You're running two experiments that are actually better to run together, for instance as a multivariate experiment, to see the potential interaction

If these conditions don't apply to you, then you probably don't need to create mutually exclusive experiments. This is because both variations of the tests are exposed to the other test proportionally.

The diagram below illustrates how two tests running simultaneously (one on your homepage and one on your search results page) would interact:

  • Each visitor in variation (version) A or B has an equal chance of seeing variation C or D.

  • Each visitor in variation C or D has an equal chance of seeing variation A or B.

  • So if there are any (slight) interaction biases, they would even out, as long as both audiences are eligible to see both experiments.

In other words, your tests are less likely to have overlapping effects if they:

  • Run on different areas of your site

  • Track different goals

However, in some cases, we do recommend creating mutually exclusive experiments or running sequential tests (waiting for one to end in order for the next to start):

  • When you are testing elements on the same page with the same goal. Consider whether to run these experiments simultaneously, sequentially, or test all the element changes in the same multivariate experiment to see individual and combined performance.

  • When you are testing subsequent pages of the funnel with the same goal. Consider whether to run these experiments simultaneously, sequentially (starting from the bottom of the funnel and moving up), or as a multi-page experiment.

Some organizations choose to run simultaneous, mutually exclusive experiments, to preserve data integrity. You can accomplish this in Optimizely with the code we provide below. Note that working with mutually exclusive experiments increases the effort involved in implementation, and can significantly slow down your testing cycle.

Mutually exclusive experiments in Optimizely

Optimizely automatically evaluates whether to run each experiment independently. If more than one experiment matches the targeting conditions on a given page, they will all execute serially. However, you cannot assume a certain order of execution. You'll need to ensure that the experiments do not interfere with each other.

If you don't want to run the same experiments on the same page at the same time, you can create a Custom JavaScript Audience condition for each mutually exclusive experiment.

Use custom JavaScript audiences for mutually exclusive experiments 

You can use a custom JavaScript audience condition to ensure that only one experiment within a particular exclusion group will execute at a time. Only experiments in the same project can be mutually exclusive.

  1. Create an audience for each experiment using a Custom JavaScript audience condition to ensure these experiments will be mutually exclusive.

  2. Paste this code into the Custom JavaScript code box:

/* CUSTOM JS TARGETING OF MUTUALLY EXCLUSIVE EXPERIMENTS
 *
 * This solution is designed to only work with single experiments in Optimizely.
 * Personalization campaigns are currently not supported.
 *
 * `campaignArray` and `currentCampaign` should be campaign IDs NOT experiment IDs.
 * The "hidden" campaign ID can be found under API Names within the experiment builder (top of the page).
 * 
 * Starting new campaigns will not affect the bucketing of existing visitors.
 *
 * Pausing a campaign originally included in `campaignArray` will result in visitors
 * of that experiment to be rebucketed.
 *
 * The `campaignArray` must always be maintained between different audiences and it should match
 * the campaigns that are running and it should include all campaigns for the group.
 * 
 */

// If left empty by changing this line to "campaignArray = []", all active experiments from the project will be used.
var campaignArray = ["<campaign_ID1>", "<campaign_ID2>", "<campaign_ID3>"];              

// Set the ID for the current campaign being evaluated.
var currentCampaign = "<campaign_ID_for_current_campaign";

// Optional. Needed if using multiple groupings. Default is "groupA".
var groupName = "groupA";

/* true - YES - if no match is found pick at random from campaignArray
 * false - NO - if no match is found, pick this experiment */
var chooseRandom = true;

// Cookie domain for the cookie set on the browser.
var cookieDomain = ".<your_domain>.com";

/*--------- DO NOT MODIFY BELOW THIS LINE ---------*/

var activeCampaigns = window.optimizely.get("data").campaigns;

var campaignStates = window.optimizely.get("state").getCampaignStates();

var chosenCampaign = "";

groupName = groupName || "groupA";

// Create namespace for cookie.
groupName = "optimizely__" + groupName;

// Cookie Setter Helper Function.
function setCookie(c_name,value,exdays,c_domain) {
    c_domain = (typeof c_domain === "undefined") ? "" : "domain=" + c_domain + ";";
    var exdate=new Date();
    exdate.setDate(exdate.getDate() + exdays);
    var c_value=escape(value) + ((exdays==null) ? "" : "; expires="+exdate.toUTCString());
    document.cookie=c_name + "=" + c_value + ";" + c_domain + "path=/";
}

// Cookie Getter Helper Function.
function getCookie(name) {
  var match = document.cookie.match(name+'=([^;]*)');
  return match ? match[1] : undefined;
}

// Populates campaignArray with all active campaigns if it was left empty.

if (campaignArray.length == 0) {
  for (key in activeCampaigns) {
    if (activeCampaigns.hasOwnProperty(key)) {
       campaignArray.push(key);
    }
  }
}

// Check if group cookie has been set (returning visitor). If yes, use cookie value stored there.
if (!!getCookie(groupName)) {
  var cookieValue = getCookie(groupName);

  // Clear the group cookie if it's not a part of activeCampaigns (i.e. it's been paused).
  if (Object.keys(activeCampaigns).indexOf(cookieValue) === -1) {
    chosenCampaign = chooseRandom ? campaignArray[Math.floor(Math.random()*campaignArray.length)] : currentCampaign;
    setCookie(groupName, chosenCampaign, 120, cookieDomain);
  } else {
    chosenCampaign = cookieValue;
  }

} else {  // If group cookie isn't set, then this is a new visitor.
  chosenCampaign = chooseRandom ? campaignArray[Math.floor(Math.random()*campaignArray.length)] : currentCampaign;
  setCookie(groupName, chosenCampaign, 120, cookieDomain);
}

// Check if the current experiment matches the global experiment. Return boolean.
chosenCampaign == currentCampaign;

Next, make three more changes to this code:

  • Change the campaignArray array to list the campaign IDs of all involved experiments. It's important to note that when Optimizely X creates an experiment it also creates an "invisible" campaign. For this solution, you must use the campaign ID associated with the experiment. This campaign ID can be found within the experiment builder under API Names.

API Names location

Below is an example of what the code should look like:

var campaignArray = ["1111111111", "2222222222", "3333333333"];
  • Change the currentCampaign variable to reflect the current experiment (which this audience will belong to).

var currentCampaign = "1111111111";
  • Change the cookieDomain to match the domain for your site.

var cookieDomain = ".optimizely.com";
  • Repeat this process for each mutually exclusive experiment. Each experiment must have its own audience or else this solution will not work.

Optional Changes

If you have distinct groups of experiments that you wish to exclude from one another, then you must use the optional groupName parameter to specify a name unique to each campaignArray. By default, the groupName is set to be "groupA".

Once this is set up, one experiment within the exclusion group will be chosen randomly to execute for each visitor. This will also act to exclude each visitor from any experiment in the exclusion group other than the one they have already been exposed to. The visitor will continue to be included in that experiment unless the audience conditions change. However, if the traffic allocation settings of the selected experiment dictate someone is excluded from that experiment, they remain eligible for the others in the exclusion group.

This mutual exclusivity does not stretch across the groups. For example, if you specify a groupA and a groupB of experiments, a visitor can be bucketed into one experiment from each exclusion group. Further, for other experiments outside of your exclusion groups, a visitor may also be bucketed into them.

Important Usage Notes

  • This solution is designed to work with single experiments in Optimizely.

  • Starting new experiments will not affect the bucketing of existing visitors.

  • Pausing an experiment and removing the campaign ID from the `campaignArray` will cause visitors originally included in that test to be re-bucketed into a new mutually exclusive experiment.

  • The `campaignArray` must always be maintained between different audiences and it should match the campaigns that are running. It should include all campaigns for a given exclusion group.