Skip to main content
menu_icon.png

Everything you need to switch from Optimizely Classic to X in one place: See the Optimizely X Web Transition Guide.

x
Optimizely Knowledge Base

Multivariate tests for Optimizely X

This article is about Optimizely X. If you're using Optimizely Classic, check this article out instead.
 
relevant products:
  • Optimizely X Web Experimentation
  • Optimizely X Web Recommendations
  • Optimizely X Web Personalization

THIS ARTICLE WILL HELP YOU:
  • Create and enable a new experiment that measures simultaneous changes to multiple page components
  • Edit and modify your multivariate experiments

Multivariate tests allow you to change several elements on your site and determine which combinations of changes performs best.

Creating a MVT is similar to creating a standard A/B test. The key difference is that you will define multiple sections in a MVT. You can think of each section as a self-contained A/B test that modifies a single element, like a button or image. By creating multiple sections, you can test all these elements at once.

Each white card represents a section, and you can create multiple variations within each section. As you build variations, Optimizely automatically generates all the combinations of these sections. Each combination pulls a variation from each of the sections you’ve already defined and rolls them together into a composite experience for your visitor. The combinations included in your MVT are visible in the Combinations tab.

 

The combination names are always made up of a series of capital letters. Each of these letters represents a variation from a particular section. For example, a MVT with three sections would have a combination named ABA. This combination would consist of the first variation of the first section (A), the second variation of the second section (B), and the first variation of the last section (A).

Below each combination name, you can also find a clear description of the different variations included in it. For example, in the image above, the combination labeled ABA is testing a combination of a specific headline (“Text A”), a particular text color (red), and a particular product shot (shoes).

Create a new multivariate test

  1. Navigate to the Experiments dashboard.

  2. Click Create New...

  3. Select Multivariate Test from the dropdown. The New Multivariate Test modal window will appear.

  4. In the Name field, enter a name for your experiment.

  5. In the Experiment Description box, you can provide a description of this experiment. This field is optional.

  6. Next, set up targeting to define where the experiment will run. From the Target By dropdown list, select either URL or Pre-existing pages:

  • If you choose URL, enter the address of the target page in the text box

  • If you choose Pre-existing pages, select the target page from the list of pre-existing pages.

  1. Next, add the audiences you want to include in your experiment.

You can combine multiple audiences in your experiment using AND and OR conditions. For example, you can target the experiment to visitors who qualify for the "Social Butterflies" audience AND the "Luxury Travelers" audience; a visitor must qualify for both to be included in the experiment. Or you can target the experiment to ("Social Butterflies" AND "Luxury Travelers") OR 5x Buyers. In this case, a visitor only needs to meet one set of qualifications or the other to enter the experiment.

  1. You also have the option to add the metrics that measure success for your experiment. Either create a metric from an existing event, or create a new event for the experiment.

  2. The final step in creating a multivariate experiment is to name each section and add as many variations as you need for each of them. Then set the traffic allocations for each variation.

The number of possible combinations in a MVT is capped at 64. This works out to, at most, six sections with two variations each (2^6, or 2x2x2x2x2x2). You should also bear in mind that as the number of combinations in a MVT increases, the amount of traffic going to each one decreases, and results take longer to achieve.

Edit variations

Once your experiment is set up, you can access your variations from either the Sections or Combinations tabs. The Sections tab displays a complete list of the sections you created for your experiment, along with all the variations contained within each one. Just click on the name of the variation or the Edit button to view and update the variation within the Visual Editor.

You can view the actual outputted combinations themselves from within the Combinations tab. Just click on the name of the combination you want to view to open it in the Visual Editor.

Change traffic allocation

You can adjust the amount of traffic allocated to each variation; however, bear in mind that this will affect the amount of traffic delivered to each combination.

To determine the percentage of traffic allocated to a particular combination, Optimizely multiplies each of the variations' percentages together. For example, in a two-variation combination, if Variation 1 has a traffic allocation of 50% in section 1 and Variation 2 has an allocation of 25%, the total traffic allocation for that specific combination will be 12.5% (.50 x .25 = .125).

Currently, Optimizely tests every combination of variations against all the other potential combinations. This is known as the full factorial approach, which has the advantage of being the most rigorous. The drawback of full factorial is that it requires higher levels of traffic to generate a significant result.

Future releases will include the ability to employ a partial factorial approach, in which you can run tests that involve only certain combinations (for example, you can prevent testing a blue button against a blue background). You will also be able to set up your own test layouts, like Taguchi templates that mix variations in specific ways that ensure equal representation, but do not require all combinations to actually be generated. The advantage of partial factorial is that you can test far fewer combinations than full factorial, while still enjoying many of the same benefits.

Additionally, future releases will also enable auto allocation of traffic between combinations, so that you can find the most effective combination even more quickly.

Results

Results for an MVT work like a typical experiment, with data gathered for each of the combinations.

MVT results are reported at the combination level. In the future, these results will also roll up at the section level, so you can analyze how each variation in a section performed against the others, summing up across all the combinations it appears in.

Multivariate testing FAQs

Can I link MVTs to ideas within Program Management?

Not yet; however, that capability will be included in a future release.

Are MVTs available in Full Stack?

Currently, MVT is only generally available for Optimizely Web. We are beta testing MVT for Full Stack as part of variables in Feature Management.