Skip to main content
menu-icon.png

 

x
Optimizely Knowledge Base

Multi-armed bandits vs Stats Accelerator: when to use each

In Optimizely, you can run tests to do one of two things: experiment or optimize.

  • When you experiment, you’re trying to test a hypothesis or validate a claim. The goal is to determine whether a variation is fundamentally different (via statistical significance), with the aim of generalizing learnings from that knowledge into future deployments or experiments.

  • When you optimize, on the other hand, you’re using a set-it-and-forget-it algorithm designed to squeeze as much lift from a set of variations as possible, without concern for visibility into whether a variation is fundamentally better or worse. In effect, this means you’re setting aside statistical significance in favor of maximizing your goal within a given time frame.

Stats Accelerator helps you algorithmically capture more value from your experiments by reducing the time to statistical significance. It does this by monitoring ongoing experiments and using machine learning to adjust traffic distribution among variations.

By contrast, multi-armed bandit (MAB) optimizations aim to maximize performance of your primary metric across all your variations. They do this by dynamically re-allocating traffic to whichever variation is currently performing best. This will help you extract as much value as possible from the leading variation during the experiment lifecycle, so you avoid the opportunity cost of showing sub-optimal experiences.

In other words, the better a variation does, the more traffic a multi-armed bandit will send its way. A/B tests don't do this. Instead, they keep traffic allocation constant for the experiment's entire lifetime, no matter how each variation performs:

mab_gif.gif

Click here for a thorough explanation of what's happening in this graph.

When to use a multi-armed bandit

Here are a couple cases that may be a better fit for a multi-armed bandit optimization than a traditional A/B experiment:

  • Promotions and offers: users who sell consumer goods on their site often focus on driving higher conversion rates. One effective way to do this is to offer special promotions that run for a limited time; for those, the changes you're making aren't intended to be permanent, and a multi-armed bandit optimization will send more traffic to the over-performing variations and less traffic to the underperforming variations for the duration of the promotion.

  • Headline testing: headlines are short-lived content which lose relevance after a fixed amount of time. If a headline experiment takes just as long to reach statistical significance as the lifespan of a headline, then any learnings gained from the experiment will be irrelevant going forward. Therefore, a multi-armed bandit optimization is a natural choice to allow you to maximize your impact without worrying about balancing experiment runtime and the natural lifespan of a headline.

  • Webinar: You can boost registration for webinars or other events by experimenting with several different versions of your landing page.

When to use Stats Accelerator

One of the biggest advantages Stats Accelerator brings to your experimentation program is its ability to cycle through many options quickly. For this reason, you should consider using Stats Accelerator whenever you want to run an A/B test that includes more than one variation against the baseline.

Some examples of situations where Stats Accelerator would be the best choice include:

  • CTAs: By testing many different copy options for a call-to-action, you can more quickly optimize a given page for lead completion.

  • Landing pages: Test several combinations of landing page copy, concept, and design at once, so you can quickly optimize for registrations, sign-ups, donations, etc.

  • Add-to-cart rate optimization: Drive an increase in your add-to-cart rate by showing visitors different default images on product pages or in search results.  

  • Search results optimization: Nudge users toward specific options (for example, a travel site might want to encourage visitors to select a specific flight) by showing different results.

  • Drive traffic to specific pages: By changing the location of the “Recommended Content” section, a media site can increase its clickthrough rates on recommended articles.

To set up a multi-armed bandit optimization, follow these steps:

If you haven't worked with multi-armed bandits before, we strongly recommend you first read our article on how to interpret results from a multi-armed bandit optimization.

New to Optimizely? Then check out our article about setting up an experiment in Optimizely.

  1. From the Experiments window, click the Create New... button in the upper right-hand corner.

  2. Select Multi-Armed Bandit from the drop-down menu.

  3. Give your multi-armed bandit a name, description and a URL to target, just as you would with any Optimizely experiment. Then click Create Bandit.

  4. Create at least two variations in the Visual Editor.

  5. Click Metrics from the left-side navigation to choose your primary metric.Your multi-armed bandit will use the primary metric to determine how traffic is distributed across variations.

Once you start your multi-armed bandit, you will not be able to change the primary metric, so choose carefully!

  1. QA your multi-armed bandit to make sure everything is set up correctly.

  2. Launch your optimization by clicking the Start Multi-Armed Bandit button.

To use Stats Accelerator in your experiment, follow these steps:

  1. From the Experiments window, click the Create New... button in the upper right-hand corner.

  2. Select A/B Test from the drop-down menu.

  3. Give your experiment a name, description and URL to target, just as you would with any Optimizely experiment. Then click Create Experiment.

  4. Create your variations in the Visual Editor. For experiments using Stats Accelerator, you should have at least two variations and a baseline.

  5. Click Metrics from the left-side navigation pane to choose your primary metric.Your experiment will use the primary metric to determine how traffic is distributed across variations.

  6. Click Traffic Allocation from the left-side navigation pane. Under Variation Traffic Distribution, click the Distribution Mode dropdown and select Stats Accelerator.

  7. QA your experiment to make sure everything is set up correctly.

  8. Launch your experiment by clicking the Start Experiment button.

This article only scratches the surface on Stats Accelerator and multi-armed bandits. Before using them in your experimentation program, you should read these articles to learn more about when to use each, how they work, and how to interpret results: