- Optimizely X Web Experimentation
- Optimizely X Web Personalization
- Optimizely X Full Stack
THIS ARTICLE WILL HELP YOU:
- Understand Optimizely's Stats Accelerator, its algorithms, and how it affects your results
- Distinguish between the two Stats Accelerator algorithms
- Determine whether to use Stats Accelerator for your experiments, as well as which algorithm to use
- Enable Stats Accelerator (beta) for your account
Stats Accelerator helps you algorithmically capture more value from your experiments, either by reducing the time to statistical significance or by increasing the number of conversions collected. To do this, Stats Accelerator monitors ongoing experiments and automatically adjusts traffic distribution among variations.
You may hear Stats Accelerator concepts described as the “multi-armed bandit” or “multi-armed bandit algorithms.” The Key terminology section included below clarifies Stats Accelerator terminology and concepts.
Stats Accelerator algorithms
When enabled, Stats Accelerator applies one of two algorithms (or optimization strategies) for the primary metric: Accelerate Learnings or Accelerate Impact. Keep reading to learn more about their differences and use cases.
The Accelerate Learnings algorithm seeks to reduce experiment duration by showing more visitors the variations that have a better chance of reaching statistical significance. Accelerate Learnings attempts to discover as many significant variations as possible.
Accelerate Learnings helps you maximize the number of learnings from experiments in a given time frame, so you spend less time waiting for results.
For Accelerate Learnings, we require at least 3 variations, including the original or holdback (baseline) variation.
The Accelerate Impact algorithm seeks to maximize the payoff of the experiment by showing more visitors the leading variation(s).
Accelerate Impact helps you exploit as much value from the leading variation as possible during the experiment lifecycle, so you avoid the opportunity cost of showing sub-optimal experiences.
For Accelerate Impact, we require at least 2 variations, including the original or holdback (baseline) variation.
If you experiment in high volumes, you face two challenges. First, data collection is costly. Time spent experimenting means you have less time to exploit the value of the eventual winner. Both algorithms solve this problem by reducing time to significance or maximally exploiting overperforming variations during an experiment.
Second, you may worry that creating more than 1 or 2 variations will delay statistical significance too long. Accelerate Learnings allows you to be bold and create more variations, while shrinking the time to significance by quickly identifying the variations that have a chance of statistical significance.
Here are a couple cases that may be a better fit for Accelerate Impact:
Promotions and Offers: users who sell consumer goods on their site often focus on driving higher conversion rates. To do so, many retailers offer special promotions that only last a certain amount of time. Instead of running a normal A/B/n test, Stats Accelerator can Accelerate Impact by sending more traffic to the overperforming variations and less traffic to the underperforming variations.
Long-running campaigns: some Optimizely Personalization users have long-running campaigns to which they continually add variations for each Experience. For example, an airline may deliver destination-specific experiences on the homepage based on past searches. Over time, the airline might add different images and messaging. For long-running Personalization campaigns, the overall objective is often to drive as many conversions as possible, making it a perfect fit for Accelerate Impact.
Stats Accelerator works with Optimizely X Web Experimentation, Personalization, and Full Stack. For Personalization, the Stats Accelerator makes adjustments to the traffic distribution among variations within an experience.
Enable Stats Accelerator
If you want to use Stats Accelerator, please contact your Customer Success Manager.
After Stats Accelerator is enabled for your account, you can find the two algorithms with the traffic allocation settings for your variations.
FDR Control with Adaptive Sequential Experimental Design is a technical white paper on the mathematical foundation of Stats Accelerator.