relevant products:
  • Optimizely X Web Experimentation
  • Optimizely X Web Personalization
  • Optimizely X Web Recommendations
  • Optimizely Classic

THIS ARTICLE WILL HELP YOU:
  • Understand Optimizely's Stats Engine, its calculations, and how it affects your results
  • Distinguish between our Stats Engine and other methodologies

When you run experiments, Optimizely determines the statistical likelihood of each variation actually leading to more conversions on your goals.Why does this matter? Because when you look at your results, you’re probably less interested in seeing how a variation compared to the baseline and more interested in predicting whether a variation will be better than baseline when implemented in the future. In other words, you want to make sure your experiment results pay off.

Optimizely’s Stats Engine powers our statistical significance calculations. It uses a statistical framework that is optimized to enable experimenters to run experiments with high statistical rigor while making it easy for anyone to interpret results. Specifically, Stats Engine allows you to make business decisions on results as experiments are running, regardless of preset sample sizes and the number of goals and variations in an experiment.

As with all statistical calculations, it is impossible to predict a variation's lift with certainty. This is why our Results page displays Optimizely’s level of confidence in the results that you see. This way, you can make sophisticated business decisions from your results without an expert level of statistical knowledge.

Optimizely is the first platform to offer this powerful but easy-to-understand statistical methodology. Other statistics frameworks don’t make this so easy.

How Stats Engine works

To learn about the nuts and bolts of how Stats Engine works and how to interpret the results it generates, check out these articles:

Statistical concepts and techniques

You may also find it useful to read up on the statistical concepts and techniques Stats Engine uses, which we describe in these articles:

Further reading 

If you'd like even more background on how Optimizely's Stats Engine works, here are a few resources: