- Decide whether to change the statistical significance settings for your projects
- Find and adjust the statistical significance setting for your projects
When you run experiments, Optimizely's Stats Engine constantly analyzes your results to determine whether your results are statistically significant: how confident you can be that the results actually reflect a change in your visitors' behavior, not just noise or randomness. With the statistical significance setting, you can set a project-wide significance level at which you'd like Optimizely to declare significant results (winners and losers) on the Results page.
Think of the statistical significance setting as a match for your organization's risk tolerance. In statistical terms, it's 1 - [p value]. If you set a significance threshold of 90%, Optimizely will declare results when it's 90% sure that you have statistically significant results, which also means you can expect a 10% error rate.
Trade-offs of changing the statistical significance setting
You should be aware of certain trade-offs associated with changing the statistical significance setting. In general, a higher significance setting is more accurate and increases the time required for Optimizely to declare significant results because it requires a larger sample size. A lower statistical significance level decreases the amount of time needed to declare significant results, but lowering the statistical significance setting also increases the chance that some of the results will be false positives.
Changing your statistical significance setting will instantly affect all currently running experiments. If your experiment has a goal with an 85% statistically significant winner, and you change your statistical significance setting from 90% to 80%, the next time you load your Results page, you will see a winner (85% > 80%). Your difference intervals will also shrink to reflect the reduced need for confidence.
Change the statistical significance setting
Navigate to Settings > Advanced. The statistical significance setting is listed under Settings > Advanced. After you adjust the setting, scroll down and click Save. Allow a few minutes for the new setting to take effect.