Skip to main content
menu-icon.png

 

x
Optimizely Knowledge Base

Stats Engine: How and why statistical significance changes over time

RELEVANT PRODUCTS:
  • Optimizely X Web Experimentation
  • Optimizely X Web Personalization

THIS ARTICLE WILL HELP YOU:
  • Understand the reasons statistical significance can change over time
  • Read and interpret the Results page

Optimizely’s Stats Engine uses sequential experimentation, not the fixed-horizon experiments that you would see in other platforms. This means that instead of fluctuating, statistical significance should generally increase over time as Optimizely collects more evidence. Stronger evidence progressively increases your statistical significance.

Optimizely collects two main forms of conclusive evidence as time goes on:

  • Larger conversion rate differences

  • Conversion rate differences that persist over more visitors

The weight of this evidence depends on time. Early in an experiment, when your sample size is still low, large deviations between conversion rates are treated more conservatively than when your experiment has a larger number of visitors. At this point, you'll see a Statistical Significance line that starts flat, but increases sharply as Optimizely begins to collect evidence.

In a controlled environment, you should expect a stepwise, always-increasing behavior for statistical significance. When the statistical significance increases sharply, you’re seeing the experiment accumulate more conclusive evidence than it had before. Conversely, during the flat periods, the Stats Engine is not finding additional conclusive evidence beyond what it already knew about your experiments.

Below, you'll see how Optimizely collects evidence over time and displays it on the Results page. The area circled in red is the "flat" line you would expect to see early in an experiment.

201241868.png

When statistical significance crosses your accepted threshold for statistical significance, we will declare a winner or loser based the direction of the improvement. Learn more about the stepwise increase in our community discussion on step-wise increase.

Corrections due to external events

In a controlled environment, Optimizely's Stats Engine will provide a statistical significance calculation that is always increasing. However, experiments in the real world are not a controlled environment, and variables can change mid-experiment. Our analysis shows that this happens rarely, in only about 4% of experiments.

If this happens, the Stats Engine may lower its statistical significance calculation. If the statistical significance lowers, it is because Optimizely has seen evidence strong enough to support one of two possibilities:

  • There was a run of data that looked significant at first, but now Optimizely has enough additional information to say that it probably isn't

  • There was an underlying change in the environment that required a more conservative approach