- Optimizely X Web Experimentation
- Optimizely X Web Personalization
THIS ARTICLE WILL HELP YOU:
- Exclude outliers from your experimental results
In statistics, an outlier is an observation that has an abnormally higher or lower value than other observations in a data set. Outliers can severely skew the accuracy of any analysis conducted on a data set, and can lead to potentially incorrect conclusions. For this reason, outliers are often excluded from the analysis.
Outliers can appear in your Optimizely results, usually as the result of unusual or unexpected behavior by a customer. For example, suppose you are running an experiment that aims to improve average order value for your e-commerce site. Your visitors usually submit orders with an average total value of $200. Now imagine there is a small number of visitors who submitted orders with 100 or even 1000 times higher value than the average. If these extreme orders are included in the result calculations, they could introduce bias into your A/B comparison, and lead you to draw the wrong conclusions from your experiment.
For this reason, Optimizely gives you the option to exclude outliers from your experiment results. This feature is currently available for revenue metrics in A/B experiments or Personalization campaigns; however, it will also be available in the future for metrics using the “total value” numerator.
How Optimizely excludes outliers
When outlier filtering is enabled, Optimizely first identifies any values that exceed the daily exclusion threshold, which is defined as three standard deviations higher than the arithmetic mean of your metric (
arithmetic mean + (3 * standard deviation)). These extreme values are labeled as outliers. Next, Optimizely replaces these outliers with the metric's arithmetic mean value. This process is known as outlier smoothing.
Optimizely recalculates the daily exclusion threshold each day, using a moving average of the arithmetic mean and standard deviation of your metric over the previous seven days. This process is repeated for each day of the experiment.
During the first seven days of the experiment, Optimizely calculates the daily exclusion threshold using all the available experiment data up to that point. For this reason, changes to the threshold may be more noticeable during an experiment's first week.
Exclude outliers from a metric (BETA)
If your account is white-listed for beta, outliers will be automatically detected and their values will be replaced by the arithmetic mean of the metric. You will see a subtle message in the metric card if outlier filtering is enabled. Contact your customer success representative to request access to the beta.