Skip to main content
menu_icon.png

Everything you need to switch from Optimizely Classic to X in one place: See the Optimizely X Web Transition Guide.

x
Optimizely Knowledge Base

Events from paused experiments, stopped variations and archived experiments

relevant products:
  • Optimizely X Web Personalization
  • Optimizely X Web Recommendations

THIS ARTICLE WILL HELP YOU:
  • Add one line of code to your site to start running Optimizely experiments (A/B tests)
  • Choose the pages where you'd like to add the Optimizely code snippet
  • Find your project code snippet in Optimizely
  • Decide where to add the code in your HTML

There are times when it is necessary to pause an experiment. Before doing so, you should consider what will happen to the user experience for visitors who were already bucketed into the experiment.

Paused Experiments

If you pause a running experiment in Optimizely X, the variation code is removed from the snippet. This means the experiment and variation code are no longer available to run on the page because the code has been removed from the snippet.

Expected behavior of experiment:

  • New visitors will not be evaluated by Optimizely for the experiment, meaning no new visitors will be bucketed into the experiment once the experiment has been paused.

  • Visitors who were already bucketed into the experiment while it was running will then see the native web page, with no experiment or variation code added.

Restarting Paused Experiments

If you restart a paused experiment, the experiment and variation code are both added back to the snippet. Visitors who had been bucketed into the experiment while it was originally active will once again see the variation they had seen before. Returning visitors will not be re-bucketed into a new variation because their bucketing information, which is stored locally in their browsers, will not change if the experiment is paused.

Results Page

When an experiment is paused, the Results page will stop tracking results for it. Tracking will resume when the experiment is restarted.  Events scoped to the Page that were triggered during the time the experiment was paused will appear on the results page. 
For events scoped to the experiment the results will not have been triggered while the experiment was paused, and thus the results page can show a gap during the time the experiment was paused.

Network Events and Raw Data

Visitors who were already bucketed into the experiment can continue to trigger events while the experiment is paused. If you are QA'ing your experiment and were bucketed into the experiment while it was running, then certain events (specifically, pageview, click events scoped to the page, and custom events) will continue to be triggered and visible in the network tab.

Events scoped to the experiment will not be triggered when an experiment is paused. Only Pageview events, click events scoped to a page, and custom events will continue to be sent to Optimizely when an experiment is paused. These events will not be visible on the results page; however, they are visible in the raw data. If you choose to export your raw data for analysis, be sure to note when an experiment was paused and/or restarted on your own.

Stopped Variation

In some cases, you may need to stop a variation, but are unable to stop the experiment. This is common when a winning variation has been declared, and losing variations have to be stopped. When a variation is stopped, Optimizely will remove both the variation code and the variation ID from the snippet, while leaving the experiment and any other variation information intact.

Expected behavior of experiment
  • New visitors will be evaluated by Optimizely for the experiment and bucketed randomly into the remaining variations. This means no new visitors will be bucketed into the stopped variation.

  • Visitors who were already bucketed into the stopped variation will then see the native web page, with no experiment or variation code added.

Once a variation has been stopped, it cannot be restarted.

Results Page

The results page will continue to collect data for the experiment. However, you should expect conversions for the stopped variation to decline

Updating visitor bucketing information

Visitors who were bucketed into the stopped variation will continue to attribute events to the results page until they navigate back to a URL where the experiment is active or running. Only then will the bucketing information be updated with the removal of the variation ID. At that point, events will no longer be attributed to the variation ID on the results page.

Network Events

If you are QA'ing a variation that has stopped, you may continue to see events triggered in the network tab. Open the network event, Navigate to the page where the experiment activates and open the network event. If the variation ID field is NULL, the bucketing information has been updated.

Screen Shot 2018-01-08 at 2.52.38 PM.png

Raw Data

When viewing the raw data for an experiment with a stopped variation, you may see data points with a value of NULL under the variation ID column. These events are from visitors who were bucketed into the variation that had stopped, but still continued to trigger events. When you see events with NULL values for variation ID, the visitors who created these events will not be attributed to the results page. The event will only be visible in the raw data.

The screenshot below shows a single visitor's activity in raw data. When the variation ******50923 was stopped, the variation_id value became null; meanwhile, the rest of the values remained the same.

Screen Shot 2017-12-28 at 10.33.50 AM copy.png

 

If an experiment that includes a stopped variation is duplicated, the duplicate version will also contain the stopped variation. To get around this, a stopped variation can be duplicated within an experiment. This will allow the code from the stopped variation to be run again without having to rebuild it. However the duplicated variation will have a new ID, and none of the bucketed visitors from the stopped variation will transfer to the duplicated copy.

Archived Experiments

When an experiment is archived, the results page will record a new stop date and time, and will cease tracking events for the experiment. The raw data will continue to receive custom events and events scoped to the page. The reason for this is that Optimizely does not change the visitors bucketing information, which is stored in the visitor's browser. As a visitor continues to trigger events after an experiment is archived, those events will be attributed to all experiments listed in the visitors layerStates, including archived experiments. This means that events not scoped to the experiment are still recorded in the raw data after an experiment is archived.

Related articles