- Understand how Optimizely executes on a page from a technical point of view, including targeting, traffic allocation, variation code, and event firing
- Understand how Optimizely evaluates variation code for an experiment
- Diagnose timing issues with experiments
This article explains the snippet's order of operations. Knowing this can help you design more effective experiments that don't detract from the user experience of your site. It's also a good place to start troubleshooting any timing issues in your experiments.
Snippet order of activation
When the snippet loads on the page where your experiment lives, it immediately applies API calls in the
activate API call. Optimizely pages include targeting conditions for the pages on your site as well as an activation mode (optional), which determines when the page targeting conditions are evaluated.
When the page's activation trigger fires and targeting conditions are passed, the page activates. Any page, regardless of activation mode, may also be triggered by name at any time by the
page API call.
When a page activates, Optimizely evaluates all the experiments that include that page.
For each experiment that's included, Optimizely evaluates whether a visitor is eligible to enter the experiment. Visitor eligibility is evaluated based on the experiment's traffic allocation and audience conditions.
If any audience conditions cannot be determined for a visitor based on known information (for example, geolocation), Optimizely may wait a short time to gather this information before deciding whether the visitor qualifies for the audience.
If both audience and traffic conditions are passed, the experiment activates.
Each visitor who is allocated to an experiment is assigned to a variation (which may be the control, or "original," variation). Based on the number of variations and the bucketing probabilities described by the experiment's traffic distribution, the visitor is bucketed into a specific variation. A decision event is logged at this stage.
Once a visitor is bucketed into a variation (which determines what changes will be made to the webpage), the snippet executes the code that applies those changes.
Once all pages are evaluated and changes from relevant experiments are applied, the rest of the webpage loads.
Now, when the visitor interacts with the webpage, they may trigger a conversion event that's tracked by Optimizely. Optimizely records the event and associates it with the variation.
If you like, you can follow a snippet's execution by reading the console logs.
How Optimizely executes code
At the variation level, changes from the Visual Editor can be configured to either run in sequence (where making one change depends on a prior change) or in parallel (where all changes execute as soon as the selector is ready). In contrast, variation custom code executes immediately, exactly as written. All timing considerations must be written into the code itself.
Custom variation CSS is added as a style tag to the bottom of the head.
How events are fired
Optimizely tracks events that you configure, such as click and custom events. The snippet also natively emits certain events, such as when a visitor is bucketed; these can be monitored via Optimizely's event listeners. All events are emitted immediately when the triggering event occurs.
However, event tracking for Optimizely's Results service can occasionally be delayed for data analysis and performance purposes. For example, the activation of a page may not be emitted until DOMContentLoaded in order to support the inclusion of visual tags with the event payload.
Here's a more detailed flow chart for internal use:
Google Drawing (clicky clicky if you want to see the unsquished flow chart. I've yet to figure out how to get it to render nicely in MindTouch)
Experimentation and Personalization
In Optimizely, experiences are organized by three concepts: campaigns, experiments, and variations.
Campaigns are the top level of organization. Your snippet can include multiple campaigns, which activate simultaneously.
Experiments are the second level of organization, inside of campaigns. Each campaign can contain multiple experiments, but a page load can only activate one experiment per campaign. This helps ensure that experiences or variations in a campaign are mutually exclusive for visitors. You can prioritize the experiments in a campaign to decide which is shown to a visitor who qualifies.
Variations are the most granular level of organization. When an experiment activates, only one variation activates (as determined by Optimizely's logic). When a visitor is bucketed into an experiment, they'll always see the same variation.
So what does this mean for the experiences you deliver in Optimizely Web? Here are a few examples:
In Optimizely Web Personalization, you create campaigns that deliver targeted experiences to specific audiences. Each Personalization campaign is a campaign in the data model. Each experience that's targeted to an audience is a single experiment (with one variation). This means that each experience in a campaign is mutually exclusive--a visitor never sees more than one experience from the same campaign.
In Optimizely Web Experimentation, you create an experiment that delivers variations to a specific audience by chance. Each experiment is also a campaign in the data model. The campaign contains a single experiment that targets the audience. Each variation in the experiment is also a variation (in the sense of our three concepts). A visitor never sees more than one variation in an experiment.
Web Personalization and Web Experimentation
Campaigns are independent, so Personalization campaigns and A/B tests activate at the same time.
If you want certain personalized messages to be mutually exclusive, create them as multiple experiences in the same Personalization campaign.
The diagram below is an example of how a Personalization campaign and an experiment are organized by the Optimizely snippet. The left-hand column represents what you create in your Optimizely project, and the right-hand column represents what runs on your site.
More specifically, in the right-hand column:
Blue: campaigns, experiments, and variations that are active on a sample page load
White: experiments and variations that can't be active on that same page load
This will have to be updated when Campaigns allow multiple experiments with multiple variations and when the UI is updated to match (instead of "Pick Personalization or A/B Test").
Optimizely only activates campaigns, experiments, and variations when the pages they live on are activated.
To learn more about change execution in Optimizely, check out this article.