- Identify discrepancies in third-party data
- Implement best practices to avoid data discrepancies
Optimizely offers many out-of-the-box analytics integrations in addition to developer tools which enable teams to use custom integrations, and has many partners offering third party integrations. With all of the analytics platforms you can use to measure the impact of your A/B tests, you’ll quickly find out that your numbers will not match perfectly from one platform to the next.
If you’re seeing a discrepancy in your data, it’s important to make sure you’re following some best practices. These are to ensure that any involved platforms or datasets are measuring the same thing. If your datasets are not based on the same information, user set, or activity, then they will probably not align.
The vast majority of data discrepancies that we’ve seen came down to root causes covered in this section. If you aren’t comparing the same data, then it probably won’t match up. Below are some best practices to follow to ensure that your data can be compared to Optimizely results. You may find that you can get the results to better align just by making adjustments to your reports.
If you don’t have an integration between Optimizely and your other platforms, there should be no expectation that they will match. Users need to be tagged with Optimizely experiment and variation information when they are exposed to an A/B test. Your other platforms’ data should be filtered to only show that which contains experiment and variation information.
We offer many out-of-the box integrations that can be enabled easily. If there is no integration for the platform you want to integrate with, you or your developers can create a custom analytics integration following our Integrations Developer Guide.
Optimizely Full Stack does not come with any integrations. Read our Integrations with Optimizely Full Stack article for some ideas for integrating your platforms.
Optimizely and other platforms have many options for results segmentation. Drill your results down to different categories of users and attributes such as web browser, location, language, and plan type. Narrow reports to specific date and time ranges, and filter out results from certain IPs. Check the following for consistency from platform to platform:
Optimizely Web Experimentation and Optimizely Full Stack calculate results based on unique user counts, whereas Optimizely Web Personalization computes based on unique sessions. Read more about how Optimizely counts conversions. Other platforms may count results differently, resulting in different counts for users who are tagged as having seen an Optimizely experiment. It’s important to make sure you understand how each platform counts so you can account for any differences.
For example, Optimizely and Google Analytics use a different “visitor” definition.
Google Analytics uses a tracking call that is session based, meaning a single visitor can trigger multiple visits over a given period of time (GA Support Article)
Optimizely, on the other hand, uses a cookie with a 6-month expiration time (from the last visit) and counts unique visitors.
If you have similar visitor counts but different conversion counts for an event, you’ll want to take a closer look at the event in each platform. Two similarly named events aren’t necessarily tracking the exact same action taken by visitors.
For example, you may have a “signup completed” event tracking a form submission. In Optimizely, you may have this configured as a “submit” button click metric or a confirmation page view metric. Each of these events represents the same thing (the user submitted the form), but they are not tracking the same action, which can lead to a discrepancy. The user may have clicked the submit button without filling out the form, or maybe the user submitted the form by pressing “enter”, bypassing the button click event.
Check with your engineers to see how each event is being tracked. Make sure your events track the same things the same way. If they don’t, you will have dissimilar data.
In Optimizely, a user's actions only count toward A/B tests for which they still meet the audience conditions. If your other analytics platform continues to track users after they no longer meet the audience condition, your Optimizely results page may show lower total conversions.
Optimizely’s attribution model may differ from those of other analytics platforms.
Optimizely has decision-first counting, which means that we only count conversions from users who have previously sent us a decision event. When an A/B test activates, the decision event fires. If a conversion event fires before the decision, it wouldn’t be counted on the results page but may be counted in your other analytics platform.
Segmentation values may have differing attribution as well. Optimizely attributes segment values at the user level using the most recent value for each segment in each session.
Discrepancies caused by differences in event tracking probably cannot be aligned without making adjustments and running a new experiment. A new experiment can track new information about users that might aid in a future investigation into the root cause of a data discrepancy. Best practice for these types of discrepancies is to run an A/A test, make adjustments, and run another A/A test to verify that the issue was corrected.
Optimizely Web Experimentation and other analytics platforms offer bot filtering for results, but Full Stack does not. If you’re seeing higher counts in a Full Stack experiment than you’re seeing in your other analytics, it could be that they are filtering out bots and Optimizely is not.
If you want to filter bots for Full Stack, you can use getVariation() for bots and
activate() for real users. That will forgo sending an impression event for any bots, which will cause them to not be counted on the results page.
Many internet users are using content blockers such as Adblock or Ghostery to block trackers and advertisers. These content blockers can block any client-side trackers, including Optimizely Web Experimentation. Content blockers cannot however block any server-side experimentation you do using Full Stack. Because the impression event and tracking happens in the backend, the user's client-side content blocking will not be effective. This could why Full Stack counts more users than a third-party platform. It’s valuable to know what portion of your end users have content blocking extensions enabled.
A difference in counts could be due to the relative distance between the points in time when Optimizely counts the user and when the other platform counts the user. During the page load process, as time elapses, the user's web browser makes requests for resources in the page and sends information to various providers. Eventually, when all requests are completed and your browser renders the page with all of its images, the page will be completely finished loading.
The best practice for Optimizely Web Experimentation is to have your snippet be one of the first resources in your page (near the top of the <head> tag) and to have it load synchronously (blocking). This helps make sure that the visual edits you make in your A/B tests are ready to display before the content exists and prevents the original version of the page from showing before the variation renders (a flash of original content).
Since Optimizely is the first thing to run, we activate A/B tests and get ready for page targeting in the order of activation. We send events as soon as possible, so Optimizely will begin making asynchronous requests to fire impression events and page views. If you’re running Optimizely Full Stack in your web server, you may have events fire before the server responds to the user's browser before any of the resources start to be downloaded.
General analytics scripts do not need to be one of the first things to load. In fact, Google Analytics recommends implementing after the <head> tag. This means that Optimizely counts users a considerable amount of time before most third-party analytics. Because a duration exists between points in time an opportunity for users to close the browser, drop internet connection, or click back and not be counted by the latter platform.
Because of this duration where Optimizely has counted a user but other services have not, users may close the browser tab, bounce, or lose internet connection. This usually manifests as Optimizely having higher counts than other platforms.
Adjust timing in Optimizely Web Experimentation
You can choose when Optimizely should send its events, even with the snippet being one of the first things to load. The default behavior is for Optimizely to send events as soon as possible, but we have a new feature for Optimizely users who want to align their data with another platform may find it worth delaying Optimizely’s events.
holdEvents() – when called, all subsequent events are kept in a queue. Use this before Optimizely to hold all events.
sendEvents() — when called, the queue created by holdEvents() is sent in a batch as one network request.
(times shown for illustrative purposes only)
sendEvents() around the same time as when your other platform fires its impression event.
Adjust timing in Optimizely Full Stack
Disparities may be the most exaggerated when comparing Full Stack results to client-side, after the
<head> implemented analytics because that setup has the largest delay between when the user is counted on each platform.
Note: Times shown are for illustration purposes only.
Full Stack gives you complete control of when to send the impression event. You can activate() which will send the impression event and return the variation, or you can getVariation() which will just return the variation.
In a Full Stack experiment running on a web server, you’ll want to get each user’s variation when the user makes a request for the page. You’ll use the variation key to respond to the request with one page or another page with edits to it.
We want to make sure you have the help that you need in case the tips above don’t point you to anything conclusive. There are three types of integrations you can have with Optimizely, so to get the best support please keep reading. When submitting support requests for data discrepancies and integrations, please provide as much detail as possible including screenshots and code samples. If you’re having an issue with a custom integration, we’ll need to know every detail of how the integration is set up and what numbers you’re comparing to Optimizely.
We may not be able to investigate experiment results older than 180 days due to our internal data retention policies.
Integrations that Optimizely offers out of the box with its Optimizely Web Experimentation product are supported by Optimizely Support. Please file a support ticket to get help with implementation and results discrepancies.
Custom analytics integrations
Custom analytics integrations are a developer API offered and supported by Optimizely for the Optimizely Web Experimentation and Personalization products. Although we are not able to fully support results discrepancies that may occur using these integrations, we are happy to take a look at your implementation and provide guidance using our APIs. We do not offer integrations for Full Stack, so any custom integrations you have between Full Stack and other platforms are not supported by Optimizely.
Many of our partners offer integrations with Optimizely. These integrations are supported by the partners who develop them. Please reach out to the partner’s support team for assistance.