Skip to main content

We are currently experiencing an issue that is preventing customers from submitting support tickets. Please contact us at (866) 819-4486 if you need immediate assistance.

x
Optimizely Knowledge Base

The Results page for Optimizely X Web Personalization

This article will help you:
  • Learn how Optimizely X Web Personalization calculates results
  • Determine the definitions and formulas for result metrics

The Results page at a glance

The Results page allows you to track how well your campaigns perform. Dive into different experiences and audiences for a granular view of your visitors' behavior and take action.

Key tips

  • A session starts when a visitor lands on your site and ends after 30 minutes of inactivity. A new session is created if there's new activity.
  • By default, Personalization maintains a 5% holdback: we show 95% of visitors in a campaign a personalized experience, and holds back 5% who will see the original. 

What to watch out for

The Results page in Optimizely X Web Personalization helps you track how well your campaigns perform. You'll use it to measure the success of each experience you deliver and explore your visitors' behavior.

Here are two ways to visit your Results page:

  • From the Campaigns dashboard, click the Results button.

  • From the Manage Campaign dashboard, click View Results.

 
Note:

All results in Personalization are in UTC time, not local time.

This article walks you through the Results page for Personalization. Click to learn more about monitoring campaigns and taking action on results.

Jump down to learn about the key terms and metrics you see on your Results page. 

Campaign Summary

The Campaign Summary helps you track your campaign's overall impact. In the sidebar, you can view when the campaign was last published and the number of days it's been running.

Overall Improvement

The Overall Improvement module provides a high-level view of your campaign's progress.

By default, the module shows you the improvement in the campaign's primary metric with a confidence interval. The improvement measures the relative difference between personalized sessions and the holdback (original) experience. It also tells you about the reach of campaign reach: the number of personalized sessions delivered to visitors and the impact of that metric per session.

Use the dropdown to toggle between metrics that are attached to your campaign.

Or, use the calendar dropdown to select a different date range and click Apply.

Campaign Reach

The Campaign Reach module gives you a sense of how many personalized experiences were delivered, by session.

Page reach shows the total number of sessions that occurred on pages that were targeted by your campaign.

Audience reach shows the percentage of sessions that was delivered to one or more audiences in the campaign.

Personalized sessions shows the number of sessions where a visitor saw a personalized experience. Holdback sessions shows the number of sessions where a visitor saw the original experience because they're bucketed into the holdback by chance.

Audience Breakdown

The Audience Breakdown provides a visual summary of impact of the campaign, broken down by audience.

The example above shows three audiences: "Shoe Shoppers," "Cold climates," and "Hikers." 

The width (x-axis) of each bar shows the reach of a particular audience. For example, the "Shoe Shoppers" audience make up 24.48% of total sessions in the campaign. All audiences in a campaign add up to 100%.

The height (y-axis) of each bar shows the change in conversion rate. The "Shoe Shoppers" audience saw $0.81 lift in the primary metric -- Revenue per Session, in this case. For conversion events such as clicks or pageviews, the y-axis shows the conversion rate. Revenue is displayed as revenue per visitor.

The color of the bars shows whether the change in conversion rate is statistically significant:

  • A green bar represents statistically significant lift

  • A red bar represents statistically significant loss

  • A gray bar represents a change that hasn't reached statistically significance

Learn about how Stats Engine calculates statistical significance.

To see more granular data about the primary metric by audience, click More Details.

The Impact column shows the difference in conversions in a personalized experience, compared to the holdback. It's broken down by audience.

Audience Summary

The Audience Summary helps you track the improvement generated by the personalized experience for each audience, broken down by metric.

Click to select an audience and view a summary.

Once you enter an audience summary, you can see how each metric performed for that audience.

The summary will always show your primary metric first (in the example below, the Added to Cart event).

To view a graph of conversions in a specific metric for a particular audience, click View graph.

Segment Results

Optimizely allows you to segment your results by Audiences or Dimensions, so you can drill down into the behavior of certain groups of visitors.

Segmenting your results gives you valuable insights about different groups of visitors, instead of simply analyzing the results across your entire visitor base. Figuring out what experiences work for different audiences will form the basis of your personalization strategy.

Watch this short video about segmentation:

 

Metrics and definitions

Below are a few terms you'll see on your Results page. 

Metric Name

Description

Formula

Sessions

A session starts when a visitor lands on your site, continues as they navigate between pages, and ends after 30 minutes of inactivity -- at which point the next visit creates a new session.

Absolute number

Personalized sessions

The number of sessions where a visitor was shown a personalized experience. This excludes visitors who either (1) don't qualify for any audience in the campaign or (2) qualify for an audience but are randomly bucketed into the holdback (control group).

Sum the treatment size over all Audiences

Reach

The percentage of total sessions that qualified for an audience (regardless of whether those visits were personalized).

Number of sessions for an audience / Total sessions

Conversions

The number of sessions in which a visitor from any campaign audience converted on this event.

Absolute number

Conversion Rate

The percentage of sessions in which a visitor from any campaign audience converted on this event.

Number of sessions where conversions took place for that audience, or for total sessions.

Holdback

Every Campaign has a “holdback,” which is like the control of an A/B test.

You can adjust the holdback for each campaign.

By default, Personalization shows 95% of visitors in a campaign a personalized experience, and holds back 5% who will see the original.

Confidence Interval

The confidence interval gives a range of values in which the conversion rate for a particular experience actually lies.

As you collect more data, the confidence interval will narrow. Once your variation reaches statistical significance, the confidence interval will always lie entirely above or below 0.

 

http://www.optimizely.com/statistics 

Improvement

The relative improvement of this event’s conversion rate for personalized sessions, compared to the holdback.

(Event Conversion Rate for Personalized visits - Event Conversion rate for Control visits) / (Event Conversion Rate for Control visits)

Improvements for an event for each audience

The % improvement in an Event’s conversion rate, comparing a personalized Audience to the control.

(Event Conversion Rate for Audience’s Personalized visits - Event Conversion rate for Audience control visits) / (Event Conversion Rate for Audience control visits)

Total Improvement in Primary Metric

The % improvement  of the Primary Metric across all sessions in the campaign (regardless of whether they fell into an audience).

(Conversion rate for sessions in the treatment - conversion rate of sessions in control) / conversion rate of sessions in control

Add or remove metrics

From the Results page, you can add or remove metrics from your campaign.

Existing events and tags are tracked all the time, so you can add them back if you want. However, if you add more than five events, it may take longer to reach statistical significance.

  1. Click Manage Metrics.

  2. Click () to add metrics, or drag them to remove.

How quickly do results update?

Personalization results update once an hour.

Reset or delete results

You can remove visitors and conversions from your Results page. 

On the Results sidebar, click Reset Results.

At this time, clicking "Reset Results" will reset your results to the state of the last hourly refresh of your results. So even after you click reset you might see conversions and visitors who were part of your Campaign from this timeframe.