relevant products:
  • Optimizely Classic

This article will help you:
  • Test headlines outside of Optimizely using the API
  • Interpret the results of your headline tests

With Optimizely's WordPress plugin, you can test WordPress headlines programmatically. You can write alternate headlines for blog posts and test them directly from the WordPress Editor instead of through Optimizely.

This is done through our REST API, which helps you seamlessly integrate Optimizely with other platforms.

If you'd like to implement the Optimizely snippet on your site using Wordpress as a tag management system, check out this article.

Set up the Optimizely WordPress plugin

After you install the Optimizely WordPress plugin, there are a few extra steps to take to start testing headlines.

  1. The default type of content on WordPress themes and setups is Posts. However, many plugins and themes use custom post types that allow you to configure different types of posts.

    Select the Post Type you want to conduct headline testing on.

  2. The default variation code created by the plugin is based on the class name (as described in this article). If you made the recommended changes by adding class names to all your post links and headlines, you can skip this step. If you didn't, you need someone who knows jQuery to write some code based on this example:

  3. If you can’t edit your HTML, you can still conduct headline testing. But, it will take a little more preparation in the Variation Code box. Optimizely uses jQuery to make changes to your webpage—we use the .text() function to make the change. When writing your jQuery code, use this format:

    $(‘css selector’).text(“$NEW_TITLE”);

    A CSS selector is the part of a CSS rule set that actually selects the content you want to style. If you aren’t sure what a CSS selector is, you might need to involve someone who understands CSS to make sure the code is set up correctly.

    You also have three dynamic variables you can use when writing your code:

    $POST_ID This will be replaced with the Wordpress ID of the post that is being tested.
    $OLD_TITLE This will be replaced with the Original Title of the post you are testing.
    $NEW_TITLE Depending on which variation of the headline the visitor will see, this will be replaced with the variation of the title you are testing.

    A single headline usually appears in multiple places on your site, so it’s important to add all possible CSS selectors where the headline can be viewed. Add each possible combination on separate lines:



    Your CSS selector MUST include a unique element. Many themes use the WordPress Post ID somewhere in the HTML to make it unique. If your theme does not add a unique ID of some kind to each post, you MUST use the method above and have your web developer add a class of optimizely-$POST_ID to the places your headlines are displayed.

  4. Configure the Activation Mode. You want to change the headline everywhere it's shown on the website. To do this, you need to run the experiment on all pages of your site (since the headline usually appears in multiple places). This is great for making sure everyone sees the proper headline, but if you run the experiment across your entire site, there's a chance that visitors who never saw the headline will be included in the experiment, skewing your results.

    Select Conditional Activation to ensure that only visitors who actually see the headline will be included in the experiment. The default JavaScript for conditional activation is based on the method above by creating a class name on all of your posts: .optimizely-$POSTID. However, you can change the code to fit your theme. If you can’t make the changes to your class names, we suggest using immediate activation so all headlines are changed. For more information on activation modes and conditional activation, check out this article on Activation Modes.

  5. Choose the maximum number of variations you’ll allow authors to add (they can add fewer variations, if they choose). The more variations you create, the longer it will take to reach statistical significance.

    To get an idea of how many visitors you will need to reach statistical significance, use our A/B Test Sample Size Calculator.

Create new posts

Now that you've configured Optimizely with WordPress, you can start your first headline test!

Authors can add different variations of their headlines directly into their posts so that their experiments integrate seamlessly with their workflows.

  1. Start a new post and add a headline . In the right column, you’ll see a module called A/B Test Headlines. Place as many different variations of the headline as you allowed in the settings. You can also leave some of the variations blank.



    Remember, the more headlines you test, the longer it'll take to reach statistical significance.

  2. When you're ready to publish your post, click Publish.

  3. After you publish the post, a Create Experiment button appears below your headline variations. Click Create Experiment to create the experiment in Optimizely (but not start it).

  4. We recommend that you select View on Optimizely to quality-check your experiment and make sure everything is set up correctly. Then, when you’re ready, click Start Experiment.

Now that your experiment is running, it’s live to the world! If you need to pause your experiment for any reason, click Pause Experiment.

View and interpret results

Now that you have an experiment running, click View Results to go directly to the Results page. Depending on the amount of traffic your site gets, you should start seeing visitors within a few minutes. If you do not see any visitors after 5 minutes, and you know that your site is getting traffic, you might want to check a couple things:

  1. Make sure the experiment has started: log in to your Optimizely Home page and navigate to the project you are using for this experiment. It should say “Running” in green, next to the experiment name. Experiment names are based on the original headline.

  2. Confirm the URL targeting was created correctly.

If both of these are set up correctly, submit a support ticket on the plugin page: https://wordpress.org/support/plugin/optimizely.

The Results page gives you valuable information about the experiment:

  • Goals: By default, we include two goals that you can select:

    • Views to Page: Measures the total number of unique visitors who saw the headline and ended up viewing the post’s detail page.

    • Engagement: Think of this as the inverse of bounce rate. Of the visitors who were included in the experiment, engagement measures the number of visitors who that clicked somewhere on the page.

      You can add your own custom goals in Optimizely, and they will appear in the Goals dropdown menu for you to select.

  • Pause/Start Experiment: Click to pause or start the experiment (depending on the status of the experiment). 

  • Edit: Takes you directly to the experiment in Optimizely in case you need to make changes to the code.

  • Full Results: Takes you directly to the experiment’s Results page in Optimizely.

  • Archive: Archives the experiment. Archiving will stop the experiment from running. 

  • Variation: The name/variation of the headline being tested. Clicking will take you to the post detail page using a force parameter so that you can see the different variations on your live site.

  • Visitors: The total number of unique visitors who were included in each variation of your experiment. 

  • Conversions: The total number of visitors who converted on the selected goal.

  • Rate: The percentage of visitors who saw the headline and converted on the selected goal.

  • Improvement: The relative improvement in conversion rate over the baseline (usually your original headline).

  • Confidence: The level of confidence that the Optimizely Stats Engine defines as the probability that the variation is a winner or loser.

  • Visitors Remaining: The estimated number of visitors the variation needs to become statistically significant.

  • Launch: Replaces your original headline with your variation headline on your live site and archives the experiment.

Optimizely uses a sophisticated algorithm to determine whether an experiment has reached statistical significance, so you don’t have to guess. We place experiments into one of two categories:

  1. Ready for Review: The Optimizely Stats Engine declared at least one winner, and it's up to you whether to launch that winner with the Launch button.

  2. Not Ready Yet: The experiment needs more visitors to accurately determine a winner. Look at the Visitors Remaining column to see how many visitors we estimate you will need per variation to become statistically significant.

Common questions

Q: If I use Wordpress.com VIP, can I use the plugin?

A: Yes. Our plugin is approved for use with WordPress.com VIP. We are working to get the plugin into the official store. For now, you can download the plugin from our page, add it to your site, and have it approved as you would any other change to your site. If the plugin isn’t approved for any reason, please file a support ticket on the plugin page: https://wordpress.org/support/plugin/optimizely

Q: Can I A/B test images and summaries as well as headlines?

A: The current version of the plugin does not allow for A/B testing images or summaries. We are working on adding this functionality.

Q: Can I test headlines on social media sites like Facebook and Twitter?

A: Unfortunately, you can only have one headline per link on social media sites. If the post is not time-sensitive, we recommend testing the headline on your site first, then sharing the winning headline on your social media channels.

Q: How do I add other goals (besides engagement and views)?

A: After you create an experiment, you can go to Optimizely and add more goals to your experiment. The goals you add will be available to select in the dropdown menu on the Results page. Learn more about setting goals in this article.

Q: Can I limit the number of visitors included in the experiment using traffic allocation?

A: The experiment will include 100% of your visitors. Depending on how many variations were created, visitors will be distributed evenly among the variations. To learn more about adjusting traffic allocation, check out this article.

Q: How many headline experiments can I test at once?

A: We have not seen a limit to the number of experiments you can run at once. However, each experiment you create adds to the size of the Optimizely snippet, so be mindful of this if you begin to see headlines flash or experience slower-than-normal page load times.

Q: How long should I run my test?

A: It depends. The most important thing to consider is the number of monthly unique visitors (MUVs) for your account. You don’t want to use up all of your MUVs on one test trying to reach statistical significance! Most experiments will not return conclusive results—what you’re looking for are the big wins. We recommend that if you do not reach statistical significance within 10,000 visitors per variation, and the estimated “visitors remaining” is still more than 2,000, it’s time to archive the experiment and move on to the next one.

It’s also important to consider how long it takes for your site to get to 10,000 visitors per headline variation. Some sites can do it in a few minutes, and other sites may take weeks. Estimating how much traffic you expect to your site will help you determine how many headline variations you should test. Remember, the more variations you test, the longer it will take to reach statistical significance.

Q: I launched the winning headline, so why does my URL still have the old title?

A: Most WordPress themes use the title of the post to add a search engine optimization (SEO)-friendly URL. When you launch the winner, Optimizely doesn’t change this URL to prevent broken links and redirects, which can hurt SEO.

Q: Can I create experiments that don’t involve headlines?

A: Yes, you can! However, the plugin was built exclusively for headline testing. If you want to create other types of A/B tests for your site, use Optimizely to set up an experiment like normal. Learn more about setting up an experiment in this article.

Q: Why do I only see results for my headline tests, and not for all my other experiments?

A: The plugin only adds the results for headline testing experiments that were created through the plugin. We plan to add results for other experiments in the future.

Q: I don’t use WordPress. Can I still test headlines using Optimizely?

A: Yes! If you’re not using WordPress, you can create your own tests. For developers, we also created resources for building your own tools.

 
Tip:

Are you a WordPress developer? Do you have an idea for a feature, or did you find a bug? The Optimizely WordPress plugin is an open-source project, and we would love your help in making this the best A/B testing plugin on the WordPress store. You can find the code here. Make a pull request, and we will contact you about adding it to the project!