• Decide whether it makes sense to run headline testing on your site
  • Reach significant wins in your headline testing program
  • Avoid common pitfalls in headline testing

STEP 1: Decide If and Where to Test

STEP 2: Technical Implementation - Create Headline Tests at Scale

STEP 3: Make Your Headline Testing Program Successful


Decide If and Where to Run Headline Testing

To decide on where and whether to test headlines at all, a good first step is gather some analytics data about your site. The two main factors are baseline conversion rates on articles as well as traffic.

Current Average Conversion Rates on Articles

Gathering this data can already be a little challenging, since there are unknowns such as ...

  • Which conversion rate are we talking about (pageviews, clicks?)
  • Which headlines in which section of your site are we talking about? 

So lets take this a step at a time.

Type of Conversion to Measure:

What type of conversion (click on headline, pageview) we should measure depends on the hypothesis that we have for headline testing. A hypothesis is generally structured the following way:

"If ____, then ____, because ____."

In our case, the hypothesis will look similar to the following:

If we change this headline/picture to ...., then more of our users will click on it, because it appeals more to the user (stands out more, makes the user curious, ....).

This already gives us the answer to the first question of what we should measure. We should measure the immediate impact that the change of the headline might have, which is whether the user clicks on that headline or not. It might make sense to additionally track the pageviews on the article as well. Keep in mind though that there are mostly several routes that the user can take to reach an article page, even after the user has been in the test.

Where to Measure Conversion Rates:

To figure out where to test, it is good practice to measure current conversion rates on all articles that might be relevant for headline testing. To achieve this, either gather your own analytics data or set up a tracking experiment that measures clicks on relevant headlines. 

Make a Decision IF and Where to Test

Let us look at an example to clarify how to make this decision.

Given the following site and analytics data, where do you think it makes sense to run Headline Testing?


Unique visitors per day: 300.000

Green Area:     13% conversion rate

Red Area:         7% conversion rate on average

Blue Area:        3% conversion rate on average

Yellow Area:     1% conversion rate on average


Screen Shot 2016-06-24 at 3.45.16 PM.png


A common answer that we get from customers is:

"I'll test it out on the smaller headlines to see how well it works for me. If I see value, I will also run Headline Testing on the big headlines".

Let's see what our sample size calculator has to say to this. We'll start with the smallest headlines with an average conversion rate of 1%.


This calculation from our sample size calculator tells us that we would need 380.000 visitors in a test with one variation, given a baseline conversion rate of 1% and an MDE of 10%. That means, even if the uplift or drop in conversion rate in the variation is 0.1%, we would not be able to measure it in a single day. Given that a headline usually only lasts a few hours or a maximum of a day, it does not make sense to run Headline Testing on these headlines.

We can do the same calculation for the other areas of the site and will receive the following results (given a 10% uplift or drop measured):

Green Area:     Given a 13% baseline conversion rate, we need a minimum of 18.400 users per test

Red Area:        Given a 7% baseline conversion rate, we need a minimum of 40.000 users per test

Blue Area:        Given a 3% baseline conversion rate, we need a minimum of 108.000 users per test

Yellow Area:     Given a 1% baseline conversion rate, we need a minimum of 380.000 users per test

Concluding, it is important that you inspect your traffic and conversion rates on the sites that you want to run Headline Testing to make a decision which areas you want to test in. The goal should be to reach significance on these tests as quickly as possible to benefit from the possible uplift for as long as you can.


Technical Implementation

Once you made a decision on whether and where to run your headline testing program, you will likely want a more scalable way of implementing tests. While the usual approach of using the visual editor and our usual testing flow is doable, you will find that this becomes cumbersome when you want to run several headline tests a day (which you should to maximize impact and learnings). I will outline four different approaches to running Headline Testing in a more scalable way.


Approach 1 - Using the Headline Testing App

You heard correctly. There is an app for that! With the recent addition of apps to Optimizely, you have the opportunity to run headline testing in an easier fashion right within your Optimizely account. It does not give you freedom to limit the test to certain pages and specify custom goals but is a great way to kickstart your headline testing program.


Approach 2 - Using the REST API  / Building a CMS Integration

If you want full flexibility and want to fully scale out your Headline Testing program, using our REST API is a great option. If you head over to our developers page, you will find that you can automate everything from creating headline tests right out of your CMS to implementing the winning variation as soon as a test has reached significance without even pressing a button. This will require developer resources on your end to integrate your CMS with our APIs but will maximize the impact the program has once you verified Headline Testing works for you.


Approach 3 - Take part in our Headline Testing Quickstart Program

For customers with select plans, we are able to offer a quickstart program to get you up and running with Headline Testing in no-time. We provide help with the technical implementation getting your editors on board and coming up with impactful headlines. Please contact your account manager if you are interested in this program.


Approach 4 - Using our Wordpress Integration

For those of you using Wordpress as a CMS, we have a standard Wordpress integration, which will do most of the work for you. Check out the article in our knowledge base if you want to know more.


Make Your Headline Testing Program Successful

Deciding if and where to test and setting up the tests from a technical perspective is only part of the way. Figuring out how to write headlines and which headlines to test against each other is much harder than you might think. Especially if you want the tests to be impactful and maximize your learnings. There are several factors that we identified that make or break a Headline Testing Program.

Editor Buy-in / Motivation

If we had to choose the factor that contributes most to Headline Testing, it would be this one. Without the buy-in from your editors you will likely not get far. After all, they are the ones that know your readers best and that can come up with impactful headlines. In many cases, they will be the ones having the task of setting up the headline tests on a daily basis. 

Make your editors drive the headline testing program. They are closest to the reader and might already know what approaches to test when coming up with headlines. Reduce friction by making setting up tests as easy as possible. Communicate results within the team to drive curiosity for running more tests.

Approach to Coming up with Headlines

This goes hand in hand with the first factor that we identified. If headline testing is something that is pushed top- down to editors, they will likely not put a lot of creativity into coming up with impactful headlines. What we see a lot in this case is the following.

Original:     "These Tips Will Help You Become Successful in Your Job"

Variation:    "The Following Tips Will Help you Become Better at Your Job"

Not a very creative change right? The core message of the headline stays the same. We essentially just changed a few words in order to create a variation. In all likelihood, you will not see a significant change in conversion rate during the time the headline is actually online. This will not be something new to experienced testers but we see this mistake being made frequently. The reason is mostly missing buy-in from the team that "has to" set up the headline tests.

Instead of moving words around or using synonyms, try creating a headline that approaches the article from a different viewpoint. There are already a multitude of articles out there which can give you ideas of how to come up with impactful headlines. 


In order to learn from and improve your testing program, it's paramount you document your findings. Write down which approaches work best with your readers and try to validate and implement them consistently in your headline tests. Only then can you learn from your Headline Testing program and improve overall conversion rates over time.

Results Reviews

Regularly meeting up to discuss especially impactful (and insignificant) tests is a good way to improve your testing process and approach to Headline Testing. Ask yourself why certain headlines worked and why others didn't or why a certain change did not have any impact. Doing this regularly will improve the knowledge you have about your readers and maximize impact of your program.