This article will help you:
- Communicate the results of your tests with others in your organization
- Focus on what to share
- Use different formats and templates to communicate results
Once you have a solid A/B testing strategy in place, you'll want to communicate the results of your program. This article looks at best practices for communicating results effectively throughout your testing organization and to stakeholders. If you do this well, you'll evangelize data-driven decision-making within your organization. We also include flexible email and presentation templates (Keynote, PowerPoint) for sharing results.
While this article primarily discusses sharing test results, we also have an article on taking action from the results you see on the Optimizely Results page.
Optimization programs are about gathering data, drawing insights from that data, and taking action. If test results aren’t widely shared, your company will have fewer insights and taking action on those insights will be less likely to happen. Conversely, the companies that consistently share their test results end up testing more, testing more effectively and move the metrics that matter to their businesses.
Despite this, effectively sharing test results is one of the most overlooked parts of testing. Follow the guidelines below to make sure your company doesn’t overlook this crucial step.
Check out how other customers are using Optimizely and tell us about your experiment success in the Optimizely Community.
Looking for the definitive guide to getting up and running with Optimizely? Learn and practice your Optimizely skills and web optimization strategy, from beginner to advanced, in Optimizely Academy.
What to Share
When you share your results, include the following:
Purpose: a brief description of the thought process behind the test
Details: include the number of variations, the differences between them, the dates the test ran, the total visitor count and visitor count by variation
Results: the numbers including percentage lift (best performing variation against your original), conversion rates by variation, and statistical significance/difference intervals
Lessons Learned: analysis of what the numbers mean, key insights based on the data
Revenue Impact: when possible, quantify the value of a given percentage lift for year over year projected revenue impact
The bullets above do not apply only to test wins. Tests that don’t produce a variation which performs better than original can still generate valuable lessons - learning what not to do can be just as valuable as learning what to do. In fact, you’re more likely to get a nuanced understanding of your visitors behaviors through test that don’t win than you are through those that do.
"Lessons Learned" is the most important part of results sharing because it tells the story that informs the actions your business takes and helps form new questions to answer with follow-up tests.
How to Share
The format to share test results depends on your team and organizational structure. However, the best testing programs use a combination of all of the options below.
- Use a consistent subject line starting with “Test Results” followed by the test’s name.
- Create a distribution list so you don’t forget to include pertinent team members. Distribution lists make it easy for team members to set up archiving rules which allows them to quickly find all test result emails in a consistent location.
- If email is your primary communication method, include screenshots directly in the email of the original page and the best performing variation.
- Include a shareable link to the Optimizely Results page (access this link from the upper right of your Results page).
- Decks (PowerPoint, Keynote or PDF)
- Results decks should be built off of a pre-existing test plan and should include all of the information detailed above under ‘What to Share.’
- Include an overview slide that describes all the variations so they can be compared against each other.
- Include a goals or metrics slide that describes what goals were included (e.g. purchases) and how they were defined (e.g. reached .../thankYouConfirmation page).
- Include variation screenshot slides that show a large image of each variation.
- A “Winner vs. Original” slide is helpful for a clear side by side comparison.
- Include a slide that lists the top five to ten key insights from the test.
- A Next Steps slide should clearly articulate the action that will be taken (if any) to implement the winning variation. For example, will the winning variation be served through Optimizely or will the development team hard code the changes?
- Shareable Spreadsheet
- Shareable spreadsheets are a great place to store a roadmap of test ideas as well as a test results - usually in a separate sheet or tab
- Make sure the spreadsheet is shareable by using something like Google Spreadsheets, Smartsheets, or a table in your team’s wiki page (e.g. Atlassian Confluence etc.)
- Within these spreadsheets, document the basic test results shown above and also link to the full results deck
- Team Meetings
- Team meetings still matter and no amount of emails, results decks or shareable spreadsheets can compensate for the impact of a face to face meeting.
- Schedule in person results review meetings on a weekly, bi-weekly or monthly cycle depending on how often your team is testing.
- Include team members who are involved in executing on the test (designers, developers), director-level decision-makers who are not typically involved in the day-to-day testing efforts, as well as a few peers who aren’t involved in testing but may have worthwhile ideas to contribute.
Create a shareable version of a spreadsheet similar to this one. Notice the three tabs in this spreadsheet for Test Ideas Roadmap, Tests In Progress and Test Results.
Subject: Test Results - [test name]
Overview: The purpose of this test was to….
Hypothesis: The current version of the page [highlight potential problem]. An alternate version that [does ____] will drive an [increase/decrease]in [metric name].
Primary Result: [describe the overall result as simply as possible]
Optimizely Shareable Results Link: ...
Original Variation Screenshot: ...
Winning Variation Screenshot: ...
Next Steps: ...
Additional Test Details: [link to document, shareable spreadsheet or team wiki results page]