Best practices for sharing testing results

This article looks at best practices for communicating results effectively to help evangelize data-driven decision-making within your organization. We also include flexible email and presentation templates (Keynote, PowerPoint) for sharing results.

Split testing is about gathering data, drawing insights from that data, and taking action. If test results aren’t widely shared, your company will have fewer insights and taking action on those insights will be less likely to happen. Conversely, the companies that consistently share their test results end up testing more, testing more effectively and move the metrics that matter to their businesses.

Despite this, effectively sharing test results is one of the most overlooked parts of testing. Follow the guidelines below to make sure your company doesn’t overlook this crucial step.

What to Share

When you share your results include the following:

  • Purpose - a brief description of the thought process behind the test

  • Details - include the number of variations, the differences between them, the dates the test ran, the total visitor count and visitor count by variation

  • Results - the numbers including percentage lift (best performing variation against your original) and conversion rates by variation

  • Lessons Learned - analysis of what the numbers mean, key insights based on the data

  • Revenue Impact - when possible, quantify the value of a given percentage lift for year over year projected revenue impact

The bullets above do not apply only to test wins. Tests that don’t produce a variation which performs better than original can still generate valuable lessons - learning what not to do can be just as valuable as learning what to do. In fact, you’re more likely to get a nuanced understanding of your visitors behaviors through test that don’t win than you are through those that do.

Lessons learned is the most important part of results sharing because they tell the story that informs the actions your business takes and helps form new questions to answer with follow-up tests.

How to Share

The format to share test results depends on your team and organizational structure. However, the best testing programs use a combination of all of the options below.

  • Emails

    • Use a consistent subject line starting with “Test Results” followed by the test’s name.

    • Create a distribution list so you don’t forget to include pertinent team members. Distribution lists make it easy for team members to set up archiving rules which allows them to quickly find all test result emails in a consistent location.

    • If email is your primary communication method, include screenshots directly in the email of the original page and the best performing variation.

    • Include a shareable link to the Optimizely results page (access this link from the upper right of your results page).

  • Decks (PowerPoint, Keynote or PDF)

    • Results decks should be built off of a pre-existing test plan and should include all of the information detailed above under ‘What to Share.’

    • Include an overview slide that describes all the variations so they can be compared against each other.

    • Include a goals or metrics slide that describes what goals were included (e.g. purchases) and how they were defined (e.g. reached .../thankYouConfirmation page).

    • Include variation screenshot slides that show a large image of each variation.

    • A “Winner vs. Original” slide is helpful for a clear side by side comparison.

    • Include a slide that lists the top five to ten key insights from the test.

    • A Next Steps slide should clearly articulate the action that will be taken (if any) to implement the winning variation. For example, will the winning variation be served through Optimizely or will the development team hard code the changes?

  • Shareable Spreadsheet

    • Shareable spreadsheets are a great place to store a roadmap of test ideas as well as a test results - usually in a separate sheet or tab

    • Make sure the spreadsheet is shareable by using something like Google Spreadsheets, Smartsheets, or a table in your team’s wiki page (e.g. Atlassian’s Confluence etc.)

    • Within these spreadsheets, document the basic test results shown above and also link to the full results deck

  • Team Meetings

    • Team meetings still matter and no amount of emails, results decks or shareable spreadsheets can compensate for the impact of a face to face meeting.

    • Schedule in person results review meetings on a weekly, bi-weekly or monthly cycle depending on how often your team is testing.

    • Include team members who are involved in executing on the test (designers, developers), director-level decision-makers who are not typically involved in the day-to-day testing efforts, as well as a few peers who aren’t involved in testing but may have worthwhile ideas to contribute.

Templates

These templates are a great starting place but you may want to modify them to fit the needs of your team.

  • Results Deck Template
    Removing a few slides from this template creates a good test plan before you run the test. Then when the test is complete, add in the pertinent results slides.
    Download Keynote (Mac)
    Download PowerPoint (Windows)

  • Spreadsheet Template
    Create a shareable version of a spreadsheet similar to this one. Notice the three tabs in this spreadsheet for Test Ideas Roadmap, Tests In Progress and Test Results.
    Download (Excel)

  • Email Template
    Subject: Test Results -  [test name]

    Content:
    Hi Test team,

    Overview: The purpose of this test was to
    ….
    Hypothesis: The current version of the page
    [highlight potential problem]. An alternate version that [does ____] will drive an [increase/decrease]in [metric name].
    Primary Result:
    [describe the overall result as simply as possible]
    Optimizely Shareable Results Link:
    ...
    Original Variation Screenshot:
    ...
    Winning Variation Screenshot:
    ...
    Next Steps:
    ...

    Additional Test Details:
    [link to document, shareable spreadsheet or team wiki results page]

    Thanks!
    [your name]

Comments

Article is closed for comments.