Skip to main content

We are currently experiencing an issue that is preventing customers from submitting support tickets. Please contact us at (866) 819-4486 if you need immediate assistance.

x
Optimizely Knowledge Base

Guide your program with an experimentation charter

This article will help you:
  • Understand the purpose of a testing charter
  • Decide what is important to include in your charter
  • Build your own testing plan using our starter template

Imagine the following scenario: You recently purchased a new A/B testing platform for your company. You have high hopes for showing a great return on your investment after having spent weeks (or maybe months) deciding on the platform, negotiating the contract’s duration and cost, and hiring a new employee (or transitioning a whole team) to use it. The platform’s code has been installed and it’s now possible to begin testing.

The question is, how ready are you? From the initial consideration phase through to purchase and installation, it’s the uncommon company that has fully considered the necessary details of how they’re going to make the most effective use of the platform. But if you immediately jump into testing, in the weeks and months that follow you may find yourself facing:

  • Inefficient test planning and setup
  • Tests launched with errors or mistakes
  • Tests with missing goals or audiences
  • Low testing velocity
  • Poor communication between team members
  • Tests that generally have no impact
  • Missed opportunities to test
  • An inability to effectively learn from test results

Each of these scenarios can be avoided or attenuated with a little planning before you get started. Broadly speaking, documenting a game plan will allow for efficient, consistent, and meaningful testing. A charter helps you track the mission, resources, and workflow that govern your A/B testing strategy.

What you need to get started:
Materials to prepare

Address the following questions before creating your testing charter.

    • Business requirements: what are the core responsibilities of the optimization team?
    • Budget: what resources do you have?
    • Performance benchmark data: what time investments are required to meet performance goals?
    • Executive sponsorship: do you have approval for resources, including time investments?
    • Team proficiency: what experience and skill sets do available team members have?

People and resources
    • Executive sponsor
    • Program manager
    • Team leads (technical, design, analytics)
    • External consultant, if needed

Actions you'll perform 


Deliverables
    • Testing charter that documents roles and responsibilities, including:
        • Role definitions
        • % time spent per role
        • Program and individual metrics
    • Optimization workflow
        • Inputs and outputs for each step of the workflow
        • Responsibilities and contributions, by step
    • Evaluation of team's skill sets against program initiatives and goals
    • Establish Optimizely user roles
 

What to watch out for

    • It can be difficult to establish roles and responsibilities before creating a well-articulated workflow
    • It can be challenging to establish staffing requirements and time investments before you have a prioritized roadmap of tests and campaigns

What your testing charter should include

Your testing charter (which you might document in a company wiki, a Word document, a Google Doc, etc.) may include a brief mission statement, but should definitely include many practical, referenceable details. When you create it, imagine a new employee, senior company leader, or anyone at your company who doesn't know about the testing program. For him or her, the testing charter is a great place to start learning about the program. Your document doesn’t need to include everything from the list below, but the more thorough you are based on the needs of your team, the better off you'll be.

The testing charter should be a working document. It will never be finished. The testing Program Manager (or equivalent) should own the charter. One or two months into the testing program, the Program Manager should revisit the document to see how the actual process compares to this plan. Depending on what’s working and what’s not, this is a good time to modify the plan. From there, the Program Manager can ensure that the program is meeting its goals by reviewing and updating the charter on a quarterly or semi-annual basis.

To get started, download our testing charter template. You can copy and modify this template in the program of your choice. Below you’ll find more detailed explanations for each section of the document.

  • Introduction: Set the context with a paragraph of introduction. Why did you purchase the Optimizely testing platform? What problems were you trying to solve?  What are the long-term goals and expected outcomes? Define the purpose for the rest of the document.
  • Process: What is the workflow from conception, execution, results-sharing, through to (potentially) onsite implementation? Create a workflow diagram in Keynote or PowerPoint and insert the image here.

    For each of the steps you outline, add a brief description and a list of key participants in each step.
  • Meetings & Meeting Cadence: When do you have testing brainstorm meetings? When do you have status meetings to check on upcoming tests and review currently running and recently completed ones? How often does the testing team meet with a broader group?
  • Test Velocity: What is the goal for testing velocity: One test always running, ten tests per week, twenty five per quarter? Whatever it is, set an initial goal and modify this as needed to ensure that it’s reasonable based on the actual pace of testing at your company.
  • Test Plan & Results Templates: Testing programs that are efficient, run meaningful tests, and act on the lessons of each test create testing documentation. Create a link to the most-updated testing template the company uses to document test plans and results.
  • Links Key: Your links key is where you link to all of other essential information including your current testing roadmap, templates (described above), and an archive of test results. If team members bookmark this page, they should easily be able to find any other relevant information they need. You may also want to link to pages that document definitions of the following:
     
    • Test Goals, for example, “Purchase Confirmation - a pageview goal targeted to http://www.example.com/thank_you_page
       
    • Targeting Conditions, for example, “Product Pages - targeted with a substring match to example.com/productID”
       
    • Audiences and Dimensions, for example, “Category Affinity defines users who are interested in a particular category and bases it off of a recent product or category pageview associated with that category”
       
  • Other Guidelines & FAQs: What guidelines and rules will your team follow when it comes to testing? Do different types of tests have different processes for QA and approval? Are there standard goals included in every test? When will a winning variation be served through the Optimizely platform and when will it be hardcoded to the site?