From Novice to Testing Hero: Your Optimizely Journey

In this article, you'll find a general roadmap to help you understand your current level of sophistication with Optimizely and with testing in general, and where you can continue to learn more. We'll focus both on your mastery of the Optimizely platform compared to other users, and the evolution of your company's testing program compared with programs at other companies. 

Click a link to jump to that part of the article:

Think of the milestones in this article in two ways: functional and tactical. Your functional skills will help you to find and use Optimizely's features. Your strategic expertise will allow you to use those features for more complex, purposeful experiments that align to your business goals and turn information about your customers into meaningful results.
 
There are are four broad categories to view the state of your testing program: structure, process, Optimizely platform setup, and Optimizely feature usage.
 
Structure Before you begin testing, these are the elements in place that allow you to be successful.
Process Your testing process allows you to execute on tests and follow-up once test results come through.
Optimizely Platform The platform setup creates the infrastructure for your team members and the other website systems to work together.
Optimizely Features Leveraging the full capabilities of Optimizely’s features will allow you to run tests that will move the metrics that matter to your business.

As you review these categories, use each of the sub-categories and the corresponding examples to gauge where your testing program is today and the ways it can evolve moving forward.


Structure
  Definition Novice Intermediate Advanced Testing Hero
Team The number of humans you devote to testing One person with testing representing less than 50% of his or her responsibilities One person mostly or fully dedicated to testing One person dedicated with design and development resources available Dedicated team including a testing program manager and dedicated design and development resources
Documentation How much you communicate on what has been, and what will be, tested No documentation or only some documentation pre-test in a variable format Pre-test templated (consistent) documentation Pre-test and post-test documentation in a variable format Pre-test and post-test documentation in a templated (consistent) format
Collaboration How your team works together A single person is responsible for generating ideas and generally keeps the learnings to him or herself Team discussion for ideas takes place but it’s irregular and infrequent. Previous test results may or may not be discussed Team discussions regularly take place for idea generation and results review Team discussions regularly take place for idea generation and results review. Additionally a web-based system (JIRA, Google Forms etc.) for idea submission is utilized by a large group of team members even if they’re not typically involved in testing
Process
  Definition Novice Intermediate Advanced Testing Hero
Efficiency How long your tests take to build Simple complexity tests take weeks, medium tests takes months, and complex tests take quarters Simple complexity tests take a week, medium tests take multiple weeks, complex tests take several weeks to over a month Simple complexity tests take days, medium tests take one week, complex tests take two weeks to a month Simple complexity tests take hours, medium tests take days, complex tests take one to two weeks
Frequency How many tests you run 1-5 per quarter 5-10 per quarter 10-20 per quarter >20 per quarter (multiple tests running every week)
Results Review How you share the results of your tests Little to no results sharing Only testing “wins” are shared All test results are shared, even where variations did not provide a l but often only via a shareable database All test results are shared in a regular and structured live meeting (not just emailed or on an accessible database)
Optimizely Platform Setup
  Definition Novice Intermediate Advanced Testing Hero
Permissions Appropriate access for individual team members No permissions set up, every team member has full access to start, stop, edit, archive (etc.) experiments (or only one team member has access) Appropriate permissions are set up for each testing team member Appropriate permissions are set up for each testing team member Appropriate permissions are set up for each testing team member and additional “view only” permissions are given to team members not directly involved in testing (but who would sometimes be interested in viewing results)
Project Organization How you organize separate domains and environments Use of only a single project (one domain) in one environment (live environment) Multiple projects are set up for each domain but only in one environment (live environment) Multiple projects are set up for each domain with appropriate projects based on environments (staging, production, live etc.) Multiple projects are set up for each domain with appropriate projects based on environments (staging, production, live etc.) and a separate project for mobile (or its domain) is utilized
Integrations How Optimizely is augmented by other best-of-breed platforms Optimizely sits by itself and is not integrated with any other platforms The analytics platform is integrated with Optimizely to be able to view analytics data broken down by Optimizely experiments and variations The analytics platform is integrated with Optimizely to be able to view analytics data broken down by Optimizely experiments and variations Additional platforms for heat mapping and clicks (e.g. CrazyEgg and ClickTale) are integrated
Optimizely Feature Usage
  Definition Novice Intermediate Advanced Testing Hero
Test Types What you're testing Single page, A/B tests Multi-page tests, multivariate tests (MVTs) and tests on global elements via URL targeting conditions Targeted tests based on incoming audiences (SEM Campaigns, Geography etc.) and mobile website tests Targeting tests based on custom cookies or JavaScript conditions
Editing and QA How you create variations for the perfect test Editing mode (only), mostly text changes and image swap tests Interactive Mode, use of the preview console, cross-browser preview and QA, rearranging page elements for alternate layouts, and running redirect experiments Edit code window for manipulation of the raw code Optimizely executes on the page Creating custom code in the edit code window or Global CSS/JavaScript to create complex functionality and experiments
Goals and Results How you define and measure success Engagement (the default goal) and click tracking Page-view goals, goal targeting, and review of default segments Adding and reviewing additional segments (URL Source, Preferred Language, IP Address etc.), goals based on custom cookies, and revenue goals Custom event goals (such as scroll depth), or JavaScript conditions

 

Comments

Article is closed for comments.