From Novice to Testing Hero: Your Optimizely Journey
In this article, you'll find a general roadmap to help you understand your current level of sophistication with Optimizely and with testing in general, and where you can continue to learn more. We'll focus both on your mastery of the Optimizely platform compared to other users, and the evolution of your company's testing program compared with programs at other companies.
Click a link to jump to that part of the article:
Think of the milestones in this article in two ways: functional and tactical. Your functional skills will help you to find and use Optimizely's features. Your strategic expertise will allow you to use those features for more complex, purposeful experiments that align to your business goals and turn information about your customers into meaningful results.
There are are four broad categories to view the state of your testing program: structure, process, Optimizely platform setup, and Optimizely feature usage.
|Structure||Before you begin testing, these are the elements in place that allow you to be successful.|
|Process||Your testing process allows you to execute on tests and follow-up once test results come through.|
|Optimizely Platform||The platform setup creates the infrastructure for your team members and the other website systems to work together.|
|Optimizely Features||Leveraging the full capabilities of Optimizely’s features will allow you to run tests that will move the metrics that matter to your business.|
As you review these categories, use each of the sub-categories and the corresponding examples to gauge where your testing program is today and the ways it can evolve moving forward.
|Team||The number of humans you devote to testing||One person with testing representing less than 50% of his or her responsibilities||One person mostly or fully dedicated to testing||One person dedicated with design and development resources available||Dedicated team including a testing program manager and dedicated design and development resources|
|Documentation||How much you communicate on what has been, and what will be, tested||No documentation or only some documentation pre-test in a variable format||Pre-test templated (consistent) documentation||Pre-test and post-test documentation in a variable format||Pre-test and post-test documentation in a templated (consistent) format|
|Collaboration||How your team works together||A single person is responsible for generating ideas and generally keeps the learnings to him or herself||Team discussion for ideas takes place but it’s irregular and infrequent. Previous test results may or may not be discussed||Team discussions regularly take place for idea generation and results review||Team discussions regularly take place for idea generation and results review. Additionally a web-based system (JIRA, Google Forms etc.) for idea submission is utilized by a large group of team members even if they’re not typically involved in testing|
|Efficiency||How long your tests take to build||Simple complexity tests take weeks, medium tests takes months, and complex tests take quarters||Simple complexity tests take a week, medium tests take multiple weeks, complex tests take several weeks to over a month||Simple complexity tests take days, medium tests take one week, complex tests take two weeks to a month||Simple complexity tests take hours, medium tests take days, complex tests take one to two weeks|
|Frequency||How many tests you run||1-5 per quarter||5-10 per quarter||10-20 per quarter||>20 per quarter (multiple tests running every week)|
|Results Review||How you share the results of your tests||Little to no results sharing||Only testing “wins” are shared||All test results are shared, even where variations did not provide a l but often only via a shareable database||All test results are shared in a regular and structured live meeting (not just emailed or on an accessible database)|
|Optimizely Platform Setup|
|Permissions||Appropriate access for individual team members||No permissions set up, every team member has full access to start, stop, edit, archive (etc.) experiments (or only one team member has access)||Appropriate permissions are set up for each testing team member||Appropriate permissions are set up for each testing team member||Appropriate permissions are set up for each testing team member and additional “view only” permissions are given to team members not directly involved in testing (but who would sometimes be interested in viewing results)|
|Project Organization||How you organize separate domains and environments||Use of only a single project (one domain) in one environment (live environment)||Multiple projects are set up for each domain but only in one environment (live environment)||Multiple projects are set up for each domain with appropriate projects based on environments (staging, production, live etc.)||Multiple projects are set up for each domain with appropriate projects based on environments (staging, production, live etc.) and a separate project for mobile (or its domain) is utilized|
|Integrations||How Optimizely is augmented by other best-of-breed platforms||Optimizely sits by itself and is not integrated with any other platforms||The analytics platform is integrated with Optimizely to be able to view analytics data broken down by Optimizely experiments and variations||The analytics platform is integrated with Optimizely to be able to view analytics data broken down by Optimizely experiments and variations||Additional platforms for heat mapping and clicks (e.g. CrazyEgg and ClickTale) are integrated|
|Optimizely Feature Usage|
|Before You Test||The Five Steps in Every Test||10 Common Experiments (and how to build them)||10 Common Support Questions (and how to solve them)|