Skip to main content
menu-icon.png

 

x
Optimizely Knowledge Base

Feature tests: Experiment on features

Relevant products:
  • Optimizely X Full Stack

THIS ARTICLE WILL HELP YOU:
  • Run A/B tests on features
  • Test your features without deploying code

Feature tests are A/B tests you conduct on features. Feature tests have all the properties of an A/B test, plus the ability to:

  • Trigger the test using the isFeatureEnabled API

  • Toggle the feature ON and OFF when defining a variation

  • Specify feature variable values as part of a feature configuration for a specific variation

Create a feature test

Feature Management adds a new type of experiment to the Experiments dashboard: Feature Test. To create a new feature test:

create-feat-test.gif

  1. Navigate to Experiments > Create New…

  2. Select Feature Test from the dropdown menu

  3. Choose the feature you want to test or click Create New Feature... to add a new feature.

    After you select a feature, Optimizely automatically generates an experiment key by appending “_test” to the end of the feature key for the feature you selected.

You can edit the experiment key if you like, as long as you always use a unique key.

Create feature test variations

Optimizely automatically suggests variation keys for your feature tests because variation keys aren't required to set up a feature test. Optimizely will assign a user to a variation, then return the values associated with the assigned variation for isFeatureEnabled and your feature configuration (i.e., variables). By default, we’ll provide two variations with the keys “variation_1” and “variation_2”. Like experiment keys, you can edit variation keys.

If you add variations, Optimizely will provide automatic suggestions according to the variation number: “variation_3”, “variation_4”, and so on. Deleting a variation will not affect the automatic numbering of Optimizely's automatic variation key suggestions. 

Feature toggles and configurations

Feature test variations include a feature toggle and the feature configuration (if one exists). By default, the toggle will be set to ON and the configuration default values will load.

A common feature test includes a feature with no configuration, with one variation set to test “toggle=ON” and another variation set to test “toggle=OFF.” This allows you to experiment on the performance of your application in its current form vs. its performance with your new feature enabled.

If the feature includes a feature configuration and you set a variation to “toggle=OFF,” Optimizely will disable the option to modify variable values and revert to the default variable values.

To create variations using feature configurations, update the variable values under each variation.

create-feat-test-var.gif

When this feature test is live, the getFeatureVariable APIs will return the values specified for the variation assigned to a visitor. Experimenting using a feature configuration allows you to iterate on a feature in between code deploys. Run a sequence of experiments with different combinations of variable values to determine the optimal experience for your users.

If a feature test is running on a feature that uses a feature configuration, the feature configuration is locked until you pause the test.

Launch a feature test

Like A/B tests, you will assign audiences (optional), metrics, traffic allocation, and mutually exclusive groups (optional) to feature tests. After saving your changes, launch your feature test using the same workflow as you would use for A/B tests in environments.

For every feature test running on the feature, we report:

  • Experiment Key: unique identifier for the test. Click to open the Feature Test modal.

  • Type: Feature Test. We plan to include other types of tests in the future.

  • Status: current status of the test (Draft, Running, or Paused)