- Optimizely X Full Stack
THIS ARTICLE WILL HELP YOU:
- Get started with Optimizely X Full Stack
- Create, test, and rollout features with Optimizely Feature Management
- Run experiments anywhere in your technology stack, using one of Optimizely’s SDKs
- Use Optimizely’s Stats Engine to measure the impact of your experiments
Optimizely X Full Stack helps you run Optimizely experiments in any application or on any connected device. It enables product and engineering teams to build, deploy, test, and roll out features anywhere in their technology stack, with server-side and client-side testing on a single testing platform.
Use Full Stack to develop and experiment in areas like pricing, search result algorithms, or redesigns. A/B test across your technology stack or slowly roll out a new feature.
Full Stack 2.0 introduces a set of new capabilities collectively called Feature Management. Feature Management is designed to make it easy for product development teams to build, deploy, and iterate on features. Feature Management introduces several new concepts that you should become familiar with:
Features are the building blocks of your Full Stack experimentation program. Develop a feature, deploy your feature using the isFeatureEnabled API (the equivalent of a feature flag), then create and launch experiments without deploying again. Toggle your feature on/off for specific users by running feature tests and feature rollouts.
Feature configurations allow you to define variables that parameterize your features. Set the value of the variables in your feature configuration as part of an experiment to iterate on your feature in between code deploys. Use Optimizely’s getFeatureVariable APIs to access your feature configuration variable values in your application.
Feature tests are A/B tests tailor-made for features. Feature tests allow you to test a new feature against your existing user experience, or to iterate on an feature by defining variations based on feature configurations.
Note that feature tests do not require using the
isFeatureEnableddoes all the work for you. When you call
isFeatureEnabled, Optimizely checks for active feature tests, evaluates whether the current user is eligible for those tests, and activates the tests if the user is eligible.
Feature rollouts are the output of your product development process. When you’re ready to expose a new feature to your users, create a feature rollout, set a traffic allocation, and, if appropriate, target an audience. Leverage feature rollouts to deploy the results of your feature tests to the right users, at the right time, with the appropriate safeguards in place.
Finally, we continue to support traditional A/B tests:
A/B tests are one-off experiments designed to answer specific questions. A/B tests use the
activateAPI, which returns the variation key corresponding to the variation assigned to a particular user. Leverage A/B tests to answer one-off questions while using feature tests to iterate on features over time.
Full Stack 2.0 introduces several other new conveniences, including environments, which allow you to map Optimizely to your development process; notification listeners, which allow you easily integrate Full Stack experiments with other services; and bucketing IDs, which allow you to assign users to variations with an identifier other than the user's ID (e.g., an Account ID).
To learn more about the new concepts introduced in Full Stack 2.0, see the following articles:
Use Full Stack SDKs to run tests in different languages and in any application. Selecting a language when creating a Full Stack SDK project allows us to show you appropriate code samples in the UI and inform you of SDK updates. Otherwise, our Full Stack SDKs are language agnostic, meaning that you can use a datafile from any Full Stack project type to initialize an Optimizely client within your app, regardless of which language you've designated for your project.
To learn more or add Full Stack to your account, contact your Customer Success Manager.
Below, we answer some frequently asked questions.
Using server-side approaches with Full Stack is significantly faster than client-side approaches.
In contrast, when using server-side approaches with Full Stack, the backend does the work beforehand. The experiment code runs before the web page ever loads. Full Stack uses in-memory bucketing so it already has all of the information it needs locally and users don't perceive any noticeable latency.
We’ve built our SDKs so you can split traffic to experiments without any network requests. All decisions are made in-memory based on the datafile cached in your application so there is negligible impact on latency.
Although you may be familiar with other platforms that call out to a third-party server to make experiment decisions, Full Stack makes all of these decisions in memory using a cached copy of the datafile. In other words, Full Stack is faster because it doesn't make any blocking API requests to get decisions about which experiment variant to use; the device (your mobile phone or server) running the code makes that decision in less than a millisecond.
The bucketing approach described above means that Full Stack won't slow down the end user's experience. However, there are still a few performance considerations to keep in mind as you scale your usage of Full Stack:
- When and how often to download the datafile that in-memory bucketing uses. Manage this by limiting the files size download frequency.
- When and how often to send data about conversion events for tracking. You can manage this by batching events and using asynchronous event dispatching (having the app send data out-of-band later).
For more information, see:
- How bucketing works in Optimizely's Full Stack SDKs
- Implementation checklist: Prepare Full Stack for production
- Best practices: Datafile management in Full Stack
Please contact us if you’re interested in seeing performance benchmarks for any of our SDKs.
Optimizely must whitelist you for Feature Management because all of your installed SDKs must meet the minimum version requirements. Older SDKs are incompatible with Feature Management. To get whitelisted for Feature Management, file a support ticket.
The minimum SDK versions for Feature Management are:
JS: 1.4.3 and later
Ruby: 1.2.0 and later
Python: 1.1.1 and later
PHP: 1.1.1 and later
Java: 1.6.0 and later
C#: 1.2.0 and later
Node: 1.2.3 and later
Our SDKs allow you to conditionally activate experiments based on user attributes you provide. Define user attributes in the Optimizely app as well as audiences consisting of one or more user attributes. An attribute is comprised of an ID, a key, and a segment ID. All of these are strings. Therefore, any values that are stored as attributes must be converted to strings.
Use our whitelisting feature to force users into specific variations for QA purposes. The whitelist feature allows you to specify a list of user IDs and their corresponding variations. We're working on adding an option to force users into variations from the SDK as well as a preview mode for our iOS and Android SDKs.
SDK projects support mutually exclusive experiments out of the box. Simply create a group and assign experiments to that group.
Our SDKs support bot detection as of version 2.1. Enabling bot filtering requires enabling the feature on your project Settings dashboard. Once the setting is enabled, events sent from web browsers have bot filtering applied automatically. Applying bot filtering to events sent from other environments requires configuration. For more context, read our developer documentation about bot filtering.
Our raw data export feature allows you to export all of the events you’ve sent to Optimizely on a daily basis.
We designed our SDKs to be platform-agnostic, so you can use whatever user ID makes the most sense for your experiments. If you’re experimenting on anonymous users, we recommend using a cookie, device ID, or an automatically generated ID from your analytics provider. If you’re experimenting on a logged-in application or device or experimenting across multiple channels, you can use a hashed email address, universal user identifier (UUID), or any other unique ID.
All of our SDKs use deterministic bucketing via MurmurHash3 to determine what experiments and variations should be active for a user. This ensures that users will be given the same treatment across multiple visits or on different channels. We also ensured that all of our SDKs give the same output no matter what language you’re using.
Yes. The datafile is language-agnostic, and you can use the same datafile from the same project across different SDKs and you would get consistent bucketing across them. This way, you can activate and track the exact same experiments across different SDKs and get consistent results.
The recommended way to manage the datafile across multiple SDKs is to have a centralized service that subscribes to the datafile webhook on your project and pushes updates to other services that subscribe to datafile file changes. Those services in turn would re-instantiate their Optimizely client using whichever Optimizely SDK they have.
Many of our customers are using a CDN to cache content, which means that using one of Optimizely’s SDKs in the backend may require custom configuration. There are several ways you can use Full Stack in your caching architecture:
1. (Recommended) Use an off-the-shelf load balancer provided by your CDN.
a. Implement one of our SDKs at this layer to run bucketing logic.
b. Use the variation key you get from bucketing and the URL as part of the cache key to determine which variation to show.
2. Build your own load balancer on top of your CDN and perform bucketing logic in it to determine which variation to show the user.
3. Use CDN side-scripting such as Edge Side Includes (ESI) for Akamai to run bucketing logic to determine the correct variation to return.
b. Script will run bucketing logic, set a cookie or URL query param, and reload the page.
c. The caching layer should use the query param or cookie along with the URL as the cache key to determine which variation to show.