- Collect accurate data in your experiments
- Avoid common mistakes in e-Commerce experiment design
In e-Commerce, it's important to pay attention to business cycles and visitor behaviors. Tests that don't consider these patterns risk losing important pieces of data from the Results page. Good experiment design can ensure that you capture an accurate reading of conversion rates and other visitor behaviors.
Don't make data-driven decisions without the full picture. Read on for tips on designing robust experiments that capture a full cycle of data for e-Commerce.
Tip 1: Test through a full business cycle
Traffic and usage of a website often follows external patterns: holiday schedules, weather, and more. Often, the clearest traffic pattern for an e-commerce business is based on the days of the week.
Let's take the traffic to Optimizely's Knowledge Base as an example:
Notice the regular cycles of traffic. The Knowledge Base is typically used by professionals while at work, so site traffic consistently dips lower on weekends.
In eCommerce, a weekly usage pattern is the most common. Visitors often have more time to browse products and make purchase decisions on the weekend. Or, depending on the type of product, visitors might more typically browse on the weekend and purchase during a work break on a weekday. The weekly usage pattern can vary, depending on the business.
Plan your experiment cycles according to your website's usage patterns. It's a good idea to run experiments for at least a full week to preserve the ratio of data you collect on weekends and weekdays (through the whole week).
Imagine you start an experiment on Friday and see statistical significance by Monday. You may be tempted to stop the test. Here's why you shouldn't: your experiment spans two (2) weekdays and two (2) weekend days; but it leaves out three (3) weekdays. Weekend behaviors are disproportionately amplified in your data because your collection period doesn't cover the full cycle. Let the test run for three more days.
A weekly cycle is common for e-commerce businesses, but your business might have bi-weekly, monthly, or other time-based cycles. Running a test for a full business cycle gives you a more accurate picture of the impact of your changes.
Tip 2: Consider time-to-purchase before stopping an experiment
Time-to-purchase measures the average time that lapses between a visitor's first site visit and a purchase event. In this period, visitors take time to get oriented, compare products and prices, and make up their minds. Before stopping an experiment, consider your average time-to-purchase.
Set your traffic allocation to 0% so no new visitors can enter the experiment, but keep running it for the length of your average time-to-purchase.
Here's why. Imagine that you're tracking purchases in an experiment. Once you get statistical significance, you stop your test immediately. Visitors who enter the experiment near the end don't have time to complete their purchase before you stop tracking. You lose data for visitors who end up buying, but whose purchases aren't captured on your Results page.
Closing the experiment to new visitors but capturing final conversions will help you get more accurate data on your conversion rate and the impact of your changes.
Time-to-purchase varies for different products and audiences, so you need data that’s specific to your website and products. Many analytics platforms can track average time-to-purchase. Here’s a guide to tracking time-to-purchase in Google Analytics.
In general, incorporate information about your site's business cycles and visitor patterns into your experiment design. This will help you design robust experiments and build a rich, accurate, data-driven picture of your visitors' responses to the changes you test on your site.