This article will help you:
  • Build a business intelligence report to drive your personalization and A/B testing strategy
  • Use direct data to generate strong hypotheses
  • Consider how to use multiple data sources together

Many organizations rely on direct data, or direct intelligence about their site experience, to drive the ideation process. And no wonder -- today’s optimizers have dozens of analytics platforms at their fingertips. But it can be a challenge to use all that data well.

Elite optimizers use direct data to tell a story about their visitors: who they are, what they’re trying to do, and how they’re trying to it. With that knowledge, they form data-driven hypotheses about the actions a visitor is likely to take -- and optimize for them.

Here’s a list of the most common direct data sources:

Don’t feel that you need to consult all these sources equally. Do use this list to consider which will generate specific, actionable insights to guide your testing and personalization efforts. With a few different data sources, you should be able see a bigger picture to better understand how visitors navigate your site experience.

Use the insights you generate from your direct data to build your business intelligence report and develop strong hypotheses for testing and personalization.

 
Note:

This article provides an overview of how you’d use each of these direct data sources for ideation. For more detailed information on how to use web analytics, check out this article or this webinar from Optimizely’s Summer School.

Web analytics

Many organizations rely heavily on behavioral web analytics for ideation. Behavioral web analytics can help you understand where to optimize, but not necessarily how. Pair this data with other direct and indirect data sources to decide how to optimize for different opportunities on your site.

Use case

Focus your analysis on figuring out where to test. Use pathing and funneling analyses to look for disproportionate drop-off rates. Or, monitor traffic patterns and “page values” -- an estimate of how much revenue a given page generates -- to find the most valuable content on your site.

If you find that 20% of your site generates 80% of the value, focus your data collection and optimization efforts on that valuable real estate.

How this adds insight

Once you’ve identified bottlenecks in your most valuable flows, layer on qualitative data such as surveys and heatmaps for additional insight. Should you surface your most valuable products where visitor attention is concentrated? Is an unclear step in your sign-up flow resulting in unusually high drop-off?

Use indirect data such as reviews of competitor sites to note different design decisions that you might test.

Experiment results

Often, the most actionable source of quantitative data is the results of your past experiments and campaigns. The difference between the success of two experiences can tell you a lot about what your visitors want.

Use case

Say you move a button to a different place on the page and this results in a higher click-through rate. You have clear evidence that button placement matters to your visitors. Are there similar types of changes that might matter? Would changing the hero image or providing different information above the fold increase conversions?

How this adds insight

Web analytics provide context for the results of a past experiment. Are some product categories more or less affected by the experience you created? Did certain media campaigns underperform?

Heatmaps and survey results can also shed light. For example, a more prominent button encourages more click-throughs but also results in more visitors exiting the site. Qualitative data can help you understand why -- and how to encourage visitors to stay and browse.

Voice of the customer

Surveys, NPS scores, and the experiences of your support team can reveal facets of your customer experience that you might not find in your other data.

 
Tip:

Due to low response rates and small sample sizes, it can be tricky to draw definitive conclusions based on survey feedback and NPS scores. Try tagging your survey responses with a topic category or keyword to find clearer trends.

Use case

At Optimizely, we use NPS surveys to gather feedback from customers on how we can improve. Over the years, these surveys have provided much of the feedback that Optimizely uses to bring products to market, such as Stats Engine.

Take a look at your NPS scores. Are there any actionable trends in that customer feedback?

How this adds insight

Pair an NPS scoring system with different experiment variations to gather direct feedback from customers about each experience. These responses can help you understand why a page is underperforming and where there are missed opportunities to optimize your site experience.

Heatmaps

Heatmaps tell you where your visitor’s attention is focused and what the most important part of your page layout is.

Use case

Landing pages are some of the most important places to optimize, particularly for B2B, lead generation, and consumer service businesses. Optimizely’s technology partner ClickTale has published a helpful guide for how to optimize landing pages based on heatmaps.

Once you understand which parts of your landing get the most attention, redesign the page to highlight the most important CTAs, headlines, or videos.

How this adds insight

If your site has multiple landing pages, it can be difficult to reduce the bounce rate across all of them. Heatmaps show you trends -- for example, if a certain element distracts visitors from your ultimate goal. You can also run a heatmap analysis on your competitor’s landing page or an admired third-party experience to compare.

 
Tip:

Certain tools like EyeQuant don’t require a behavioral tracking code, so you can run them on any site you’re interested in.

User personas

User personas distill the complex and varied characteristics of your customers into a few themes that help you make savvy marketing decisions. Interviews, surveys, analysis, and brainstorms help you create personas that inform your testing and personalization efforts.

Use case

User personas are a helpful cue for framing campaigns and experiments. Use them to decide how to target headline copy to a C-suite executive, a budget shopper, or a social sharer, for example. You don’t necessarily have to develop full, formal profiles. Informal personas such as “customers who haven’t purchased in a long time” or “visitors who are actively comparing our services to our competitors’ services” can be just as actionable.

How this adds insight

When combined with web analytics, user personas help you understand customers’ motivations. For example, if certain shopping behaviors such a cart that includes kitchenware and holiday decorations indicate a customer buying for a family. What targeted message or a promotional incentive can you offer that persona?

Personas can also shed light on indirect data like competitor reviews. Does your competitor’s site and messaging target a specific persona? How does this affect how you position your own brand or design messaging for your site?

Usability tests

Usability testing shows you common pitfalls in your site experience. Conduct manual tests with live sessions or use a platform like UserTesting.com to gather data asynchronously.

Use case

Usability testing helps you answer important questions such as:

  • Can users who are unfamiliar with your site successfully complete important tasks?
  • Are certain moments of complexity or uncertainty in your site experience prompting visitors to exit a funnel?

Review mockups, competitors’ sites, and new features before you take them to market.

How this adds insight

Use your survey results, competitor reviews, and web analytics to develop prompts for usability testing. Imagine that your web analytics indicate poor performance on a landing page and a heatmap shows that visitors aren’t engaging with the CTA. Based on this information, design a user testing prompt that asks: “Tell us the #1 thing you feel this page communicates. How long did it take you understand that message? Is it compelling? What else would you like to see? What’s the next step you would take on this page?”

Customer decision models

Some consumer decisions -- such as buying a home, enrolling in a new health insurance plan, or researching a SaaS product -- require multiple touch points before a final decision can be reached. Often, these touch points occur offline as well as online. Smaller online touchpoints are sometimes referred to as micro-conversions.

Build a model to describe the triggers and interactions a customer is involved in during the decision-making process. This can help you optimize for the micro-conversions that lead a customer to complete your key transaction.

Use case

Create hypotheses that focus on the most important micro-conversions in your customer decision model. What triggers indicate success in this model (scroll depth, video views, or broad content consumption)? What behaviors should you optimize for? Monitor how optimizing these micro-conversions affects your leading indicators.

How this adds insight

Look for patterns among visitors who display the behaviors that you consider micro-conversions. Combine your customer decision model with your CRM, web analytics, and personalization strategy to target CTAs and value propositions based on where a customer is in the decision-making process.

Do new visitors want to see proof that your service is reputable, before they engage with videos or demos? Conditionally show those visitors social proof and data points to encourage them to engage.

Once a visitor has seen your basic content, consider moving them deeper into the model; the visitor may now be ready to provide an email address in exchange for additional content like technical specifications.

Metrics

Often, there’s an implicit tradeoff between one metric versus another. Monitor the interaction effects between different metrics to understand how an improvement to one might detract from another. You might also directly test these theories to optimize for aggregate profit.

Use case

Imagine an online retailer finds that offering a promotional discount improves her conversion rates. However, this improvement is at the expense of cart size and average order value (AOV). She might decide to experiment with those discount levels to model a price-elasticity curve. This model can help her maximize total revenue based on tradeoffs between unit margins, purchase conversion rates, and AOV.

Or, in media publishing, prominent social share buttons can reduce the likelihood that a visitor will click through to another article. Likewise, optimizing for ad impressions above the fold can detract from scroll depth and article completion rates. Monitor how competing conversion metrics affect important downstream goals like acquisition and retention.

How this adds insight

Best-in-class site experiences, academic publications, and your competitors’ sites can provide a sense of how far you can push the envelope when optimizing for different metrics. These sources help you understand how other players in your space negotiate the tradeoffs. For example, research shows that business travelers are less likely to sacrifice details in their travel itineraries for a lower price than leisure travelers. For travel providers, this industry-level insight provides context for how to optimize for different metrics.

Web analytics and heatmaps also help you design experiments that test different metrics. For example, the exit rate at different stages of your funnel can indicate when to introduce a promotional banner into your visitor’s experience.

There are many direct data sources you can draw from for ideation. Turn data into action by building key insights from these sources into your intelligence report. Over time, your direct data will help you develop a deep understanding of your customers so you can create strong hypotheses and a robust optimization roadmap.