- Form your own data-driven hypothesis
- Create strong hypotheses using "Problem, Solution, Result" framework
- Identify the components of an effective hypothesis
A strong hypothesis is the heart of data-driven optimization. Hypothesis statements help you turn a wealth of data and insights about your visitors’ behavior into focused proposals that you’ll take action on.
Use hypotheses to drive meaningful testing and personalization campaigns. Each hypothesis is an idea to be tested; every idea that’s confirmed or rejected informs you about your visitors’ expectations and behaviors, and feeds your iterative optimization process. Consistent, hypothesis-driven testing helps your program make an impact on company goals.
For example, imagine that you notice a high rate of abandonment in your purchase funnel. You hypothesize that links in the funnel are distracting to visitors, so you test removing them. The improvement in completed purchases is about more than just lift. It confirms your hypothesis and understanding of your visitors and site experience. Use this insight to help decide what to optimize next, why, and how you’ll measure results.
Hypothesis-driven optimization sets your company up for long-term gains. This approach helps you:
- Build a mechanism for constant inquiry
- Understand your user’s pain-points
- Discover and prioritize the potential benefits of your tests
- Establish a common language for ideation and research
- Connect experiments to your company’s challenges
Optimization programs that neglect to generate strong hypotheses for tests and campaigns risk devoting resources to unfocused experimentation that fails to make a business impact.
Read on to learn more about creating strong hypotheses.
Download this hypothesis worksheet to start drafting your hypothesis!
What is a hypothesis?
A hypothesis is a prediction you create prior to running an experiment. The common format is:
If [cause], then [effect], because [rationale].
In the world of experience optimization, strong hypotheses consist of three distinct parts: a definition of the problem, a proposed solution, and a result.
Let’s discuss each component in greater detail.
Experience optimization is most impactful when it solves a problem in the customer experience. When defining a hypothesis, start with a meaningful problem: an issue or pain-point in your visitor’s experience that you'd like to solve.
Use qualitative and quantitative sources to validate your problem. Often, it can be tempting to slide into assumption when thinking about the user experience. It's important to use data to confirm the issue you're trying to solve.
Explain the problem from the visitor's perspective. Doing so will help you understand the issue more deeply and generate hypotheses that get to the root cause.
For the problem validation, try the following format:
Problem definition: Users don't see the filters on our search results page.
Data validation: Less than 15% of users use filters when searching for products, which is very low compared to industry standards.
Next, propose a solution. Describe the change so someone who reads the hypothesis can understand the change, without screenshots. Then, add a rationale that offers a theory about why this solution is the right one to solve the identified problem.
Proposed solution: Move the filters to the left side of the results.
Rationale: This is the most common place for filters to be; users are more likely to notice them there.
Finally, predict a result that ties your hypothesis back to your key business metrics. Include metrics that determine the success or failure of your experiment. Once you know those, decide on the specific metrics you'll use to track success in the experiment or campaign.
Primary metric: % of users who use filters will increase
What does success look like: % of users who move on to a product page will increase; purchase rate will increase.
Examples of strong hypotheses
Let’s look at a couple of examples.
Problem: Users find the featured products on the homepage irrelevant. Only 12% of users click on them. 9 out of 11 users state they never found interesting products on the homepage.
Solution: Set the algorithm for featured products to display products from recent categories the user has visited. If the user visits the category, they have expressed interest in those types of products.
Result: Percentage of users that click on the featured products and Percentage of users that added a product to cart will increase.
What makes this hypothesis extra strong?
- Qualitative and quantitative validations to the problem
- Solution is based on a common UI practice in the industry
Problem: Users do not understand the name on the tab of our financial services (“money services”), as explained in usability interviews. Therefore, 85% of users drop-off after getting to the page.
Solution: Try different names for the sections such as: “financial services”, “professional services”, or “money”.
Result: More clicks on the tab and smaller drop-off rate on the section landing page.
What makes this hypothesis extra strong?
- Clear problem identification, including symptom
- The results encapsulate quantity (more clicks to section) and quality (less drop off at section)
Here are some best practices to follow when creating hypotheses.
Engage in discovery
Consider building a business intelligence report. One key difference between a well-formulated hypothesis and a guess is data.
Form a hypothesis with what you already know. Dive into existing data sources and carefully observe your customer’s journey through your site:
- Learn about generating ideas with direct data sources, such as visitor paths, voice of the customer feedback, analytics data, user testing, and more.
- Learn about generating ideas with indirect data sources, such as a competitive overviews, industry data, academic research, and more.
Use data that’s strongly linked to your company’s goals to ensure you’re focusing on areas of significant impact rather than making UX changes in isolation. For example, how does your company define success on the web? Use data sources to validate the problem and the solution rationale. Avoid hypotheses generating that are based simply on intuition.
Create a testable hypothesis
To efficiently test your hypothesis, identify the metrics to be tracked and define a clear criteria for success/failure.
For example, you hypothesize that removing breadcrumb navigation from the checkout page will help visitors stay in the funnel and increase conversions. The difference between the original and the variation is the presence or absence of breadcrumbs. The effect of that change can be measured in the number of conversions.
Use insights as a learning opportunity
Use hypothesis-driven testing to gain insight into your customer’s behaviors. These insights will generate additional questions about your visitors and site experience and drive an iterative learning process.
Ideally, the cycle follows this general pattern:
- Gather data about your visitor’s behaviors and industry and use insights from that data to ask questions
- Formulate a hypothesis based on insights from your data
- Design and implement an Experiment or Campaign based on your hypothesis
- Analyze your results to decide whether your hypothesis is confirmed or rejected
- Create and document conclusions
- Use conclusions to create new questions
Connect to your company’s problems
Begin your process with a problem, not a solution. If a solution fails to deliver the result you expect, use the problem statement to explore new potential solutions and iterate. This practice enables you to focus on the problems that prevent your company from reaching its goals.