Bot and spider filtering in Optimizely X
- Last updated
- 11:18, 5 Jun 2017
Save as PDF
- Optimizely X Web Experimentation
- Optimizely X Web Personalization
- Optimizely X Web Recommendations
This article will help you:
- Identify which bots are filtered out by Optimizely X Web
- Learn how bots affect experiment results, monthly unique visitors (MUVs), and billing
Internet bots and spiders are software applications that run repetitive, automated tasks over the Internet. Optimizely X Web uses the IAB/ABC international spiders and bots list to classify non-human traffic and filter it out of your results. The IAB/ABC list is an actively maintained record of user agents used by known spiders and bots. The list is updated on a monthly basis.
Bots and spiders don't behave like real users. They typically run repetitive, automated tasks over your website or app at a much higher rate than real users, with the purpose of crawling your content and analyzing it. As a result, bots and spiders usually inflate the volume of traffic to your site and may skew your conversion metrics.
Removing bot and spider traffic from your results generally decreases the volume of traffic, such as the total number of visitors or sessions, and makes your measurements more accurate.
Optimizely reads the User-Agent header
for each request and determines whether it matches with an entry in the IAB/ABC list. If there is a match, Optimizely classifies the request as non-human and automatically excludes it from the results.
Bot and spider filtering relies on the client’s User-Agent header
to work. For web traffic, the User-Agent header is normally sent with every request, so Optimizely X Web customers will automatically benefit from it.
Optimizely X Mobile and Full Stack customers have to send the User-Agent header with every request to ensure bot and spider filtering is applied to their results.
No. Optimizely X Web automatically applies bot and spider filtering to all traffic as long as the User-Agent header
is present in the request.
Traffic from bots and spiders that are filtered is not included in the MUV allotment for your account.
We will waive any overage charges incurred as a result of bot or spider traffic. Please contact your Customer Success Manager if you believe you have incurred overage charges due to bot or spider traffic.
Contact your Customer Success Manager to learn more about your site’s historical bot and spider traffic levels.