The most active crawlers and bots on the web (2024)

Allowing web crawlers to scan your site is vital if you want your web pages to appear in Google, Bing and other search results. Here, we list the most common crawlers alongside their User Agents.

Updated: March 2023

Allowing web crawlers to scan your site is vital if you want your web pages to appear in Google, Bing and other search results. However, unwanted traffic spikes caused by non-human visitors can be costly in terms of bandwidth, CPU time, website stability, potentially leading to site outages.

We've just updated our list of most active web crawlers, bots and spiders visiting websites, i.e. the most common instances of non-human traffic that we see in our data. We have also included their user-agents for reference. It's important to note that this list only includes bots which identify themselves; to learn about both self-declared and undeclared bots visiting websites, check out these articles Introduction to Bot Traffic — Part One of our Bot Analytics Series and Dark Traffic and Misrepresentation - Analyzing the Web Analysers (Part 2).

Highlights from the article

• User Agent Client-Hints supported only by few crawlers
• Android dominates mobile crawling traffic
• The HTTP libraries between the most active crawlers
• Mid tier "devices" used for the crawling (mostly from 2016)
• Almost no crawlers using tablet User-Agent

What are crawlers used for?

Web crawlers, also known as web spiders or bots, are automated programs used to browse the web and collect information about websites. They are most commonly used to index websites for search engines, but are also used for other tasks such as monitoring online content, validating HTML code, testing web performance and feeding language models.

Web crawler engine

The most common crawlers hitting any site are in-house scraping engines like Google, Bing or DuckDuckGo. Those engines include the ability to scale, sophisticated logic to crawl the site without causing any impact and to store and process massive data sets.

There are also many open source engines available with interesting features such as ability to simulate human behavior, rate control, distributed architecture or parsing of various document formats.

• In-house
  Google, Bing, Yahoo, DuckDuckGo and others…
• Open source
  Scarpy, Pyspider, Crawlee, Heritrix, Web-Harvest, Apify, MechanicalSoup, Apache Nutch, Node Crawler and many, many more…
• HTTP library
  OkHttp, Java, PHP, Python, NodeJS…
• Command line tools
  Wget, cURL (also integrated as a library by other languages)

Crawlers list

Names of the most active crawlers, bots and other non-human traffic on the web as seen by our device detection Cloud Service. It is not to be interpreted as traffic directly due to the caching mechanism used by the Cloud Service clients which might favor services using various User-Agent versions. It’s a combination of normalized traffic and “popularity” of the crawlers within our user base.

User-Agents of most active crawlers

OkHttp library

Not a crawler as such but the most spread HTTP library generating non-human traffic. Each request might have a different purpose as anybody can incorporate this library by their own means. The most popular variant seems to be version 4.9.2 and version 3.12.10 where the latter one is around two years old.

Google

There is no surprise that most crawling requests are coming from Google bots. That includes Googlebot, Google Ads bot, Google-Read-Aloud bot and others. Some of them even include two variants - desktop and mobile.

Beware that due to its popularity there might be other services pretending to be the Googlebot or there might be individuals trying to get past the paywalls.

Headless Chromium

Headless Chromium allows running Chromium in a headless/server environment. Expected use cases include loading web pages, extracting metadata (e.g., the DOM) and generating bitmaps from the page contents.

e.g. Used for the PageSpeed Insights service

Facebook

The Facebook crawler which prefetches a page to generate a preview of the page which usually consist of title, short description and thumbnail image.

Python

Nessus

cURL

Bing

Others to note

Crawlers by hardware type

There are many ways we can categorize non-human traffic. It can be by their purpose, it can be by their engine, as already mentioned above or it can be by how they advertise themselves to the websites.

Crawlers can use non-specific keywords in the User-Agent - having just their own name as the pivotal point - or they can pretend to come from a specific hardware like desktop, mobile phone or tablet.

Client-Hints support

Even though it is primarily Google who wants to deprecate and freeze User-Agent their Googlebots fleet was left behind. The mobile bots still rely on the classic User-Agent parsing without any “sec-ch” headers. We tried to reach out and asked about crawler’s Client-Hints support but there was no reply.

So far it’s only such non-human traffic which functions as a proxy, like Amazon CloudFront, or using a real Chrome engine, like Chrome-Lighthouse, HeadlessChrome or SpeedCurve, which support Client-Hints. There is zero Client Hints support from all the rest.

Amazon CloudFront HTTP headers sample

sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"sec-ch-ua-mobile: ?1sec-ch-ua-platform: "Android"user-agent: Amazon CloudFront

Chrome-LightHouse HTTP headers sample

sec-ch-ua: "Chromium";v="98", "Google Chrome";v="98", "Lighthouse";v="9.6.6"sec-ch-ua-mobile: ?1sec-ch-ua-platform: "Android"user-agent: Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse

SpeedCurve (PTST) HTTP headers sample

sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="109", "Google Chrome";v="109"sec-ch-ua-mobile: ?1sec-ch-ua-platform: "Android"user-agent: Mozilla/5.0 (Linux; Android 8.1.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36 PTST/230125.191541

Headless Chrome HTTP headers sample

sec-ch-ua:sec-ch-ua-mobile: ?0sec-ch-ua-platform:user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/104.0.5112.101 Safari/537.36

Mobile crawlers

Here is a special breakdown of a traffic where crawlers are pretending to be a specific mobile device or a tablet. Overall it’s the Android models that dominate crawling part of the traffic, regardless of Apple's policy of the model obfuscation in the User-Agent.

Reference mobile phones

There seems to be no real demand to use the most up-to-date mobile devices. All the crawler User-Agents have been picked and set many years ago and it’s working as expected. Pretty well representing the mid tier mobile device category.

Let’s look at most popular devices to see what mobile devices have been picked by crawling giants and for what functionality, image size and screen dimensions optimize non-human traffic.

Google (LG) Nexus 5X

• Proportion of reference traffic: 30 % - 33 %
• Reported Android version: 6.0.1
• Year released: 2015
• Viewport width: 412 (1080 physical)
• Diagonal screen size: 5.2”
• Processor: Hexa-core (1.8 GHz, 1.4 GHz)
• Used by Google

Google Nexus 5X can be called a reference Android mobile phone. Decent physical screen resolution supporting Full HD, six cores and 5.2” diagonal screen size. Along with the Apple iPhone it is accounting for the majority of the crawlers traffic pretending to be a mobile device.

Heavily used by various Google crawling services or by other crawlers pretending to be Google. The Android version seems to be fixed at 6.0.1 while Chrome browser versions are getting updated, at least for the official Google bot User-Agents.

Apple iPhone

• Proportion of reference traffic: 24 % - 26 %
• Reported iOS version: 6.0 - current
• Year released: 2009 (iPhone 3GS) - current
• Viewport width: Anything from 320 to 430 (640 to 1290 physical)
• Diagonal screen size: Anything from 3.5” to 6.69”
• Processor: Anything from Single-core (600 MHz) to Hexa-core (3.46 GHz)
• Used by Google, Semrush, Bing, Yisou, Baidu and many others

Despite the iPhone User-Agent being used the most by the mobile crawlers it’s actually the most difficult one for the content optimisation. Lack of a specific model makes the optimization very hard due to the performance and screen resolution quite big differences across the whole iPhone product line.

Verizon (Samsung) Galaxy S7

• Proportion of reference traffic: 12 % - 13 %
• Reported Android version: 7.0
• Year released: 2016
• Viewport width: 360 (1440 physical)
• Diagonal screen size: 5.1”
• Processor: Quad-core (2.16 GHz, 2.15 GHz)
• Used by Google

The Google Read Aloud service seems to be popular enough to trigger many requests pretending to be a Verizon Galaxy S7 mobile device and end up as the third most popular crawler mobile device across the Cloud Service user base.

There seem to be only single User-Agent in-use with fixed Android version and also fixed Chrome browser version. Either this service “crawling” part is working seamlessly and doesn’t require any updates or it’s not in the spotlight as it is not directly generating any revenue.

Why Google is using a branded Verizon model variant (SM-G930V) and not a “generic” unlocked model version (SM-G930U) is not fully clear.

Motorola (Lenovo) Moto G4

• Proportion of reference traffic: 9 % - 11 %
• Reported Android versions: 6.0.1, 7.0 and 8.1.0
• Year released: 2016
• Viewport width: 360 (1080 physical)
• Diagonal screen size: 5.5”
• Processor: Octa-core (1.5 GHz, 1.2 GHz)
• Used by Chrome-Lighthouse and SpeedCurve WebPage Test

The Moto G4 is used as a default mobile device for the Chrome Lighthouse plugin and also used by SpeedCurve tests, being the fourth most popular mobile device in the crawling space.

Google (LG) Nexus 5

• Proportion of reference traffic: 4 % - 7 %
• Reported Android versions: 4.2.1
• Year released: 2013
• Viewport width: 360 (1080 physical)
• Diagonal screen size: 5”
• Processor: Quad-core (2.3 GHz)
• Used by Google

Mainly used by Google Web Light service which is used to optimize web pages for users with slower internet connections. Using quite old Android (4.2.1) and Chrome versions (38.x). The use of this crawler / service seems to be continuously quite decreasing over time.

Generic Android

• Proportion of reference traffic: 6 % - 7 %
• Reported Android versions: 5.0 and 7.0
• Year released: unknown
• Viewport width: unknown
• Diagonal screen size: unknown
• Processor: unknown
• Used by PetalBot and Bytespider

There are also some crawlers using generic Android User-Agents without any model information. This might once become a standard if the adoption of Client Hints properly kicks off. You can find more information about the Client Hints in our Resources section.

Reference tablets

For a given period there was a very limited use of tablet User-Agents by crawlers (below 1 % of all crawlers traffic) and none of it by the big search engine players like Google or Bing.

Apple iPad

Various Android tablets

Web crawlers detection

With DeviceAtlas you can identify non-human traffic (robots, crawlers, checkers, download agents, spam harvesters and feed readers) in real-time. You can then decide how to act on this information, whether to block all undesired bots at the door, or just treat them in a different way to legitimate human visitors.

Read more about bot detection and how it can help your business to:

• Limit Ad-fraud
• Protect your site

We also have some handy resource lists such as a list of User Agents for the most popular smartphones and devices and the most common mobile browsers across 35 countries.