Google's common crawlers | Google Crawling Infrastructure  |  Crawling infrastructure  |  Google for Developers
Skip to main content
Crawling infrastructure
/
English
Deutsch
Español
Español – América Latina
Français
Indonesia
Italiano
Polski
Português – Brasil
Tiếng Việt
Türkçe
Русский
العربيّة
हिंदी
ภาษาไทย
中文 – 简体
中文 – 繁體
日本語
한국어
Sign in
Home
Docs
Crawling infrastructure
Home
Docs
Intro
About Google's web crawling
How to...
Verify requests from Google
Authenticate requests with Web Bot Auth (experimental)
Reduce Google's crawl rate
Use robots.txt to manage crawling
Create and submit a robots.txt file
How Google interprets the robots.txt specification
Update your robots.txt file
List of useful robots.txt rules
Optimize crawling performance
Optimize your crawl budget
Myths about crawling
Improve crawling of faceted navigation URLs
Reference
Common crawlers
Special case crawlers
User-triggered fetchers
Specific crawlers and user-triggered fetchers
APIs-Google
Feedfetcher
Googlebot
Google Read Aloud
Troubleshooting
HTTP status codes
Network and DNS errors
What's new
Changelog
Home
Crawling infrastructure
Docs
Send feedback
Stay organized with collections
Save and categorize content based on your preferences.
List of Google's common crawlers
Google's common crawlers are used to find information for building Google's search indexes,
perform other product specific crawls, and for analysis. They always obey
robots.txt rules
when crawling automatically. The general
technical properties
of Google's crawlers also apply to the common crawlers.
The common crawlers generally crawl from the IP ranges published in the
common-crawlers.json object, and the reverse DNS mask
of their hostname matches crawl-***-***-***-***.googlebot.com or
geo-crawl-***-***-***-***.geo.googlebot.com.
The following list shows the common crawlers, their user agent strings as they appear in the HTTP
requests, their user agent tokens for the User-agent: line in robots.txt, and the
products that are affected by crawl preferences for the crawler. Some crawlers have more than one
user agent token; you need to match only one crawler token for a rule to apply. The list is not
exhaustive, it only covers the requestors that are more likely to show up in log files and that
we've received questions about.
Caution: The HTTP user agent string can be spoofed.
Learn how to verify if a visitor is a Google crawler.
Googlebot
User-Agent in HTTP requests
Googlebot Smartphone
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot Desktop
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Rarely:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
robots.txt
User-agent token in robots.txt
Googlebot
Example robots.txt group
user-agent: Googlebot
allow: /archive/1Q84
disallow: /archive
Affected products
Crawling preferences addressed to the Googlebot user agent affect Google
Search (including Discover and all Google Search features), as well as other products
such as Google Images, Google Video, Google News, and Discover.
Googlebot Image
User-Agent in HTTP requests
Googlebot-Image/1.0
robots.txt
User-agent token in robots.txt
Googlebot-Image
Googlebot
Example robots.txt group
user-agent: Googlebot-Image
allow: /archive/1Q84
disallow: /archive/moons.jpg
Affected products
Crawling preferences addressed to the Googlebot-Image user agent affect
Google Images, Discover, Google Video, and all features in Google Search where images,
logos, and favicons are presented.
Googlebot Video
User-Agent in HTTP requests
Googlebot-Video/1.0
robots.txt
User-agent token in robots.txt
Googlebot-Video
Googlebot
Example robots.txt group
user-agent: Googlebot-Video
allow: /archive/1Q84
disallow: /archive/
Affected products
Crawling preferences addressed to the Googlebot-Video user agent affect
video-related Google Search features and other products dependent on videos.
Googlebot News
User-Agent in HTTP requests
Googlebot-news doesn't have a separate HTTP request user agent string. Crawling is done
with various Googlebot user agent strings.
robots.txt
User-agent token in robots.txt
Googlebot-News
Googlebot
Example robots.txt group
user-agent: Googlebot-News
allow: /archive/1Q84
disallow: /archive/
Affected products
Crawling preferences addressed to the Googlebot-News user agent affect
the Google News product, including
news.google.com and the
Google News app.
Google StoreBot
User-Agent in HTTP requests
Desktop agent
Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36
Mobile agent
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
robots.txt
User-agent token in robots.txt
Storebot-Google
Example robots.txt group
user-agent: Storebot-Google
allow: /archive/1Q84
disallow: /archive/konbini
Affected products
Crawling preferences addressed to the Storebot-Google user agent affect
all surfaces of Google Shopping (for example, the Shopping tab in Google Search and
Google Shopping).
Google-InspectionTool
User-Agent in HTTP requests
Desktop agent
Mozilla/5.0 (compatible; Google-InspectionTool/1.0)
Mobile agent
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)
robots.txt
User-agent token in robots.txt
Google-InspectionTool
Googlebot
Example robots.txt group
user-agent: Google-InspectionTool
allow: /archive/1Q84
disallow: /archive/
Affected products
Crawling preferences addressed to the Google-InspectionTool user agent
affect Search testing tools such as the
Rich Result Test
and
URL inspection
in Search Console. It has no effect on Google Search or other products.
GoogleOther
User-Agent in HTTP requests
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36
robots.txt
User-agent token in robots.txt
GoogleOther
Example robots.txt group
user-agent: GoogleOther
allow: /archive/1Q84
disallow: /archive/
Affected products
Crawling preferences addressed to the GoogleOther user agent don't affect
any specific product. GoogleOther is the generic crawler that may be used by various
product teams for fetching publicly accessible content from sites. For example, it may
be used for one-off crawls for internal research and development.
GoogleOther-Image
User-Agent in HTTP requests
GoogleOther-Image/1.0
robots.txt
User-agent token in robots.txt
GoogleOther-Image
GoogleOther
Example robots.txt group
user-agent: GoogleOther-Image
allow: /archive/1Q84
disallow: /archive/moon.jpg
Affected products
Crawling preferences addressed to the GoogleOther-Image user agent don't
affect any specific product, similar to GoogleOther. GoogleOther-Image is the version
of GoogleOther optimized for fetching publicly accessible image URLs.
GoogleOther-Video
User-Agent in HTTP requests
GoogleOther-Video/1.0
robots.txt
User-agent token in robots.txt
GoogleOther-Video
GoogleOther
Example robots.txt group
user-agent: GoogleOther-Video
allow: /archive/1Q84
disallow: /archive
Affected products
Crawling preferences addressed to the GoogleOther-Video user agent don't
affect any specific product, similar to GoogleOther. GoogleOther-Video is the version
of GoogleOther optimized for fetching publicly accessible video URLs.
Google-CloudVertexBot
User-Agent substring in HTTP requests
Google-CloudVertexBot
robots.txt
User-agent token in robots.txt
Google-CloudVertexBot
Googlebot
Example robots.txt group
user-agent: Google-CloudVertexBot
allow: /archive/1Q84
disallow: /archive/
Affected products
Crawling preferences addressed to the Google-CloudVertexBot user agent
affect crawls requested by the site owners' for building
Vertex AI Agents.
It has no effect on Google Search or other products.
Google-Extended
User-Agent in HTTP requests
Google-Extended doesn't have a separate HTTP request user agent string. Crawling is
done with existing Google user agent strings; the robots.txt user-agent token is used
in a control capacity.
robots.txt
User-agent token in robots.txt
Google-Extended
Example robots.txt group
user-agent: Google-Extended
allow: /archive/1Q84
disallow: /archive/
Affected products
Google-Extended is a standalone product token that web publishers can use
to manage whether content Google crawls from their sites may be used for training future
generations of Gemini models that power
Gemini Apps
and Vertex AI API for Gemini
and for grounding (providing content from the Google Search index to the model at prompt
time to improve factuality and relevancy) in
Gemini Apps
and Grounding with Google Search on Vertex AI.
Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal
in Google Search.
A note about Chrome/W.X.Y.Z in user agents
The string Chrome/W.X.Y.Z in the user agent strings in the list is a placeholder
that represents the version of the Chrome browser used by that user agent: for example,
41.0.2272.96. This version number increases over time to
match the latest Chromium release version used by Googlebot.
If you are searching your logs or filtering your server for a user agent with this pattern, use
wildcards for the version number rather than specifying an exact version number.
Send feedback
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2026-07-14 UTC.
Need to tell us more?
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2026-07-14 UTC."],[],[]]
Connect
Blog
Bluesky
Instagram
LinkedIn
X (Twitter)
YouTube
Programs
Google Developer Program
Google Developer Groups
Google Developer Experts
Accelerators
Google Cloud & NVIDIA
Developer consoles
Google API Console
Google Cloud Platform Console
Google Play Console
Firebase Console
Actions on Google Console
Cast SDK Developer Console
Chrome Web Store Dashboard
Google Home Developer Console
Android
Chrome
Firebase
Google Cloud Platform
Google AI
All products
Terms
Privacy
Manage cookies
English
Deutsch
Español
Español – América Latina
Français
Indonesia
Italiano
Polski
Português – Brasil
Tiếng Việt
Türkçe
Русский
العربيّة
हिंदी
ภาษาไทย
中文 – 简体
中文 – 繁體
日本語
한국어