A summary of the Googlebot Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the Googlebot robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the Googlebot web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 66.249.70.17 - - [24/Mar/2019:16:17:32 +0000] "GET /security-role-rsvp-code/ HTTP/1.1" 200 7618 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
HTTP User Agent
Googlebot/2.1
IP Addresses
The observed IP address was 66.249.70.17.
WHOIS DNS command gives the following information about the IP address:
| NetRange: | 66.249.64.0 – 66.249.95.255 |
| NetName: | |
| OrgName: | Google LLC |
| Address: | 1600 Amphitheatre Parkway |
| City: | Mountain View |
| PostalCode: | 94043 |
| StateProv: | CA |
| Country: | US |
| Updated: | 2018-10-24 |
As can be seen from the above the observed IP address is a part of a block assigned to Google LLC.
Owner
Country
USA
Verification
The Googlebot web page suggests using a reverse DNS lookup to verify that the observed visitor request does come from Googlebot.
Details about verifying the authenticity of the visitor are given on the Verifying Googlebot page.
On this page examples are given for using the host command to identify the IP address and subsequently to confirm that the IP address matches the given domain name.
Exclusion
The user-agent string includes a reference to the website http://www.google.com/bot.html.
The referenced website confirms that the bot supports the robots exclusion text and also obeys the crawl delay.
The Google website provides details about how their robot complies with the robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
Details are given about both preventing the robot from indexing the website and how to adjust its crawl rate.
Their advice is to include the following entry in the robots.txt file to prevent Googlebot from visiting your site
User-agent: Googlebot Disallow: /
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
Google uses its Googlebot as both a desktop crawler and a mobile crawler. Each simulates the corresponding visitor to the website.
