A summary of the bitlybot Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the bitlybot robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the bitlybot web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 54.86.37.102 - - [24/Apr/2019:11:21:39 +0100] "GET /robots.txt HTTP/1.1" 200 117 "-" "bitlybot/3.0 (+http://bit.ly/)"
HTTP User Agent
bitlybot/3.0
IP Addresses
The observed IP address was 54.86.37.102.
WHOIS DNS command gives the following information about the IP address:
| NetRange: | 54.86.37.102 |
| NetName: | AMAZON-2011L |
| OrgName: | Amazon Technologies Inc. |
| Address: | 410 Terry Ave N. |
| City: | Seattle |
| PostalCode: | 98109 |
| StateProv: | WA |
| Country: | US |
| Updated: | 2017-01-28 |
As can be seen from the above the observed IP address is a part of a block assigned to Amazon Technologies Inc.
Owner
Bitly, Inc
Country
USA
Exclusion
The user-agent string includes a reference to the website http://bit.ly/.
The referenced website does not provide details about the bitlybot robot. There are no reference details confirming whether the bot supports the robots exclusion text and also obeys the crawl delay.
The robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
You may wish to trial including the following entry in the robots.txt file to prevent bitlybot from visiting your site
User-agent: bitlybot Disallow: /
Also to control the frequency of bitlybot visiting your site, setting a minimum acceptable delay between consecutive requests try adding the following added to the robots.txt file:
User-agent: bitlybot Crawl-Delay: 10
In this example the delay has been set to 10 seconds.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
Bitly provides a link shortening service. The shortened links can be easier to write and reference than a full link. Particularly if the URL to be referenced is long coming from a domain a number of sub pages and a long page or document name.
