A summary of the BingBot Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the BingBot robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the BingBot web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 157.55.39.54 - - [04/May/2019:05:14:58 +0100] "GET /wkhtmltopdf-web-page-screen-capture/ HTTP/1.1" 200 8929 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
HTTP User Agent
bingbot/2.0
IP Addresses
The observed IP address was 157.55.39.54.
WHOIS DNS command gives the following information about the IP address:
| NetRange: | 157.54.0.0 – 157.60.255.255 |
| NetName: | MSFT-GFS |
| OrgName: | Microsoft Corporation |
| Address: | One Microsoft Way |
| City: | Redmond |
| PostalCode: | 98052 |
| StateProv: | WA |
| Country: | US |
| Updated: | 2017-01-28 |
As can be seen from the above the observed IP address is a part of a block assigned to Microsoft Corporation Azure hosting.
nslookup DNS command gives
54.39.55.157.in-addr.arpa name = msnbot-157-55-39-54.search.msn.com.
Owner
Microsoft
Country
USA
Verification
The bing website confirms example uses of their robot user-agent string:
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Exclusion
The user-agent string includes a reference to the website http://www.bing.com/bingbot.htm.
The Bing website confirms that their robot complies with the robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1.
However, details are not given about both preventing the robot from indexing the website and how to adjust its crawl rate.
You may wish to try adding the following entry in the robots.txt file to prevent BingBot from visiting your site
User-agent: BingBot Disallow: /
Similarly to control the frequency of BingBot visiting your site, setting a minimum acceptable delay between consecutive requests try setting with the following added to the robots.txt file:
User-agent: BingBot Crawl-Delay: 10
In this example the delay has been set to 10 seconds.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
Bing is Microsoft’s search engine.
To take further advantage of Microsoft’s Bing search engine you may wish to review the Webmaster pages.
