A summary of the MBCrawler Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the MBCrawler robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the MBCrawler web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 174.129.1.66 - - [03/Jun/2019:14:06:57 +0100] "GET /robots.txt HTTP/1.1" 200 117 "-" "MBCrawler/1.0 (https://monitorbacklinks.com/robot)"
HTTP User Agent
MBCrawler/1.0
IP Addresses
The observed IP address was 174.129.1.66.
WHOIS DNS command gives the following information about the IP address:
| NetRange: | 174.129.0.0 – 174.129.255.255 |
| NetName: | AMAZON-EC2-5 |
| OrgName: | Amazon Web Services Inc. |
| Address: | P.O. Box 81226 |
| City: | Seattle |
| StateProv: | WA |
| PostalCode: | 98108-1226 |
| Country: | US |
| Updated: | 2018-09-19 |
As can be seen from the above the observed IP address is a part of a block assigned to Amazon Technologies Inc.
nslookup DNS command gives
66.1.129.174.in-addr.arpa name = ec2-174-129-1-66.compute-1.amazonaws.com.
Owner
Monitor Backlinks
Country
Hong Kong
Exclusion
The user-agent string includes a reference to the website https://monitorbacklinks.com/robot. (Currently not available in 2021)
The referenced website confirms that the bot supports the robots exclusion text and also obeys the crawl delay.
The MBCrawler website provides details about how their robot complies with the robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1.
Details are given about preventing the robot from indexing the website but no mention about how to adjust its crawl rate.
Their advice is based upon whitelist their web robot using the user agent MBCrawler/1.0. You may wish to try including the following entry in the robots.txt file to prevent MBCrawler from visiting your site
User-agent: MBCrawler/1.0 Disallow: /
You may wish to trial controlling the frequency of MBCrawler visiting your site with a minimum acceptable delay between consecutive requests by adding the following to the robots.txt file:
User-agent: MBCrawler/1.0 Crawl-Delay: 10
In this example the delay has been set to 10 seconds.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
Monitor Backlinks uses this robot/crawler as a part of their linking analysis.
Their website highlights the following as their benefits, helping you to:
- Build great links
- Monitor your brand and create relationships with influencers
- Recover valuable lost links
- Track keyword rankings over time
- Reverse engineer competitors keyword and backlink strategies
- Fight negative SEO by Efficiently disavowing bad links
- Simplify – manage all your backlink data in one place
The intent of the website service is to provide a monitoring tool for backlinks and keywords.
