A summary of the Mappy Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the Mappy robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the Mappy web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
www.vntweb.co.uk 2400:8500:1302:910:150:95:213:177 - - [24/Jul/2019:15:38:32 +0100] "GET /robots.txt HTTP/1.1" 200 116 "-" "Mozilla/5.0 (compatible; Mappy/1.0; +http://mappydata.net/bot/)"
HTTP User Agent
Mappy/1.0
IP Addresses
The observed IP address was 2400:8500:1302:910:150:95:213:177.
WHOIS DNS command gives the following information about the IP address:
| inet6num: | 2400:8500::/32 |
| netname: | interQ-JPNIC-JP-20110425 |
| descr: | Japan Network Information Center |
| address: | Urbannet-Kanda Bldg 4F, 3-6-2 Uchi-Kanda |
| address: | Chiyoda-ku, Tokyo 101-0047, Japan |
| Country: | JP |
| last-modified: | 2017-10-18T10:21:54Z |
As can be seen from the above the observed IP address is a part of a block assigned to JPNIC.
Owner
Country
Japan
Exclusion
The user-agent string includes a reference to the website http://mappydata.net/bot/.
The referenced website confirms that the bot supports the robots exclusion text and also obeys the crawl delay.
The Mappy bot website provides details about how their robot complies with the robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
Details are given about preventing the robot from indexing the website but no mention about how to adjust its crawl rate.
Their advice is to include the following entry in the robots.txt file to prevent Mappy bot from visiting your site
User-agent: Mappy Disallow: /
No details are provided regards controlling the frequency of the Mappy bot visiting your site. You may wish to trial setting a minimum acceptable delay between consecutive requests with the following added to the robots.txt file:
User-agent: Mappy Crawl-Delay: 10
In this example the delay has been set to 10 seconds.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
The mappy web robot is a crawler operated by CINC Corp in Japan.
The crawler is used as a part of their data gathering, visiting every web page on the the World Wide Web. The gathered data is analysed for use by the ContenSearch search engine.
