A summary of the Mediatoolkitbot Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the Mediatoolkitbot robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the Mediatoolkitbot web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 213.186.1.201 - - [18/Apr/2019:11:21:59 +0100] "GET /robots.txt HTTP/1.1" 301 323 "-" "Mediatoolkitbot (complaints@mediatoolkit.com)"
HTTP User Agent
Mediatoolkitbot
IP Addresses
The observed IP address was 213.186.1.201.
WHOIS DNS command gives the following information about the IP address:
| inetnum: | 213.186.1.0 – 213.186.1.255 |
| netnum: | OMONIA_DC_PUBLIC |
| address: | Poljicka 5 HR-10000 Zagreb Croatia |
| country: | HR |
| last-modified: | 2018-12-20T09:12:07Z |
As can be seen from the above the observed IP address is a part of a block assigned to Omonia d.o.o..
nslookup DNS command gives
** server can’t find 201.1.186.213.in-addr.arpa: NXDOMAIN
Owner
Mediatoolkit
Country
Croatia
Exclusion
The user-agent string doesn’t reference a dedicated website page. But this page is available https://www.mediatoolkit.com/robot.
The page confirms that the bot supports the robots exclusion text and also obeys the crawl delay.
The robots.txt exclusion standard which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
The page provides details about preventing the robot from indexing the website and how to adjust its crawl rate.
It is confirmed that to prevent Mediatoolkitbot from visiting your site include the following entry in the robots.txt file
User-agent: Mediatoolkitbot Disallow: /
Also controlling the frequency of Mediatoolkitbot visiting your site, setting a minimum acceptable delay between consecutive requests can be set with the following added to the robots.txt file:
User-agent: Mediatoolkitbot Crawl-Delay: 10
In this example the delay has been set to 10 seconds.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
Mediatoolkit provides media monitoring and intelligence. Allowing you to know what people are saying. Including competitor analysis.
Their analysis tools include these features:
- Website and social media monitoring
- Search filters to fine tune the results
- Data analysis and dashboards
- Custom alerts and digests
- Data export
