A summary of the DomainCrawler Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the DomainCrawler robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the DomainCrawler web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 176.74.192.36 - - [16/Apr/2019:22:00:08 +0100] "GET /robots.txt HTTP/1.1" 301 315 "-" "DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/vntweb.co.uk)"
HTTP User Agent
DomainCrawler/3.0
IP Addresses
The observed IP address was 176.74.192.36.
WHOIS DNS command gives the following information about the IP address:
| inetnum: | 176.74.192.0 – 176.74.192.63 |
| netname: | TRALEX-NET |
| org-name: | OVH-DEDICATED-FO |
| descr: | Tralex AB |
| address: | Box 45009 |
| address: | SE 103 40 Stockholm |
| address: | SWEDEN |
| last-modified: | 2011-04-12T10:07:40Z |
As can be seen from the above the observed IP address is a part of a block assigned to OVH.
Owner
Internet Vikings
Country
Sweden
Exclusion
The user-agent string does not include a reference to a page providing information about the robot. It does include a reference to the domain http://www.domaincrawler.com/
You may wish to trial including the following entry in the robots.txt file to prevent DomainCrawler from visiting your site
User-agent: DomainCrawler Disallow: /
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
The Domaincrawler website provides an About Us page describing the history of the project.
The page confirms that the project was started in 2006 by Rickard Vikström.
The project is the creation of a database with domain and SEO related information.
Updating of the website looks to have stalled. The vision for the future was the release of version 2.5, expected in 2010.
