A summary of the crawler4j Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the crawler4j robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the crawler4j web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 212.227.216.141 - - [23/Mar/2019:20:43:24 +0000] "GET /robots.txt HTTP/1.1" 301 315 "-" "crawler4j (https://github.com/yasserg/crawler4j/)"
HTTP User Agent
crawler4j
IP Addresses
The observed IP address was 212.227.216.141.
WHOIS DNS command gives the following information about the IP address:
| inetnum: | 54.36.148.0 – 54.36.151.255 |
| netname: | SCHLUND-NET |
| org-name: | 1&1 Internet SE |
| address: | Brauerstrasse 48 |
| address: | 76135 |
| address: | Karlsruhe |
| address: | GERMANY |
| last-modified: | 2016-12-02T16:53:23Z |
As can be seen from the above the observed IP address is a part of a block assigned to 1&1 Internet (Ionos).
nslookup DNS command gives
141.216.227.212.in-addr.arpa name = crawlbap135.1and1.org.
Owner
Yasser Ganjisaffar
Country
USA
Exclusion
The user-agent string includes a reference to the website https://github.com/yasserg/crawler4j/.
The referenced website doesn’t directly confirm that the bot supports the robots exclusion text But does show that within the code there is RobotstxtConfig() class.
The robots.txt exclusion standard,which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
It is to be assumed that the robot can be prevented from indexing the website and the crawl rate adjusted.
You may wish to try including the following entry in the robots.txt file to prevent Crawler4j from visiting your site
User-agent: Crawler4j Disallow: /
similarly to control the frequency of Crawler4j visiting your site, setting a minimum acceptable delay between consecutive requests try adding the following added to the robots.txt file:
User-agent: Crawler4j Crawl-Delay: 10
In this example the delay has been set to 10 seconds.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
The link in the sever log references a GitHub project under the ownership of Yasser Ganjisaffar, who is located in the San Francisco Bay area.
The Crawler4j is sub-titled as an open source web crawler for Java.
Utilising Crawler4j a multi-threaded web crawler can be setup in just a few minutes.
Many of the robots and crawlers which will visit your website can be reliably known to come from a consistent source. for example the GoogleBot will always originate from Google.
However, because Crawler4j is a configurable project available on GitHub its possible that different visits may be from different owners, with a correspondingly different set of configuration options. As an example the observed IP address is assigned to the ISP Ionos.
