A summary of the SEOkicks Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the SEOkicks robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the SEOkicks web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 95.216.96.244 - - [13/May/2019:12:10:58 +0100] "GET /robots.txt HTTP/1.1" 301 323 "-" "Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)"
HTTP User Agent
SEOkicks
IP Addresses
The observed IP address was 95.216.96.244.
WHOIS DNS command gives the following information about the IP address:
| inetnum: | 95.216.0.0 – 95.217.255.255 |
| netname: | DE-HETZNER-20090224 |
| org-name: | Hetzner Online GmbH |
| address: | Industriestrasse 25 |
| address: | D-91710 |
| address: | Gunzenhausen |
| address: | GERMANY |
| last-modified: | 2016-08-25T13:26:09Z |
As can be seen from the above the observed IP address is a part of a block assigned to Hetzner Online GmbH.
nslookup DNS command gives
244.96.216.95.in-addr.arpa name = crawler.seokicks.de.
Owner
SEOkicks
Country
Germany
Exclusion
The user-agent string includes a reference to the website https://www.seokicks.de/robot.html.
The referenced website confirms that the bot supports the robots exclusion text and by inference also obeys the crawl delay.
The SEOkicks website provides background about web robot terms, such as User agents and their own robot and why and how it works.
However, there are no specific details about how their robot complies with the robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1.
Taking their confirmation of SEOkicks as the user agent it would be expected that to prevent the robot from visiting all of your website the following entry should be added in the robots.txt file.
User-agent: SEOkicks Disallow: /
Also to control the frequency of SEOkicks visiting your site, configuring a minimum acceptable delay between consecutive requests, you may wish to try adding the following to the robots.txt file:
User-agent: SEOkicks Crawl-Delay: 10
In this example, the delay has been set to 10 seconds.
The website confirms that the break between visits is 2 seconds. The intention is not to put a burden on the website which is more than a human visitor. to this end the delay can be automatically extended if the response time is slow.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
SEOkicks is a backlink checker, based in Germany. Here it is one of the most frequently used backlink databases, according to the website.
Using its own crawlers SEOkicks collects link data which is made available via the website.
Entering a website address for link checking gives a list of:
- Referring domains
- Referring pages
- Top pages
- Status codes
- Anchor text
- Reverse IP
Some of the items in the list above require the user to be registered and logged in. There is a monthly charge to access the collated data in their database.
