A summary of the Gigabot Internet robot.Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the Gigabot robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the Gigabot web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 200.54.63.246 - - [14/Apr/2019:06:27:42 +0100] "GET /revslider-invalid-in-operand-a/downloader/ HTTP/1.1" 301 355 "-" "Gigabot/3.0 (http://www.gigablast.com/spider.html)"
HTTP User Agent
Gigabot/3.0
IP Addresses
The observed IP address was 200.54.63.246.
WHOIS DNS command gives the following information about the IP address:
| inetnum: | 200.54.63/24 |
| status: | reallocated |
| owner: | CL-TEEMSR-LACNIC |
| ownerid: | CL-CLTE-LACNIC |
| address: | Providencia, 111, 11 |
| address: | 1 – Santiago – |
| country: | CL |
| changed: | 20070628 |
Owner
Gigablast
Country
USA
Exclusion
The user-agent string includes a reference to the website http://www.gigablast.com/spider.html.
The referenced website doesn’t provide specific information about the spider. It references a bot/spider called EventGuruBot. It confirms that the bot supports the robots exclusion text but doesn’t mention the crawl delay.
The robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
Details are not given about both preventing the robot from indexing the website and how to adjust its crawl rate.
You may wish to trial including the following entry in the robots.txt file to prevent Gigabot from visiting your site:
User-agent: Gigabot Disallow: /
Also to control the frequency of Gigabot visiting your site, setting a minimum acceptable delay between consecutive requests trial with the following added to the robots.txt file:
User-agent: Gigabot Crawl-Delay: 10
In this example the delay has been set to 10 seconds.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
The Gigablast website is a search engine developed by Matt Wells.
The website referenced by the Gigablast spider summarises covers a search engine called Event Guru dedicated to providing an index of events in the USA.
Following a referenced link for Event Guru takes the visitor to the Gigablast home page.
The Gigablast home page shows the indexing is for more than just events.
The site is proud to state that it maintains its own searchable index.
