A summary of the GridBot Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the GridBot robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the GridBot web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 165.193.233.222 - - [25/Mar/2019:14:23:39 +0000] "GET / HTTP/1.1" 200 10726 "-" "GridBot/1.0 crawler@sysomos.com"
HTTP User Agent
GridBot/1.0
IP Addresses
The observed IP address was 165.193.233.222.
WHOIS DNS command gives the following information about the IP address:
| NetRange: | 165.193.0.0 – 165.193.255.255 |
| NetName: | CENTURYLINK-LEGACY-SAVVIS-BLK194 |
| OrgName: | CenturyLink Communications, LLC |
| Address: | 100 CENTURYLINK DR |
| City: | Monroe |
| PostalCode: | 71201 |
| StateProv: | LA |
| Country: | US |
| Updated: | 2018-08-15 |
As can be seen from the above the observed IP address is a part of a block assigned to CenturyLink Data Services.
Owner
Sysomos with a re-branding to Meltwater Social
Country
USA
Exclusion
The user-agent string does not inclue a reference to a website. The link included is a contact email address for a domain.
The Sysomos website provides details about how their robot complies with the robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
There are no details about preventing the robot from indexing the website and how to adjust its crawl rate.
You may wish to try including the following entry in the robots.txt file to prevent GridBot from visiting your site
User-agent: GridBot Disallow: /
Similarly you may wist to trial controlling the frequency of GridBot visiting your site, setting a minimum acceptable delay between consecutive requests try adding the following to the robots.txt file:
User-agent: GridBot Crawl-Delay: 10
In this example, the delay has been set to 10 seconds.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
According to the website, Sysomos provides social media solutions for marketeers.
The company offers the automation, with an AI powered social media analytics engine, the ability to turn trillions of data points into contextual insights and predictions.
The company offers the following apps:
- Search
A social media monitoring and research engine - Discover
Build, scale and measure your social influencer programs - Display
Data visualization for real-time assessment - Publish
Manage content and relationships across social channels - Engage
Enhance your customer service - Analyze
visibility and control over your earned and owned data
