A summary of the ltx71 Internet robot.Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the ltx71 robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the ltx71 web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 35.202.223.242 - - [15/Apr/2019:19:34:06 +0100] "GET /robots.txt HTTP/1.1" 301 323 "-" "ltx71 - (http://ltx71.com/)"
HTTP User Agent
ltx71
IP Addresses
The observed IP address was 35.202.223.242.
WHOIS DNS command gives the following information about the IP address:
| NetRange: | 35.192.0.0 – 35.207.255.255 |
| NetName: | GOOGLE-CLOUD |
| OrgName: | Google LLC |
| Address: | 1600 Amphitheatre Parkway |
| City: | Mountain View |
| PostalCode: | 94043 |
| StateProv: | CA |
| Country: | US |
| Updated: | 2017-12-21 |
As can be seen from the above the observed IP address is a part of a block assigned to Google LLC.
WHOIS for the domain includes
Domain Name: LTX71.COM
Registry Domain ID: 1854306429_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.tucows.com
Registrar URL: http://tucowsdomains.com
Updated Date: 2017-11-19T16:46:34
Creation Date: 2014-04-11T14:53:50
Registrar Registration Expiration Date: 2021-04-11T14:53:50
Registrar: TUCOWS, INC.
Registrar IANA ID: 69
Reseller: Hover
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
Registry Registrant ID:
Registrant Name: Contact Privacy Inc. Customer 0137199112
Registrant Organization: Contact Privacy Inc. Customer 0137199112
As can be seen the owners information has been hidden.
Owner
Unknown
Country
Unknown
Exclusion
The user-agent string includes a reference to the website http://ltx71.com/
The referenced website doesn’t give much information apart from a few details about the website.
The given user agent referenced in the robots .txt file is particularly long using
ltx71 – (http://ltx71.com/)
Unusually the information covers setting the exclusion based on both the robot and different directories.
The referenced website confirms that the bot supports the robots exclusion text and also obeys the crawl delay.
The ltx71 website provides details about how their robot complies with the robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
Details are given about both preventing the robot from indexing the website and how to adjust its crawl rate.
Their advice is to include the following entry in the robots.txt file to prevent ltx71 from visiting your site
User-agent: ltx71 - (http://ltx71.com/) Disallow: /
Also to control the frequency of ltx71 visiting your site, setting a minimum acceptable delay between consecutive requests can be set with the following added to the robots.txt file:
User-agent: ltx71 Crawl-Delay: 10
In this example, taken from their website the delay has been set to 10 seconds.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
The website says that the bot is used for security research purposes.
Also the website confirms that their crawling of a website is not malicious, noting only summary information of a page.
