A summary of the AhrefsBot Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the AhrefsBot robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the AhrefsBot web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 54.36.149.31 - - [29/Apr/2019:13:58:35 +0100] "GET /change-colour-of-text-underline/ HTTP/1.1" 200 7990 "-" "Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)"
HTTP User Agent
AhrefsBot/6.1
IP Addresses
The observed IP address was 54.36.149.31.
WHOIS DNS command gives the following information about the IP address:
| inetnum: | 54.36.148.0 – 54.36.151.255 |
| org-name: | OVH-DEDICATED-FO |
| descr: | OVH SAS |
| address: | 2 rue Kellermann |
| address: | 59100 |
| address: | Roubaix |
| address: | FRANCE |
| last-modified: | 2014-09-05T10:47:15Z |
As can be seen from the above the observed IP address is a part of a block assigned to OVH.
nslookup DNS command gives
31.149.36.54.in-addr.arpa name = ip-54-36-149-31.a.ahrefs.com.
Owner
Ahrefs is an SEO analysis website based around link research. The site’s tools include:
- link checking
- rank tracking
- SERPs analysis
The AhrefsBot bot is used as a part of their data gathering.
Country
The company is headquartered in Singapore.
Exclusion
The user-agent string includes a reference to the website http://ahrefs.com/robot/
The referenced website confirms that the bot supports the robots exclusion text and also obeys the crawl delay.
The ahrefs website provides details about how their robot complies with the robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1.
Details are given about both preventing the robot from indexing the website and how to adjust its crawl rate.
Their advice is to include the following entry in the robots.txt file to prevent AhrefsBot from visiting your site
User-agent: AhrefsBot Disallow: /
Also to control the frequency of AhrefsBot visiting your site, setting a minimum acceptable delay between consecutive requests can be set with the following added to the robots.txt file:
User-agent: AhrefsBot Crawl-Delay: [value]
Where the value of Crawl-Delay parameter is a time in seconds (you can use fractional seconds too)
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
The AhrefsBot is an Internet robot which belongs to the Ahrefs website.
Ahrefs provides an SEO service allowing knowledge to be gained about your competitors.
The site lists many tools, these are the ones highlighted:
- Organic search report – understand the keywords for which your competitors rank in organic search.
- Keyword research – Use this to get keyword suggestions. Associated tools can help to decide which keywords to target.
- Backlink data – With its large index of live backlinks, backlinks and SEO metric information for any website can be shown.
- Content research – A keyword tool to highlight content which has performed best for social buzz, number of backlinks and organic search traffic.
- Rank checking – Review graphed data relating to desktop and mobile keyword rankings.
- Web monitoring – Emails are sent when backlinks are lost or gained either by you or your competitors.
The website claims that they maintain the world’s largest third-party database of search queries,
