A summary of the DuckDuckBot Internet robot. Including details for the owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
Who owns the DuckDuckBot robot? Is it a good or a bad robot? And why is it visiting your website?
Shown below is a sample log file entry for the DuckDuckBot web robot. It’s derived from an Apache web server log file. From the log entry information about how the robot identifies itself, HTTP User Agent, and where it is hosted are given.
Server Log File
vntweb.co.uk 52.5.190.19 - - [30/Apr/2019:00:48:48 +0100] "GET /robots.txt HTTP/1.1" 301 323 "http://xxxsd.pro/" "Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)"
HTTP User Agent
DuckDuckBot-Https/1.1
IP Addresses
The observed IP address was 52.5.190.19.
WHOIS DNS command gives the following information about the IP address:
| NetRange: | 52.0.0.0 – 52.31.255.255 |
| NetName: | AT-88-Z |
| OrgName: | Amazon Technologies Inc. |
| Address: | 410 Terry Ave N. |
| City: | Seattle |
| StateProv: | WA |
| Country: | US |
| Updated: | 2017-01-28 |
| OrgNOCName: | Amazon AWS Network Operations |
As can be seen from the above the observed IP address is a part of a block assigned to Amazon AWS Network Operations.
Owner
DuckDuckGo
Country
USA
Verification
The DuckDuckGo website confirms the IP addresses used as:
- 107.20.237.51
- 23.21.226.191
- 107.21.1.8
- 54.208.102.37
Exclusion
The user-agent string includes a reference to the website https://duckduckgo.com/duckduckbot.
The referenced website confirms that the bot supports the WWW:RobotRules.
The DuckDuckGo website provides details about the IP addresses used and the changelog for their robot. Note that the changelog confirms a change was required to ensure conformity with the robots.txt exclusion standard, which was described at http://www.robotstxt.org/wc/exclusion.html#robotstxt, but is not currently available. Information is available on the same website https://www.robotstxt.org/robotstxt.html and also on the w3c website at https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1.
Details are not given about both preventing the robot from indexing the website and how to adjust its crawl rate.
To prevent DuckDuckGo from visiting your site, setting you may wish to try including the following entry in the robots.txt file.
User-agent: DuckDuckGo Disallow: /
Similarly to control the frequency of DuckDuckGo visiting your site, setting a minimum acceptable delay between consecutive requests try adding the following to the robots.txt file:
User-agent: DuckDuckGo Crawl-Delay: 10
In this example the delay has been set to 10 seconds.
As is common with website crawlers there is a delay between changes made to the robots.txt file and the change being implemented.
Take care making changes to the robots.txt file. A misunderstanding in configuration or an error in configuration can lead to important search engines excluding your website.
Further Info
DuckDuckGo is an Internet search website.
The website claims that they do not store personal information.
Selling a strong personal privacy message the site also says that they don’t track visitors outside of their website. and they respect the status of private browsing mode.
