Closed Bug 1393145 Opened 3 years ago Closed 3 years ago
Add crawl-delay directive to robots
I think there's an instruction that can be added to robots.txt that causes reasonable search engines to crawl more slowly -- can you look this up and add it? thanks!
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Summary: See if we can add a directive/rule to robots.txt to get search engines to crawl more slowly → Add crawl-delay directive to robots.txt
You need to log in before you can comment on or make changes to this bug.