Google's webmaster tools (https://www.google.com/webmasters/tools/robots-testing-tool?hl=en&authuser=1&siteUrl=https://developer.mozilla.org/) reports failures in fetching robots.txt: 6/24/16, 4:56 AM - Failed 6/24/16, 4:54 AM - Failed 6/24/16, 3:03 AM - Failed 6/24/16, 3:02 AM - Failed 6/24/16, 2:43 AM - Failed 6/24/16, 2:41 AM - Failed 6/23/16, 1:16 PM - Failed 6/21/16, 1:51 PM - Success 6/18/16, 9:50 PM - Success I believe these are PST times. Using the "Fetch as Google page", feature, attempting to fetch the page takes 15 seconds and reports "Temporarily unreachable". It has the same response for the homepage and /en-US/Firefox A production push went out on June 21st that included a new robots.txt (bug 1281270). A second production push went out an hour ago. The content of the new robots.txt was checked with Google's tools, and allows expected URLs. These fetches work in a browser, and when using Google's user agent Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) There may be an IP block for Googlebot, or other load balancer weirdness.
Created attachment 8764998 [details] Failed Attempts.png The output of attempting to fetch URLs in Google's search console. clicking on the lines gives no further details.
There have been additional fetches: 6/24/16, 12:25 PM - Success 6/24/16, 10:58 AM - Failure 6/24/16, 10:54 AM - Success 2 out of 3 isn't too bad... This looks like an intermittent issue, possibly around the changed file, that is becoming less of an issue. I'm lowering the priority but will check in again Monday.
Severity: normal → minor
No further errors or attempts to fetch /robots.txt over the weekend. A manual fetch of /robots.txt using the "Fetch as Google" feature returned immediately.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WORKSFORME
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.