1282126 - Google unable to fetch https://developer.mozilla.org

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Description

[John Whitlock [:jwhitlock]]

Google's webmaster tools (https://www.google.com/webmasters/tools/robots-testing-tool?hl=en&authuser=1&siteUrl=https://developer.mozilla.org/) reports failures in fetching robots.txt:

6/24/16, 4:56 AM - Failed
6/24/16, 4:54 AM - Failed
6/24/16, 3:03 AM - Failed
6/24/16, 3:02 AM - Failed 
6/24/16, 2:43 AM - Failed
6/24/16, 2:41 AM - Failed
6/23/16, 1:16 PM - Failed
6/21/16, 1:51 PM - Success
6/18/16, 9:50 PM - Success

I believe these are PST times.

Using the "Fetch as Google page", feature, attempting to fetch the page takes 15 seconds and reports "Temporarily unreachable".  It has the same response for the homepage and /en-US/Firefox

A production push went out on June 21st that included a new robots.txt (bug 1281270).  A second production push went out an hour ago.  The content of the new robots.txt was checked with Google's tools, and allows expected URLs.

These fetches work in a browser, and when using Google's user agent

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

There may be an IP block for Googlebot, or other load balancer weirdness.

Comment 1

[John Whitlock [:jwhitlock]]

Attachment: Failed Attempts.png

The output of attempting to fetch URLs in Google's search console. clicking on the lines gives no further details.

Comment 2

[John Whitlock [:jwhitlock]]

There have been additional fetches:

6/24/16, 12:25 PM - Success
6/24/16, 10:58 AM - Failure
6/24/16, 10:54 AM - Success

2 out of 3 isn't too bad... This looks like an intermittent issue, possibly around the changed file, that is becoming less of an issue. I'm lowering the priority but will check in again Monday.

Comment 3

[John Whitlock [:jwhitlock]]

No further errors or attempts to fetch /robots.txt over the weekend.  A manual fetch of /robots.txt using the "Fetch as Google" feature returned immediately.