Google unable to fetch



2 years ago
3 months ago


(Reporter: jwhitlock, Unassigned)



(Whiteboard: [kanban:])


(1 attachment)



2 years ago
Google's webmaster tools ( reports failures in fetching robots.txt:

6/24/16, 4:56 AM - Failed
6/24/16, 4:54 AM - Failed
6/24/16, 3:03 AM - Failed
6/24/16, 3:02 AM - Failed 
6/24/16, 2:43 AM - Failed
6/24/16, 2:41 AM - Failed
6/23/16, 1:16 PM - Failed
6/21/16, 1:51 PM - Success
6/18/16, 9:50 PM - Success

I believe these are PST times.

Using the "Fetch as Google page", feature, attempting to fetch the page takes 15 seconds and reports "Temporarily unreachable".  It has the same response for the homepage and /en-US/Firefox

A production push went out on June 21st that included a new robots.txt (bug 1281270).  A second production push went out an hour ago.  The content of the new robots.txt was checked with Google's tools, and allows expected URLs.

These fetches work in a browser, and when using Google's user agent

Mozilla/5.0 (compatible; Googlebot/2.1; +

There may be an IP block for Googlebot, or other load balancer weirdness.


2 years ago
Whiteboard: [kanban:]

Comment 1

2 years ago
Created attachment 8764998 [details]
Failed Attempts.png

The output of attempting to fetch URLs in Google's search console. clicking on the lines gives no further details.

Comment 2

2 years ago
There have been additional fetches:

6/24/16, 12:25 PM - Success
6/24/16, 10:58 AM - Failure
6/24/16, 10:54 AM - Success

2 out of 3 isn't too bad... This looks like an intermittent issue, possibly around the changed file, that is becoming less of an issue. I'm lowering the priority but will check in again Monday.
Severity: normal → minor

Comment 3

2 years ago
No further errors or attempts to fetch /robots.txt over the weekend.  A manual fetch of /robots.txt using the "Fetch as Google" feature returned immediately.
Last Resolved: 2 years ago
Resolution: --- → WORKSFORME


3 months ago
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.