Closed
Bug 1282126
Opened 9 years ago
Closed 9 years ago
Google unable to fetch https://developer.mozilla.org
Categories
(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)
Infrastructure & Operations Graveyard
WebOps: Community Platform
Tracking
(Not tracked)
RESOLVED
WORKSFORME
People
(Reporter: jwhitlock, Unassigned)
Details
(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/3150])
Attachments
(1 file)
135.53 KB,
image/png
|
Details |
Google's webmaster tools (https://www.google.com/webmasters/tools/robots-testing-tool?hl=en&authuser=1&siteUrl=https://developer.mozilla.org/) reports failures in fetching robots.txt:
6/24/16, 4:56 AM - Failed
6/24/16, 4:54 AM - Failed
6/24/16, 3:03 AM - Failed
6/24/16, 3:02 AM - Failed
6/24/16, 2:43 AM - Failed
6/24/16, 2:41 AM - Failed
6/23/16, 1:16 PM - Failed
6/21/16, 1:51 PM - Success
6/18/16, 9:50 PM - Success
I believe these are PST times.
Using the "Fetch as Google page", feature, attempting to fetch the page takes 15 seconds and reports "Temporarily unreachable". It has the same response for the homepage and /en-US/Firefox
A production push went out on June 21st that included a new robots.txt (bug 1281270). A second production push went out an hour ago. The content of the new robots.txt was checked with Google's tools, and allows expected URLs.
These fetches work in a browser, and when using Google's user agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
There may be an IP block for Googlebot, or other load balancer weirdness.
Reporter | ||
Comment 1•9 years ago
|
||
The output of attempting to fetch URLs in Google's search console. clicking on the lines gives no further details.
Reporter | ||
Comment 2•9 years ago
|
||
There have been additional fetches:
6/24/16, 12:25 PM - Success
6/24/16, 10:58 AM - Failure
6/24/16, 10:54 AM - Success
2 out of 3 isn't too bad... This looks like an intermittent issue, possibly around the changed file, that is becoming less of an issue. I'm lowering the priority but will check in again Monday.
Severity: normal → minor
Reporter | ||
Comment 3•9 years ago
|
||
No further errors or attempts to fetch /robots.txt over the weekend. A manual fetch of /robots.txt using the "Fetch as Google" feature returned immediately.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
Updated•7 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•