408846 - Mysterious temporary 404s (like bigfoot!)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

I've heard this from enough people that it warrants some investigation.  Steps to reproduce:

1) Load http://www.mozilla.com
2) Get 302'd to http://www.mozilla.com/en-US/www.mozilla.com/en-US

If someone could give any other details as to headers or redirect patterns please do.  I'll dig into prefetch.php when I get a chance.

Comment 1

[Reed Loden [:reed]]

So, this also affects download.mozilla.org when it occurs. Why are we thinking this is a www.mozilla.com problem? It's possible, but still... It just seems strange how kicking apache seems to fix this (afaik?).

From my logs, this same thing has happened March 1st, August 10th, December 12th, and December 18th. It may have happened other times, but those are the only logs I have for it currently. Gavin might have better luck searching his logs.

Comment 2

[Jeremy Orem [:oremj]]

download.mozilla.org == 302 ->  www.mozilla.com

Comment 3

[:Gavin Sharp [email: [e-mail]]]

Attachment: #bmo nagios logrep

Comment 4

[Reed Loden [:reed]]

(In reply to comment #2)
> download.mozilla.org == 302 ->  www.mozilla.com

Hmm, you're right. I thought that got changed during the bouncer upgrade in August.

Comment 5

[Justin Fitzhugh]

mrz has some history on this...copying him for the background.

Comment 6

[matthew zeier [:mrz]]

Sounds like my Citrix case 31560639.

In this case, the 404 was generated by a Squid proxy on the Internet sending the following headers:

GET /en-us/firefox/ HTTP/1.0
HOST: www.mozilla.com
HOST: www.mozilla.com

Apache concatenates the host headers and returned a 404 for the host "www.mozilla.comwww.mozilla.com, which as explained below ended up cached.

From my case notes:

*** 

In the packet 533751 in trace nstrace10.conf (this was in response to the client request to the NS in packet 533726), the NS receives a 404 response from the Apache server and caches it. Per NS Engineering, a 404 response will be cached if –weakNegRelExpiry is greater than 0 and if the 404 Not Found response does not contain the Cache-Control: no-cache header.

Once the 404 response in the Cache expires, NS  will start storing 200-OK-with-no-cahce responses to this request as well. Engineering is treating this behavior as a bug and will fix it in the next release of 7.0.

Note that in this case, the weakNegRelExpiry on the DEFAULT content group is greater than 0 and the 404 Not Found response from the Apache server does not contain the Cache-Control: no-cache header.

weakNegRelExpiry
Use this parameter for all negative responses. This value is used only if the expiry time could not be figured out from any other source. Default value: 600 Minimum value: 0
Maximum value: 31536000

If weakNegRelExpiry is set to 0, the 404 response will not be cached at all.

***

However, due to another bug, I wasn't able to set that to 0 and have had it set to 1 (0 doesn't persist across reboots).  I had an email last week from Citrix letting me know that that bug has been fixed.  

We're running build 55.4 and the fix is in 58.2, the latest.  

Comment 7

[matthew zeier [:mrz]]

For the sake of being complete, this was fixed in build 57.1 (7.0) and build 49.1 (8.0).

Comment 8

[Michael Morgan [:morgamic]]

(In reply to comment #4)
> (In reply to comment #2)
> > download.mozilla.org == 302 ->  www.mozilla.com
> 
> Hmm, you're right. I thought that got changed during the bouncer upgrade in
> August.
It only 404's for services.

Comment 9

[matthew zeier [:mrz]]

Do you have a reproducible test case?

Comment 10

[matthew zeier [:mrz]]

btw, Big Foot isn't mysterious - wikipedia says so. 

http://en.wikipedia.org/wiki/Big_Foot :)

Comment 11

[matthew zeier [:mrz]]

Netscaler code upgrade was secretly pushed out during tonight's window.  I set weakNegRelExpiry to 0.  

Didn't have method to reproduce this prior so I'm not sure how fixed this is.  Re-open if it's not fixed.