681868 - Sometimes, loading www.mozilla.org/firefox/ gives you Finnish content with 404 HTTP code

Status: VERIFIED FIXED

(www.mozilla.org :: General, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

STR:

1. Load http://www.mozilla.net
2. Sometimes, you'll get the Finnish page (wtf?)

Here are some headers:

[22:29:38.326] GET http://mozilla.net/ [undefined 38ms]
[22:29:38.377] GET http://www.mozilla.org/firefox/ [HTTP/1.1 404 Not Found 136ms]

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: First GET request

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: What pp-web03 returns, cached (it hits a 404)

Comment 3

[Fred Wenzel [:wenzel]]

Seems like a trailing / is causing this: www.mozilla.org/firefox/ (sic) seems to trigger it consistently.

Comment 4

[Fred Wenzel [:wenzel]]

Nope, that's not it. "consistently" as in, for a few minutes, until everything went back to the expected behavior again.

Comment 5

[Andrei Hajdukewycz [:sancus]]

http://pastebin.mozilla.org/1309808 (lwp-request data point)

Comment 7

[Kohei Yoshino [:kohei]]

The same issue is happening sometimes on a PC in our office in Tokyo, Japan.
www.mozilla.com -> http://www.mozilla.org/fi/firefox/

Comment 8

[Anthony Ricaud (:rik)]

Kohei: Are you sure you are redirected to this URL ? What we've seen so far is Finnish content on http://www.mozilla.org/firefox without redirect and with a 404 return code.

Comment 9

[Kohei Yoshino [:kohei]]

So our issue might be different. Will file a new bug...

Comment 10

[Kohei Yoshino [:kohei]]

Um, our issue is the same. My comment 7 was wrong. Now I can confirm http://www.mozilla.org/firefox/ shows the Finnish content with 404. Sorry for confusing.

Comment 11

[:glob ✱]

i'm seeing this too (in australia); following a http://www.mozilla.com/ link redirected me to http://www.mozilla.org/firefox/ however the content was finnish.  refreshing the finnish page redirects me to http://www.mozilla.org/en-US/firefox/fx/

Comment 12

[Anthony Ricaud (:rik)]

We've seen reports from people on IE and iPhone. So I think we can rule out the issue coming from the User-Agent.

My next idea is some code reads the path (/firefox), trims it to two letters which gives finnish content. It is just a hunch and I can't find the code that does that and that wouldn't explain the transient issue.

Comment 15

[Shyam Mani [:fox2mike]]

Attachment: iPhone screenshot

That's what I consistently get on the iPhone.

Comment 17

[Michael Kurze [:michaelk]]

Getting the intermittent Finnish+404 problem as well (from the MV WiFi, both on and off MPT VPN).

The X-Backend-Server does not seem to affect things.
Also, trailing slash or not does not seem to make a difference.

Behavior seems to be consistent for duration of keepalive connection (I always get right/wrong page several times in a row).

Outsider's wild crystal ball guess:
We are using RewriteMap (got that from James Long's webdev blog post), and that has a randomize feature --> possibly new source for non-determinism. If there are only some stale caches or misconfigured webheads left, those might be hit intermittently.

Comment 18

[Peter Bengtsson [:peterbe]]

I can confirm that it happens for www.mozilla.com too. I'm in the UK. Strangely only happens in Firefox and Safari, not in Chrome. Just reloading in Safari fixes it as that puts you on the /en-US/... path.

Comment 19

[Jake Maul [:jakem]]

Have done a lot of digging on this.

The issue only seems to appear to when using "Accept-Encoding: gzip, deflate". gzip alone, or "gzip, deflates", or no Accept-Encoding works properly.

I believe this is a side-effect, however, of a deeper problem. I think what's happening is that someone, somehow generates a request that causes the servers to return a 404 instead of a 302. Zeus caches this 404 for ~30 seconds or so, causing this problem. However, if a 302 gets cached, that lasts for 10 minutes. So it's inconsistent.

"gzip, deflate" is by far the most common setting. The servers (and Zeus) return a "Vary: Content-Encoding" header, causing different versions to be cached based on the value of Content-Encoding. This is why the encoding is a side-effect... only "gzip, deflate" is affected, because whatever is triggering the problem uses that header, so the cache for that is the only one that gets messed up... or at least, the only one that gets messed up often enough that it ever fails.


At this point my guess is that something in the logic which determines what page you're supposed to land on is faulty. http://www.mozilla.org/firefox/ should redirect to either http://www.mozilla.org/en-US/firefox/fx/ or http://www.mozilla.org/en-US/firefox/new. I think something is causing it to spit out a 404 instead.

I think we have a separate bug causing the 404 page to misinterpret the first 2 characters of '/firefox' as a locale, and spit back the Finnish 404 page.

Here is a sample of some access_log entries that throw the 404: http://pastebin.mozilla.org/1310052

At a glance, I see Firefox 6, IE6, IE8, IE9, Firefox/1.0.7 (!), Firefox/1.5 (!), AppleWebKit/535.1, Firefox/3.6.18.

It's easy to generate such a list... just grep for 'GET /firefox/ ' and ' 404 '. :)

Comment 20

[Jake Maul [:jakem]]

Also... I can confirm this is not being caused by just one web server... I see the 404's in the access_logs on multiple frontends.

Comment 21

[Jake Maul [:jakem]]

I have reduced the Zeus setting 'webcache!errorpage_time' to 1 second. This will cause Zeus to only cache error pages (such as 404's) for 1 second, instead of 30. Effectively, the problem should now be 1/30th as prevalent as it was a few minutes ago.

This is *NOT* a fix... just a stopgap to make this have a smaller effect on things. We still need to find and fix the cause of this, and then ideally re-set this value back to the default of 30 seconds.

Comment 22

[John Slater]

Thanks Jake. Please keep us posted on further developments, as this is one of the big remaining things from the merge to fix. Definitely an odd problem!

Comment 23

[James Long (:jlongster)]

should be fixed in r94321

Comment 24

[James Long (:jlongster)]

pushed to production r94323.

This should be fixed now. The way to reproduce is to hit the mozilla.org/firefox URL with an Accept Language header of fi-FI.

curl -v -H 'Accept-Language: fi-FI' http://www.mozilla.org/firefox/

Comment 25

[raymond [:retornam] (inactive)]

tried  fr, de, en-US and they all work  on production http://www.mozilla.com -- >
http://www.mozilla.org/en-US/firefox/fx/

mozilla.net --> http://www.mozilla.org/en-US/firefox/fx/

Comment 26

[Anthony Ricaud (:rik)]

For posterity, this bug was already there before the merge but we never found out about it because most users would go from mozilla.com to mozilla.com/{locale}/firefox (or variants of this).

Comment 27

[John Slater]

Thanks guys! Definitely an interesting problem to dig up and fix...

Comment 28

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED -- checked on my iPhone, too.

Comment 29

[David Lawrence [:dkl]]

Restoring unintended removal of in-testsuite flag.