434065 - Bouncer sometimes returns a 404

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

In testing updates for 3.0rc1 on the releasetest channel, QA is finding that sometimes they fail.

An example URL is 
  http://download.mozilla.org/?product=firefox-3.0rc1-complete&os=linux&lang=en-US

If I |curl -IL| that I sometimes get 
  nrthomas@mobi-mandala ~ $ curl -IL "http://download.mozilla.org/?product=firefox-3.0rc1-complete&os=linux&lang=en-US"
  HTTP/1.1 404 Not Found
  Date: Fri, 16 May 2008 17:15:23 GMT
  Server: Apache/2.2.3 (Red Hat)
  Set-Cookie: dmo=10.2.81.4.1210958123835435; path=/; expires=Sat, 16-May-09  17:15:23 GMT
  X-Powered-By: PHP/5.1.6
  Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private
  Pragma: no-cache
  Content-Type: text/html; charset=UTF-8

instead of something like 
  nrthomas@mobi-mandala ~ $ curl -IL "http://download.mozilla.org/?product=firefox-3.0rc1-complete&os=linux&lang=en-US"
  HTTP/1.1 302 Found
  Date: Fri, 16 May 2008 17:15:46 GMT
  Server: Apache/2.2.3 (Red Hat)
  Set-Cookie: dmo=10.2.81.4.1210958146586169; path=/; expires=Sat, 16-May-09 17:15:46 GMT
  X-Powered-By: PHP/5.1.6
  Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private
  Pragma: no-cache
  Location: http://chuangtzu.acc.umu.se/pub/mozilla.org/firefox/releases/3.0rc1/update/linux-i686/en-US/firefox-3.0rc1.complete.mar
Content-Type: text/html; charset=UTF-8

  HTTP/1.1 200 OK
  Date: Fri, 16 May 2008 17:15:46 GMT
  Server: Apache/2.2.6 (Unix)
  Last-Modified: Tue, 13 May 2008 18:19:56 GMT
  ETag: "110181a-8bba4e-b6158b00"
  Accept-Ranges: bytes
  Content-Length: 9157198
  Content-Type: application/octet-stream

ie get a 404 instead of a 302 to a mirror. Could this be fallout from the db move last night ? Getting a 404 has a bad affect on the updater, as it doesn't handle the error properly and ends up in a confused state.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

(from irc) oremj says that one of the slave databases for bouncer hadn't been replicating from its master. It's syncing up now, ETA an hour or so.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Added dependency, as nick says on irc this is blocking testing.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

What happens if someone testing hits this 404? If they retry, do they get redirected differently, and get successful update?

Comment 4

[Al Billings [:abillings - ex-MoCo]]

It just sits there trying to update forever.

I'm getting this over and over on Linux. Every now and then, I can get an update.

Comment 5

[Tony Chung [:tchung]]

This repros on windows and mac also.  Can we get an ETA when this will be fixed?  it's been over the hour mark.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Any update? Talking to QA on irc, they are still hitting this now at 1pm.

Comment 7

[Jeremy Orem [:oremj]]

Replication is caught up and I can't reproduce the problem anymore.  I'm out to lunch now, but would you check in with Mark if it is still a problem.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Mark is looking at it, and talking with QA on irc. QA was still having problems a few mins ago (13.05). They are retrying to see if its still a problem.

Comment 9

[Jeremy Orem [:oremj]]

This is fixed now.

Comment 10

[Tony Chung [:tchung]]

Verified.  normal speeds now.