Older release Bouncer URLs are 404

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
7 years ago
3 years ago

People

(Reporter: Smokey Ardisson (offline for a while; not following bugs - do not email), Assigned: justdave)

Tracking

Details

Attachments

(1 attachment)

I've just learned via a user in a bug that all the old Camino release Bouncer URLs are returning 404 (2.1a1, 2.0.7, and 1.6.11 appear to work; none of the others that I've tried still work).  They all used to work a couple of months ago when I last worked on the web pages using the Bouncer links.

Not sure if this is a releng issue or an IT issue or a "Bouncer" software issue; filing in releng first since that's where things get added to Bouncer.
More than just Camino it turns out. Bouncer only reports uptake for the following products:
Camino-1.6.11
Camino-2.0.7
Camino-2.1a1
Firefox-3.5.19
Firefox-3.5.19-Complete
Firefox-3.5.19-Partial-3.5.18
Firefox-3.6.16
Firefox-3.6.16-EUballot
Firefox-3.6.16-Partial-3.6.15
Firefox-3.6.17
Firefox-3.6.17-Complete
Firefox-3.6.17-Partial-3.6.16
Firefox-3.6.18
Firefox-3.6.18-Complete
Firefox-3.6.18-Partial-3.6.17
Firefox-3.6.18-Partial-3.6.17
Firefox-4.0
Firefox-4.0-Complete
Firefox-4.0-EUBallot
Firefox-4.0.1
Firefox-4.0.1-Complete
Firefox-4.0.1-EUballot
Firefox-4.0.1-Partial-4.0rc2
Firefox-4.0b11
Firefox-4.0b11-Complete
Firefox-4.0b11-Partial-4.0b10
Firefox-4.0b12
Firefox-4.0b12-Complete
Firefox-4.0b12-Partial-4.0b11
Firefox-4.0rc2
Firefox-4.0rc2-Complete
Firefox-4.0rc2-Partial-4.0rc1
Firefox-5.0
Firefox-5.0-Complete
Firefox-5.0-EUballot
Firefox-5.0-Partial-4.0.1
Firefox-5.0b2
Firefox-5.0b2-Complete
Firefox-5.0b2-Partial-5.0b1
Firefox-5.0b3
Firefox-5.0b3-Complete
Firefox-5.0b3-Partial-5.0b2
Firefox-5.0b5
Firefox-5.0b5-Complete
Firefox-5.0b5-Partial-5.0b4
Firefox-5.0b6
Firefox-5.0b6-Complete
Firefox-5.0b6-Partial-5.0b5
Firefox-5.0b7
Firefox-5.0b7-Complete
Firefox-5.0b7-Partial-5.0b6
Firefox-Mobile-1.1
Firefox-Mobile-1.1-N900
Firefox-Mobile-4.0
Firefox-Mobile-4.0.1
Firefox-Mobile-4.0b5
Firefox-Mobile-4.0rc1
Firefox-Mobile-5.0
Firefox-Mobile-5.0b2
Firefox-Mobile-5.0b3
Firefox-Mobile-5.0b4
Firefox-Mobile-5.0b7
Nagios-Test-Product
SeaMonkey-2.0.14
SeaMonkey-2.0.14-Complete
SeaMonkey-2.0.14-Partial-2.0.13
SeaMonkey-2.1
SeaMonkey-2.1-Complete
SeaMonkey-2.1-Partial-2.1rc2
SeaMonkey-2.1b3
SeaMonkey-2.1b3-Complete
SeaMonkey-2.1b3-Partial-2.1b2
SeaMonkey-2.1rc1
SeaMonkey-2.1rc1-complete
SeaMonkey-2.1rc1-partial-2.1b3
SeaMonkey-2.1rc2
SeaMonkey-2.1rc2-Complete
SeaMonkey-2.1rc2-Partial-2.1rc1
SeaMonkey-2.2b1
SeaMonkey-2.2b1-Complete
Sunbird-1.0b1
Sunbird-1.0b1-Complete
Thunderbird-3.1.10
Thunderbird-3.1.10-Complete
Thunderbird-3.1.10-Partial-3.1.9
Thunderbird-3.1.11
Thunderbird-3.1.11-Complete
Thunderbird-3.1.11-Partial-3.1.10
Thunderbird-3.1.7
Thunderbird-3.1.7-04
Thunderbird-3.1.7-Complete
Thunderbird-3.1.7-Partial-3.1.6
Thunderbird-3.3a3
Thunderbird-3.3a3-Complete
Thunderbird-3.3a3-Partial-3.3a2
Thunderbird-5.0b1
Thunderbird-5.0b1-Complete
Thunderbird-5.0b1-Partial-3.3a3

That's a pretty good overlap with the list of products with Check Now set, which get checked every 5 minutes, modulo a couple of extras and some gaps for releases that were never pushed. I've left off the OS for brevity there.

What should be happening is that all products/OS pairs should have a weight of at least 1, from dm-download02. So either
* there's something wrong with the once-a-day check that looks at all products
* dm-download02 was AWOL while the above check last ran

Either way, over to Server Ops.
Assignee: nobody → server-ops
Component: Release Engineering → Server Operations
QA Contact: release → mrz
Summary: Older Camino release Bouncer URLs are 404 → Older release Bouncer URLs are 404
(In reply to comment #1)
> * dm-download02 was AWOL while the above check last ran

Not completely AWOL anyway, eg it's the only mirror providing coverage for Firefox-5.0b11.
Are all these back now?  dm-download02 is the only source for most of these, and it was down a couple times around the time this bug was filed under release load.
Assignee: server-ops → justdave
I just checked all of the Camino 2.0.x and 1.6.x releases, and everything is still 404 except for 2.0.7 and 1.6.11 (the two full releases listed in comment 0/1).
OK, I think the daily run is probably busted.  There was a 6 minute time limit placed on the checks for a given site on the every-5-minute checks to try to cut down on overlapping checks backing up, and I bet that time limit isn't excluding the daily checks (which it shouldn't be applied to)
(In reply to comment #5)
> OK, I think the daily run is probably busted.

Confirmed, fix has been applied, I just had it re-scan dm-download02.  Try again?
Created attachment 541170 [details]
dm-download02 scan report
(In reply to comment #6)
> (In reply to comment #5)
> > OK, I think the daily run is probably busted.
> 
> Confirmed, fix has been applied, I just had it re-scan dm-download02.  Try
> again?

All the Camino releases look good; I started a couple of downloads and then verified everything else with curl header inspection.  Thanks, justdave!
Something strange is still going on. Symptoms:
* All mirrors pulling mozilla-releases should be carrying all of camino/releases, yet the uptake looks like this
 Product        OS      Available  Total
 Camino-2.0.6	osx	33511      167624
 Camino-2.0.7	osx	115431     167624
The only difference there is that 2.0.6 is not Check Now while 2.0.7 is.
* Something old like Firefox-1.5.0.6 has no uptake at all, should have dm-download02 for weight 1
* If you look up the locations status for Firefox-1.5.0.6 (eg https://10.2.75.6/stats/locations/?p=120) then at the very bottom of the list there are few with ok status for mirrors which are not working

Perhaps the nightly run is still terminating early, or not running against all mirrors ?
What do you mean by "not working"?  Do you mean they time out, or you get a 404, or?

I know the OSX stuff tends to have a lower uptake in general because a lot of people ignore our .htaccess file, and don't apply the rules manually, either, which causes the dmg files to be served with the wrong mime type.  Bouncer checks that and pulls them out if they don't have the right mime type.  That wouldn't explain one release being different than the other though.

We do have some mirror admins who cherry-pick the releases they carry, and don't actually carry the entire module, too.  Generally this is okay, as bouncer won't send them traffic for releases they aren't carrying.  I could see that causing some of a difference if someone's excluding older versions of things.  The size of the difference seems unlikely though.
I took a closer look at the Firefox-1.5.0.6 example, and turns out it's red herring. It's a reporting problem in bouncer, where it says that location-mirror pairs are enabled even when the mirror isn't active. Presumably I just never noticed this before. Filed bug 667375.

Camino is still reporting less uptake for old versions than the most recent one - about 73k vs 119k today. Agree that your two points are going to cause a gap that big. I tried enabling Check Now on Camino-2.0.6 and it jumped up to 119k, which says to me the daily sentry run isn't a testing everything, or something.
FWIW, the download metrics show a big drop in downloads of old updates on 2011-05-26, which recovered on 2011-06-22.
I spot-checked a couple of old camino URLs and they worked fine.  If this is still a problem please let us know what versions/urls specifically do not work.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(In reply to Corey Shields [:cshields] from comment #13)
> I spot-checked a couple of old camino URLs and they worked fine.  If this is
> still a problem please let us know what versions/urls specifically do not
> work.

The actual 404 issue was fixed in comment 6 / confirmed in comment 8.

nthomas identified some other (backend?) issue in comment 9 et seq.; I don't understand what that is, so if it's still a problem, perhaps it needs a new bug.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.