I've just learned via a user in a bug that all the old Camino release Bouncer URLs are returning 404 (2.1a1, 2.0.7, and 1.6.11 appear to work; none of the others that I've tried still work). They all used to work a couple of months ago when I last worked on the web pages using the Bouncer links. Not sure if this is a releng issue or an IT issue or a "Bouncer" software issue; filing in releng first since that's where things get added to Bouncer.
More than just Camino it turns out. Bouncer only reports uptake for the following products: Camino-1.6.11 Camino-2.0.7 Camino-2.1a1 Firefox-3.5.19 Firefox-3.5.19-Complete Firefox-3.5.19-Partial-3.5.18 Firefox-3.6.16 Firefox-3.6.16-EUballot Firefox-3.6.16-Partial-3.6.15 Firefox-3.6.17 Firefox-3.6.17-Complete Firefox-3.6.17-Partial-3.6.16 Firefox-3.6.18 Firefox-3.6.18-Complete Firefox-3.6.18-Partial-3.6.17 Firefox-3.6.18-Partial-3.6.17 Firefox-4.0 Firefox-4.0-Complete Firefox-4.0-EUBallot Firefox-4.0.1 Firefox-4.0.1-Complete Firefox-4.0.1-EUballot Firefox-4.0.1-Partial-4.0rc2 Firefox-4.0b11 Firefox-4.0b11-Complete Firefox-4.0b11-Partial-4.0b10 Firefox-4.0b12 Firefox-4.0b12-Complete Firefox-4.0b12-Partial-4.0b11 Firefox-4.0rc2 Firefox-4.0rc2-Complete Firefox-4.0rc2-Partial-4.0rc1 Firefox-5.0 Firefox-5.0-Complete Firefox-5.0-EUballot Firefox-5.0-Partial-4.0.1 Firefox-5.0b2 Firefox-5.0b2-Complete Firefox-5.0b2-Partial-5.0b1 Firefox-5.0b3 Firefox-5.0b3-Complete Firefox-5.0b3-Partial-5.0b2 Firefox-5.0b5 Firefox-5.0b5-Complete Firefox-5.0b5-Partial-5.0b4 Firefox-5.0b6 Firefox-5.0b6-Complete Firefox-5.0b6-Partial-5.0b5 Firefox-5.0b7 Firefox-5.0b7-Complete Firefox-5.0b7-Partial-5.0b6 Firefox-Mobile-1.1 Firefox-Mobile-1.1-N900 Firefox-Mobile-4.0 Firefox-Mobile-4.0.1 Firefox-Mobile-4.0b5 Firefox-Mobile-4.0rc1 Firefox-Mobile-5.0 Firefox-Mobile-5.0b2 Firefox-Mobile-5.0b3 Firefox-Mobile-5.0b4 Firefox-Mobile-5.0b7 Nagios-Test-Product SeaMonkey-2.0.14 SeaMonkey-2.0.14-Complete SeaMonkey-2.0.14-Partial-2.0.13 SeaMonkey-2.1 SeaMonkey-2.1-Complete SeaMonkey-2.1-Partial-2.1rc2 SeaMonkey-2.1b3 SeaMonkey-2.1b3-Complete SeaMonkey-2.1b3-Partial-2.1b2 SeaMonkey-2.1rc1 SeaMonkey-2.1rc1-complete SeaMonkey-2.1rc1-partial-2.1b3 SeaMonkey-2.1rc2 SeaMonkey-2.1rc2-Complete SeaMonkey-2.1rc2-Partial-2.1rc1 SeaMonkey-2.2b1 SeaMonkey-2.2b1-Complete Sunbird-1.0b1 Sunbird-1.0b1-Complete Thunderbird-3.1.10 Thunderbird-3.1.10-Complete Thunderbird-3.1.10-Partial-3.1.9 Thunderbird-3.1.11 Thunderbird-3.1.11-Complete Thunderbird-3.1.11-Partial-3.1.10 Thunderbird-3.1.7 Thunderbird-3.1.7-04 Thunderbird-3.1.7-Complete Thunderbird-3.1.7-Partial-3.1.6 Thunderbird-3.3a3 Thunderbird-3.3a3-Complete Thunderbird-3.3a3-Partial-3.3a2 Thunderbird-5.0b1 Thunderbird-5.0b1-Complete Thunderbird-5.0b1-Partial-3.3a3 That's a pretty good overlap with the list of products with Check Now set, which get checked every 5 minutes, modulo a couple of extras and some gaps for releases that were never pushed. I've left off the OS for brevity there. What should be happening is that all products/OS pairs should have a weight of at least 1, from dm-download02. So either * there's something wrong with the once-a-day check that looks at all products * dm-download02 was AWOL while the above check last ran Either way, over to Server Ops.
Assignee: nobody → server-ops
Component: Release Engineering → Server Operations
QA Contact: release → mrz
Summary: Older Camino release Bouncer URLs are 404 → Older release Bouncer URLs are 404
(In reply to comment #1) > * dm-download02 was AWOL while the above check last ran Not completely AWOL anyway, eg it's the only mirror providing coverage for Firefox-5.0b11.
Are all these back now? dm-download02 is the only source for most of these, and it was down a couple times around the time this bug was filed under release load.
Assignee: server-ops → justdave
I just checked all of the Camino 2.0.x and 1.6.x releases, and everything is still 404 except for 2.0.7 and 1.6.11 (the two full releases listed in comment 0/1).
OK, I think the daily run is probably busted. There was a 6 minute time limit placed on the checks for a given site on the every-5-minute checks to try to cut down on overlapping checks backing up, and I bet that time limit isn't excluding the daily checks (which it shouldn't be applied to)
(In reply to comment #5) > OK, I think the daily run is probably busted. Confirmed, fix has been applied, I just had it re-scan dm-download02. Try again?
(In reply to comment #6) > (In reply to comment #5) > > OK, I think the daily run is probably busted. > > Confirmed, fix has been applied, I just had it re-scan dm-download02. Try > again? All the Camino releases look good; I started a couple of downloads and then verified everything else with curl header inspection. Thanks, justdave!
Something strange is still going on. Symptoms: * All mirrors pulling mozilla-releases should be carrying all of camino/releases, yet the uptake looks like this Product OS Available Total Camino-2.0.6 osx 33511 167624 Camino-2.0.7 osx 115431 167624 The only difference there is that 2.0.6 is not Check Now while 2.0.7 is. * Something old like Firefox-22.214.171.124 has no uptake at all, should have dm-download02 for weight 1 * If you look up the locations status for Firefox-126.96.36.199 (eg https://10.2.75.6/stats/locations/?p=120) then at the very bottom of the list there are few with ok status for mirrors which are not working Perhaps the nightly run is still terminating early, or not running against all mirrors ?
What do you mean by "not working"? Do you mean they time out, or you get a 404, or? I know the OSX stuff tends to have a lower uptake in general because a lot of people ignore our .htaccess file, and don't apply the rules manually, either, which causes the dmg files to be served with the wrong mime type. Bouncer checks that and pulls them out if they don't have the right mime type. That wouldn't explain one release being different than the other though. We do have some mirror admins who cherry-pick the releases they carry, and don't actually carry the entire module, too. Generally this is okay, as bouncer won't send them traffic for releases they aren't carrying. I could see that causing some of a difference if someone's excluding older versions of things. The size of the difference seems unlikely though.
I took a closer look at the Firefox-188.8.131.52 example, and turns out it's red herring. It's a reporting problem in bouncer, where it says that location-mirror pairs are enabled even when the mirror isn't active. Presumably I just never noticed this before. Filed bug 667375. Camino is still reporting less uptake for old versions than the most recent one - about 73k vs 119k today. Agree that your two points are going to cause a gap that big. I tried enabling Check Now on Camino-2.0.6 and it jumped up to 119k, which says to me the daily sentry run isn't a testing everything, or something.
FWIW, the download metrics show a big drop in downloads of old updates on 2011-05-26, which recovered on 2011-06-22.
I spot-checked a couple of old camino URLs and they worked fine. If this is still a problem please let us know what versions/urls specifically do not work.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(In reply to Corey Shields [:cshields] from comment #13) > I spot-checked a couple of old camino URLs and they worked fine. If this is > still a problem please let us know what versions/urls specifically do not > work. The actual 404 issue was fixed in comment 6 / confirmed in comment 8. nthomas identified some other (backend?) issue in comment 9 et seq.; I don't understand what that is, so if it's still a problem, perhaps it needs a new bug.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.