Unable to download update MARs for 48.0build2 win32 on beta



Release Engineering
3 months ago
3 months ago


(Reporter: mhowell, Assigned: Callek)


Firefox Tracking Flags

(Not tracked)




3 months ago
There appear to be two issues with downloading update MARs for 48.0b2, which are resulting in failure of all updates to that version.
1) The path to the complete MAR returned in the update.xml returns a 404 error. That URL is:
2) Versions that (as far as I can tell) should be offered a partial MAR for 48.0b2 (I tested 47.0b9) are getting update.xml results that only list the complete update.

This may be a cause of bug 1277925.

Note I'm assuming 48.0b2 is a watershed, because all betas between 44 and 47 get offered 48.0b2, but 48.0b1 doesn't, so I'm not sure.

For reference, I have 47.0b9 requesting this update.xml URL:

And receiving this response:
<update type="minor" displayVersion="48.0" appVersion="48.0" platformVersion="48.0"
<patch type="complete"
Redirecting to releaseduty folks - this doesn't look like a problem with Balrog itself.
Component: Balrog: Backend → Releases
Flags: needinfo?(mtabara)
Flags: needinfo?(jlund)
QA Contact: bhearsum → rail
I was also unable to download the mar for 44.0b1

Comment 3

3 months ago
Good detective work!

What you are being served: It's a bit confusing but the beta you are being served is 48.0build2, not 48.0b2 (beta 2). This is essentially the 48.0 release candidate that eventually becomes the 48.0 release we serve to release users but we test it against beta users first. Users would have originally got it after 48.0beta10.

why you are not being offered a partial: partials available for this version are only: "47.0.1build1,47.0build3,48.0b10build1"

why you are being served it: As to why you are being served it, yes, you're right, it's a watershed. This watershed is for SSE (Bug 1271761). I presume to force SSE detection as we force all windows beta users to update to it prior to latest beta or sse de-support if you start declaring SSE after 48.0. However, see below as to why I suspect the beta rules are setup incorrectly for SSE.

why it is 404ing: This is the core of the issue. I can see the complete mar exists on archive.m.o http://archive.mozilla.org/pub/firefox/releases/48.0/update/win32/en-US/firefox-48.0.complete.mar and its sha+size matches what update.xml reports. However, download.m.o isn't aware about it. I suspect it was removed from bouncer. Maybe we stopped wanting to offer this version? I see on the release channel, the equivalent SSE watershed rules differ. SSE detection is 47.0.2, and 48.0.2 (not 48.0!) is the last available release if you have SSE.

rail, can you confirm 48.0build2 is the version to detect SSE and that it should be available on download.m.o.
Flags: needinfo?(rail)
Flags: needinfo?(mtabara)
Flags: needinfo?(jlund)

Comment 4

3 months ago
(In reply to Jordan Lund (:jlund) from comment #3)

> rail, can you confirm 48.0build2 is the version to detect SSE and that it
> should be available on download.m.o.

actually, reading the history of the rule: https://aus4-admin.mozilla.org/rules/370

it seems originally this detection was: Firefox-48.0b1-build2 and has since undergone 3 mapping/version changes via callek. 302ing needinfo
Flags: needinfo?(rail) → needinfo?(bugspam.Callek)
The "build2" suffix usually means that we serve the files from the candidates directory. I suspect that those files have been archived and this is why they are 404.

We probably should go through all used rules and the corresponding release blobs and remove those suffixes so we use files from the releases directory.

We also should think how to prevent these in the future.

Comment 6

3 months ago
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #5)
> The "build2" suffix usually means that we serve the files from the
> candidates directory. I suspect that those files have been archived and this
> is why they are 404.
> We probably should go through all used rules and the corresponding release
> blobs and remove those suffixes so we use files from the releases directory.
> We also should think how to prevent these in the future.

Looking deeper:

* There is no balrog rule not specifying "build2" for 48.0
* Archive.m.o has it in the official release dir, but *not* in the candidates dir:
  - https://archive.mozilla.org/pub/firefox/candidates/
  - Likely purged due to time
* The underlying issue seems to be a 'we can't point a beta release at an rc for a long standing rule'

I updated the rule just now, to point at the 'release' channel download.m.o location. Can we confirm its fixed?
Flags: needinfo?(bugspam.Callek) → needinfo?(mhowell)
On Windows 10 with Firefox 32 bit
44.0b1 updated to 48.0 using a complete update
48.0 updated to 52.0b1 using a complete update
Flags: needinfo?(mhowell)
This is a great news, bravo!


3 months ago
Assignee: nobody → bugspam.Callek
Last Resolved: 3 months ago
Resolution: --- → FIXED

Comment 9

3 months ago
Yep, looks good from here too. Thanks everyone!
Matt, thanks for the great detective work and sorry for the churn. :(
The other way to fix this is to remove the beta, beta-cdntest, and beta-localtest blocks from the fileUrls section in the Firefox-48.0-build2 release. That'd get rid of the partials which still point to the broken bouncer products (which goes back to me tidying up the candidates directory). Then requests would use the '*' block, and that uses the bouncer products which refer to firefox/releases/.

Turns out Firefox-44.0-build3 (which is a watershed for GTK3 requirement on beta) also has this problem. I trimmed the fileUrls block for it.


3 months ago
Summary: Unable to download updates MAR's for 48.0b2 win32 → Unable to download update MARs for 48.0build2 win32 on beta

Comment 12

3 months ago
hi, is something similar going on in bug 1335736?
I wrote up a quick hack to check all of the Releases that Balrog currently points at (code here: https://github.com/mozilla/balrog/compare/master...mozbhearsum:find-bad-mars), and found a bunch that weren't OK. Most of them appear to be test channels pointing at archive.mozilla.org, but there's about 2,000 download.mozilla.org URLs in the list, too. I've put the full list here: https://people-mozilla.org/~bhearsum/bad-sorted.txt

Those download.mozilla.org ones probably need some further digging - I don't have time to do that at the moment.
I had a look through the results. 

1, 20.5k out of 22.6k bad urls are for candidates directories, but if you drop the test channels you're down to ~500 in candidates. I suggest ignoring any channels in fileUrls which end in 'test' (ie *-localtest, *-cdntest), since they are not seen by end users. That also chops 35k out of the list of 75k urls to check.  The remaining issues with candidates are for beta in Firefox-12.0-build1 and Thunderbird-12.0.1-build1. Since we don't point at those releases on beta there's actually no issue there, but I've set it use download.m.o instead (like release does):

2, Ignored maple/oak/ash, and https://mozilla-nightly-updates.s3.amazonaws.com/esr-switcher/update.mar (from testing?)

3, The aurora 20160923004002 404s are from locales which are discontinued or broken, but still exist in Firefox-mozilla-aurora-nightly-latest. This is brx, kok, ks, and sat - bug 1303401. Their files have expired off mozilla-nightly-updates.s3.amazonaws.com so we get a 404. Removed them from Firefox-mozilla-aurora-nightly-latest:

4, Similarly aurora 20161101004006 is from the be locale dropped in bug 1304749, removed in from the latest blob with

5, http://download.mozilla.org/?product=fennec-52.0b2&os=android-api-15&lang=an and the rest of the locales. These single-locale updates are not a functional AFAIK. We've been using 'android' for the OS in bouncer for the last few cycles, but have "OS_BOUNCER": "android-api-15" in the Fennec-52.0b2 blob. Suspect a long-standing issue, no action taken. This failure will follow the current shipped Fennec version.

6, win64 partials for all locales, eg
We didn't have win64 builds until 42.0. I suspect the url creator doesn't know how to deal with the mismatch between the fileUrls block vs the platform one. No action taken.

7, gn locale prior to 45.0 on release, and 45.0b4 on beta, eg
http://download.mozilla.org/?product=firefox-45.0.2-partial-39.0&os=linux64&lang=gn (also other platforms and 41.0.2 & 42.0)
gn was added in Firefox 45 by bug 1241916, so another edge case for the url creator I think. No action taken.

8, backfill of 43.0.1 -> 47.0.2 partials, linux, linux64 and mac
Only generated win32 and win64 partials in bug 1309130. Hard for the url generator because the Firefox-47.0.2-build2-43.0.1 blob has non-windows platforms in it (Y???), although they do lack partials references for 43.0.1. The rule only targets windows. No action taken.

9, cak locale, all platforms, eg
cak was added in Firefox 47 (bug 1263473), so same as 7. No action taken.

10, 48.0 for all locales and files, eg
This is the beta channel and fallout from cleanup of the candidates directory last year. Swapped to products like 'firefox-48.0-partial-47.0' which are in the firefox/releases/48.0/ directory via

11, ka & kab locales, all other platforms too
Bug 1309232 added ka and kab in Firefox 51, so no partial possible from earlier than that. Like 7 and 9, no action taken.

12, https://cdmdownload.adobe.com/firefox/win/x86/primetime_gmp_win_x86_gmc_3052p - didn't manage to trace this one, perhaps async mangled the line ??
You need to log in before you can comment on or make changes to this bug.