1166057 - mac/win l10n aurora/dev edition update don't apply

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Axel Hecht [:Pike]]

I failed to get updates for a few days on my mac german aurora install.

From my browser console:

AUS:SVC Downloader:onProgress - progress: 97402036/97402036
AUS:SVC Downloader:onStopRequest - original URI spec: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/05/2015-05-17-00-40-04-mozilla-aurora-l10n/firefox-40.0a2.de.mac.complete.mar, final URI spec: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/05/2015-05-17-00-40-04-mozilla-aurora-l10n/firefox-40.0a2.de.mac.complete.mar, status: 0
AUS:SVC Downloader:onStopRequest - status: 0, current fail: 0, max fail: 10, retryTimeout: 2000
AUS:SVC Downloader:_verifyDownload called
AUS:SVC Downloader:_verifyDownload downloaded size == expected size.
AUS:SVC Downloader:_verifyDownload hashes match.
AUS:SVC Downloader:onStopRequest - setting state to: pending
AUS:SVC Downloader:onStopRequest - attempting to stage update: Firefox Developer Edition 40.0a2
AUS:SVC readStatusFile - status: failed: 23, path: /Users/ahecht/Library/Caches/Mozilla/updates/Applications/FirefoxAurora/updates/0/update.status
AUS:SVC handleFallbackToCompleteUpdate - install of complete or only one patch offered failed.
AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: failed, status: failed: 23

Update URL is https://aus4.mozilla.org/update/3/Firefox/40.0a2/20150515091916/Darwin_x86_64-gcc3-u-i386-x86_64/de/aurora/Darwin%2014.1.0/default/default/update.xml?force=1

Comment 1

[Robert Strong (they/them - no direct email)]

Note: the error code is for a version downgrade mismatch

Comment 2

[Robert Strong (they/them - no direct email)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #1)
> Note: the error code is for a version downgrade mismatch
s/mismatch/error/

Comment 3

[Robert Strong (they/them - no direct email)]

The mar's product info is as follows:
Signature block found with 1 signature
1 additional block found:
  - Product Information Block:
    - MAR channel name: firefox-mozilla-aurora
    - Product version: 39.0a2

It should be 40.0a2

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Windows and mac repacks are still using the mar utility from before the version bump. This comes from this code snippet:
 http://hg.mozilla.org/build/buildbotcustom/file/99b03dfbc0c1/process/factory.py#l3524
Mac and windows don't clobber very often, and may be in a jacuzzi (small pool of dedicated machines), so the mar file is already on disk and we don't fetch a new copy.

We could fix this a few ways
* short term, clobber the builders  (DONE for all platforms, nightly & dep)
* fix the buildbot code to always grab the mar executable, it's not a big hit
* pass the version explicitly when creating mar files, so that we don't fallback to the compiled in default in the mar utility

mozilla-central doesn't seem to be affected, so http://hg.mozilla.org/build/mozharness/file/8bc2ecf8ad09/scripts/desktop_l10n.py may be doing something different (this is the future!).

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Updating summary, this affects mac & windows, all locales, partial and complete upadtes.

Comment 6

[Axel Hecht [:Pike]]

Nick, thanks for the analysis, and the band aid.

I guess most prominently my concern would be what happens on next merge day, both on aurora and on beta.

CC Laura 'cause there's an "anything goes" conversation around doing l10n assets.

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to Nick Thomas [:nthomas] from comment #4) 
> We could fix this a few ways
> * pass the version explicitly when creating mar files, so that we don't
> fallback to the compiled in default in the mar utility

We should require the version to avoid this. That would also let us deploy known-good versions of mar and mbsdiff via tooltool instead of building them every time.

Comment 9

[Robert Strong (they/them - no direct email)]

As long as they can be kept in sync with each branch as changes are made to the code without holding back client changes.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

There are a few locales which haven't built a new nightly yet
May 17 10:27 ./firefox-40.0a2.mk.mac.complete.mar    - Product version: 39.0a2
May 17 10:28 ./firefox-40.0a2.sr.mac.complete.mar    - Product version: 39.0a2
May 17 10:28 ./firefox-40.0a2.km.mac.complete.mar    - Product version: 39.0a2
May 17 10:38 ./firefox-40.0a2.es-MX.mac.complete.mar - Product version: 39.0a2
May 17 10:39 ./firefox-40.0a2.sl.mac.complete.mar    - Product version: 39.0a2
May 17 10:21 ./firefox-40.0a2.ms.mac.complete.mar    - Product version: 39.0a2
May 17 10:23 ./firefox-40.0a2.az.mac.complete.mar    - Product version: 39.0a2

Otherwise everything is 40.0a2.