1220764 - desktop firefox beta repacks failing to some locale's complete mar

Status: RESOLVED WORKSFORME

(Release Engineering :: Release Automation, defect)

Description

[Jordan Lund (:jlund)]

Traceback (most recent call last):
  File "c:/builds/moz2_slave/rel-m-beta-w64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 386, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "c:/builds/moz2_slave/rel-m-beta-w64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 183, in createRepacks
    "Failed locales: %s" % " ".join([x for x, _ in failed]))
Exception: Failed locales: be

Comment 1

[Jordan Lund (:jlund)]

be:
   linux32: http://buildbot-master74.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-beta-linux_repack_1%2F10/builds/8/steps/run_script/logs/stdio
   win64: http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-win64_repack_1%2F10/builds/17/steps/run_script/logs/stdio

cy:
   win32: http://buildbot-master85.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-win32_repack_2%2F10/builds/24/steps/run_script/logs/stdio

bn-IN:
   win64: http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-win64_repack_2%2F10/builds/17/steps/run_script/logs/stdio

vi:
   win64: http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-win64_repack_10%2F10/builds/13/steps/run_script/logs/stdio




I'm guessing the win64 failures are due to this being a new platform. still investigating.

are these locales supposed to be enabled for beta on the platforms its failing?

Comment 2

[Jordan Lund (:jlund)]

it looks like we are failing to run `mar -t $archive`[1] against some previous complete mars.

e.g. 
command: START
command: perl ../../../tools/update-packaging/unwrap_full_update.pl ../../../..//firefox-42.0b9.be.complete.mar
command: cwd: /builds/slave/rel-m-beta-lx_rpk_1-0000000000/mozilla-beta/obj-l10n/dist/previous
...
... # env
command: output:
Couldn't run "/builds/slave/rel-m-beta-lx_rpk_1-0000000000/mozilla-beta/obj-l10n/dist/host/bin/mar" -t at ../../../tools/update-packaging/unwrap_full_update.pl line 51.
command: ERROR

looking for that mar, it seems like it downloaded okay, and I can verify it exists:
Downloading http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/42.0b9-candidates/build1/update/linux-i686/be/firefox-42.0b9.complete.mar to firefox-42.0b9.be.complete.mar

[1] https://dxr.mozilla.org/mozilla-central/source/tools/update-packaging/unwrap_full_update.pl?from=unwrap_full_update.pl#51

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

If they all look like that it's bug 1214971, which we need to reopen with details including estimated time, which datacenter the slave is in. We've mostly seen this on linux repacks up till now so it's interesting that it was mostly windows this time. An mtr/traceroute to archive.mozilla.org would be useful, given bug 1170832 and the weird routing for Cloudfront (which is serving archive.m.o).

To recover the release, make a list of all the locales that failed for each platform, and run the standalone repack jobs
 https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Tagging_through_Signing#Standalone_Repack_Builders
You'll need to force the repack_complete for each platform after the standlone job is done, otherwise the updates job won't start.

Comment 4

[Jordan Lund (:jlund)]

I'll re-open 1214971 with the details requested. I won't bother doing the standalone runs as we are going to do a build2

all failures in comment 1 look very similar

Comment 5

[Jordan Lund (:jlund)]

fwiw - I'm just reading https://bugzilla.mozilla.org/show_bug.cgi?id=1214971#c0 and I noticed that my logs (comment 1) don't show a RETRY in the download getting the complete mar in question.

Comment 6

[Jordan Lund (:jlund)]

these failures look identical to the beta9 issues rail had:

tracker: https://wiki.mozilla.org/Releases/Firefox_42.0b9/BuildNotes#Firefox_2

log example of beta9 failure: http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-win64_repack_1%2F10/builds/16/steps/run_script/logs/stdio

if it happens again for build2, I'll run standalone builders

Comment 7

[Jordan Lund (:jlund)]

similar things happened for build2. re-running the failed l10n builders worked so far.

Comment 8

[Jordan Lund (:jlund)]

I'll leave this bug open as a blocker for all things 43. Not sure if we have another bug currently open for needing to run standalone repacks against intermittently failing locales when unpacking their complete mar

Comment 9

[Justin Wood (:Callek)]

Haven't seen this since november