Closed Bug 1220764 Opened 9 years ago Closed 8 years ago

desktop firefox beta repacks failing to some locale's complete mar

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: jlund, Assigned: jlund)

References

Details

Traceback (most recent call last):
  File "c:/builds/moz2_slave/rel-m-beta-w64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 386, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "c:/builds/moz2_slave/rel-m-beta-w64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 183, in createRepacks
    "Failed locales: %s" % " ".join([x for x, _ in failed]))
Exception: Failed locales: be
Assignee: nobody → jlund
it looks like we are failing to run `mar -t $archive`[1] against some previous complete mars.

e.g. 
command: START
command: perl ../../../tools/update-packaging/unwrap_full_update.pl ../../../..//firefox-42.0b9.be.complete.mar
command: cwd: /builds/slave/rel-m-beta-lx_rpk_1-0000000000/mozilla-beta/obj-l10n/dist/previous
...
... # env
command: output:
Couldn't run "/builds/slave/rel-m-beta-lx_rpk_1-0000000000/mozilla-beta/obj-l10n/dist/host/bin/mar" -t at ../../../tools/update-packaging/unwrap_full_update.pl line 51.
command: ERROR

looking for that mar, it seems like it downloaded okay, and I can verify it exists:
Downloading http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/42.0b9-candidates/build1/update/linux-i686/be/firefox-42.0b9.complete.mar to firefox-42.0b9.be.complete.mar

[1] https://dxr.mozilla.org/mozilla-central/source/tools/update-packaging/unwrap_full_update.pl?from=unwrap_full_update.pl#51
If they all look like that it's bug 1214971, which we need to reopen with details including estimated time, which datacenter the slave is in. We've mostly seen this on linux repacks up till now so it's interesting that it was mostly windows this time. An mtr/traceroute to archive.mozilla.org would be useful, given bug 1170832 and the weird routing for Cloudfront (which is serving archive.m.o).

To recover the release, make a list of all the locales that failed for each platform, and run the standalone repack jobs
 https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Tagging_through_Signing#Standalone_Repack_Builders
You'll need to force the repack_complete for each platform after the standlone job is done, otherwise the updates job won't start.
I'll re-open 1214971 with the details requested. I won't bother doing the standalone runs as we are going to do a build2

all failures in comment 1 look very similar
fwiw - I'm just reading https://bugzilla.mozilla.org/show_bug.cgi?id=1214971#c0 and I noticed that my logs (comment 1) don't show a RETRY in the download getting the complete mar in question.
these failures look identical to the beta9 issues rail had:

tracker: https://wiki.mozilla.org/Releases/Firefox_42.0b9/BuildNotes#Firefox_2

log example of beta9 failure: http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-win64_repack_1%2F10/builds/16/steps/run_script/logs/stdio

if it happens again for build2, I'll run standalone builders
Blocks: 1211313
similar things happened for build2. re-running the failed l10n builders worked so far.
I'll leave this bug open as a blocker for all things 43. Not sure if we have another bug currently open for needing to run standalone repacks against intermittently failing locales when unpacking their complete mar
Summary: 43.0b1 build1 locales vi, cy, be, bn-IN failing across various platforms → desktop firefox beta repacks failing to some locale's complete mar
Haven't seen this since november
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.