desktop firefox beta repacks failing to some locale's complete mar

RESOLVED WORKSFORME

Status

Release Engineering
Release Automation: Other
RESOLVED WORKSFORME
3 years ago
2 years ago

People

(Reporter: jlund, Assigned: jlund)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Assignee)

Description

3 years ago
Traceback (most recent call last):
  File "c:/builds/moz2_slave/rel-m-beta-w64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 386, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "c:/builds/moz2_slave/rel-m-beta-w64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 183, in createRepacks
    "Failed locales: %s" % " ".join([x for x, _ in failed]))
Exception: Failed locales: be
(Assignee)

Updated

3 years ago
Assignee: nobody → jlund
(Assignee)

Comment 2

3 years ago
it looks like we are failing to run `mar -t $archive`[1] against some previous complete mars.

e.g. 
command: START
command: perl ../../../tools/update-packaging/unwrap_full_update.pl ../../../..//firefox-42.0b9.be.complete.mar
command: cwd: /builds/slave/rel-m-beta-lx_rpk_1-0000000000/mozilla-beta/obj-l10n/dist/previous
...
... # env
command: output:
Couldn't run "/builds/slave/rel-m-beta-lx_rpk_1-0000000000/mozilla-beta/obj-l10n/dist/host/bin/mar" -t at ../../../tools/update-packaging/unwrap_full_update.pl line 51.
command: ERROR

looking for that mar, it seems like it downloaded okay, and I can verify it exists:
Downloading http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/42.0b9-candidates/build1/update/linux-i686/be/firefox-42.0b9.complete.mar to firefox-42.0b9.be.complete.mar

[1] https://dxr.mozilla.org/mozilla-central/source/tools/update-packaging/unwrap_full_update.pl?from=unwrap_full_update.pl#51
If they all look like that it's bug 1214971, which we need to reopen with details including estimated time, which datacenter the slave is in. We've mostly seen this on linux repacks up till now so it's interesting that it was mostly windows this time. An mtr/traceroute to archive.mozilla.org would be useful, given bug 1170832 and the weird routing for Cloudfront (which is serving archive.m.o).

To recover the release, make a list of all the locales that failed for each platform, and run the standalone repack jobs
 https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Tagging_through_Signing#Standalone_Repack_Builders
You'll need to force the repack_complete for each platform after the standlone job is done, otherwise the updates job won't start.
(Assignee)

Comment 4

3 years ago
I'll re-open 1214971 with the details requested. I won't bother doing the standalone runs as we are going to do a build2

all failures in comment 1 look very similar
(Assignee)

Comment 5

3 years ago
fwiw - I'm just reading https://bugzilla.mozilla.org/show_bug.cgi?id=1214971#c0 and I noticed that my logs (comment 1) don't show a RETRY in the download getting the complete mar in question.
(Assignee)

Comment 6

3 years ago
these failures look identical to the beta9 issues rail had:

tracker: https://wiki.mozilla.org/Releases/Firefox_42.0b9/BuildNotes#Firefox_2

log example of beta9 failure: http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-win64_repack_1%2F10/builds/16/steps/run_script/logs/stdio

if it happens again for build2, I'll run standalone builders
(Assignee)

Updated

3 years ago
Blocks: 1211313
(Assignee)

Comment 7

3 years ago
similar things happened for build2. re-running the failed l10n builders worked so far.
(Assignee)

Comment 8

3 years ago
I'll leave this bug open as a blocker for all things 43. Not sure if we have another bug currently open for needing to run standalone repacks against intermittently failing locales when unpacking their complete mar
Summary: 43.0b1 build1 locales vi, cy, be, bn-IN failing across various platforms → desktop firefox beta repacks failing to some locale's complete mar
Haven't seen this since november
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.