1405203 - Problems with win32 en-ZA and eo for DevEdition 57.0b5

Status: RESOLVED DUPLICATE of bug 1406403

(Release Engineering :: Release Automation: Other, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We had a lot of windows jobs retry for deved 57.0b5 because high prices in the AWS spot market caused instances to be terminated. This lead to 'devedition mozilla-beta win32 l10n repack 3/10' task [1] running twice, almost simultaneously [2,3]. Most probably we ended up with a mixture of artifacts attached to XvOSfX9CRKGDttkP3mkVQQ, and ended up with update verify errors [4-6] like

calling QuitProgressUI
Files source/bin/uninstall/helper.exe and target/bin/uninstall/helper.exe differ
FAIL: binary files found in diff
FAIL: [57.0 en-ZA complete] check_updates returned failure for WINNT_x86-msvc-x86 downloads/Firefox Setup 57.0b1.exe vs. downloads/Firefox Setup 57.0b5.exe: 1

helper.exe is localised, so it is signed in each l10n repack run. I think we ended up with the complete.mar and installer.exe from different runs for en-ZA and eo (South African English and Esperanto).


[1] https://tools.taskcluster.net/groups/aoaObs6aROaQ4Y9T3Tlj-A/tasks/JHvTQlOiT9q3EJZLU0wWcQ/details
[2] http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-mozilla-beta_devedition_win32_l10n_repack/builds/15
[3] http://buildbot-master77.bb.releng.use1.mozilla.com:8001/builders/release-mozilla-beta_devedition_win32_l10n_repack/builds/19
[4] https://archive.mozilla.org/pub/devedition/tinderbox-builds/mozilla-beta-win32/release-mozilla-beta_devedition_win32_update_verify-bm71-build1-build26.txt.gz
[5] https://archive.mozilla.org/pub/devedition/tinderbox-builds/mozilla-beta-win32/release-mozilla-beta_devedition_win32_update_verify-bm70-build1-build16.txt.gz
[6] https://archive.mozilla.org/pub/devedition/tinderbox-builds/mozilla-beta-win32/release-mozilla-beta_devedition_win32_update_verify-bm73-build1-build166.txt.gz

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Nothing much to see at https://papertrailapp.com/groups/2819754/events?q=JHvTQlOiT9q3EJZLU0wWcQ

The buildbot start times are only one second apart, so perhaps we hit a race claiming the buildrequest in the buildbot db. I don't recall this being a problem before but it's the sort of thing that could happen at the start of any job.

Comment 2

[Jordan Lund (:jlund)]

ftr - I removed en-ZA and eo locales under win32 in release blob: https://aus4-admin.mozilla.org/releases/Devedition-57.0b5-build1

Comment 3

[Axel Hecht [:Pike]]

Say this was a chemspill and windows german, what options would we have?

Comment 4

[Jordan Lund (:jlund)]

(In reply to Axel Hecht [:Pike] from comment #3)
> Say this was a chemspill and windows german, what options would we have?

We would be forced into delaying the release for every platform and locale and do a build2.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

We shipped 57.0b5. I can't quite predict what's going to happen with subsequent betas, but if we get a lot of update verify failures we could consider a patch like this:

diff --git a/release/patcher-configs/mozDevedition-branch-patcher2.cfg b/release/patcher-configs/mozDevedition-branch-patcher2.cfg
--- a/release/patcher-configs/mozDevedition-branch-patcher2.cfg
+++ b/release/patcher-configs/mozDevedition-branch-patcher2.cfg
@@ -331,12 +331,14 @@
             <57.0b5>
                 checksumsurl   http://archive.mozilla.org/pub/devedition/candidates/57.0b5-candidates/build1/%platform%/%locale%/firefox-57.0b5.checksums
                 completemarurl   http://archive.mozilla.org/pub/devedition/candidates/57.0b5-candidates/build1/update/%platform%/%locale%/firefox-57.0b5.complete.mar
                 <exceptions>
                     ja   linux-i686, linux-x86_64, win32, win64
                     ja-JP-mac   mac
+                    en-ZA  win32
+                    eo   win32
                 </exceptions>
                 extension-version   57.0
                 locales   ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ja-JP-mac ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW
                 <platforms>
                     linux-i686   20171002181526
                     linux-x86_64   20171002181526

Comment 6

[Johan Lorenzo [:jlorenzo]]

Even though the patch in comment 5 landed[1], we still got failures[2]. That makes me wonder if we shouldn't have:

> en-ZA  mac, linux-i686, linux-x86_64, win64, 
> eo   mac, linux-i686, linux-x86_64, win64, 

instead. If I understand the the japanese lines correctly, ja-JP-mac is for mac. Then, "en-ZA  win32" may mean: this locale only exist for win32. Reopening the bug. 

[1] https://hg.mozilla.org/build/tools/rev/b1437b16de6b
[2] https://archive.mozilla.org/pub/devedition/tinderbox-builds/mozilla-beta-win32/release-mozilla-beta_devedition_win32_update_verify-bm91-build1-build186.txt.gz