We had a lot of windows jobs retry for deved 57.0b5 because high prices in the AWS spot market caused instances to be terminated. This lead to 'devedition mozilla-beta win32 l10n repack 3/10' task  running twice, almost simultaneously [2,3]. Most probably we ended up with a mixture of artifacts attached to XvOSfX9CRKGDttkP3mkVQQ, and ended up with update verify errors [4-6] like calling QuitProgressUI Files source/bin/uninstall/helper.exe and target/bin/uninstall/helper.exe differ FAIL: binary files found in diff FAIL: [57.0 en-ZA complete] check_updates returned failure for WINNT_x86-msvc-x86 downloads/Firefox Setup 57.0b1.exe vs. downloads/Firefox Setup 57.0b5.exe: 1 helper.exe is localised, so it is signed in each l10n repack run. I think we ended up with the complete.mar and installer.exe from different runs for en-ZA and eo (South African English and Esperanto).  https://tools.taskcluster.net/groups/aoaObs6aROaQ4Y9T3Tlj-A/tasks/JHvTQlOiT9q3EJZLU0wWcQ/details  http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-mozilla-beta_devedition_win32_l10n_repack/builds/15  http://buildbot-master77.bb.releng.use1.mozilla.com:8001/builders/release-mozilla-beta_devedition_win32_l10n_repack/builds/19  https://archive.mozilla.org/pub/devedition/tinderbox-builds/mozilla-beta-win32/release-mozilla-beta_devedition_win32_update_verify-bm71-build1-build26.txt.gz  https://archive.mozilla.org/pub/devedition/tinderbox-builds/mozilla-beta-win32/release-mozilla-beta_devedition_win32_update_verify-bm70-build1-build16.txt.gz  https://archive.mozilla.org/pub/devedition/tinderbox-builds/mozilla-beta-win32/release-mozilla-beta_devedition_win32_update_verify-bm73-build1-build166.txt.gz
Nothing much to see at https://papertrailapp.com/groups/2819754/events?q=JHvTQlOiT9q3EJZLU0wWcQ The buildbot start times are only one second apart, so perhaps we hit a race claiming the buildrequest in the buildbot db. I don't recall this being a problem before but it's the sort of thing that could happen at the start of any job.
ftr - I removed en-ZA and eo locales under win32 in release blob: https://aus4-admin.mozilla.org/releases/Devedition-57.0b5-build1
Say this was a chemspill and windows german, what options would we have?
(In reply to Axel Hecht [:Pike] from comment #3) > Say this was a chemspill and windows german, what options would we have? We would be forced into delaying the release for every platform and locale and do a build2.
We shipped 57.0b5. I can't quite predict what's going to happen with subsequent betas, but if we get a lot of update verify failures we could consider a patch like this: diff --git a/release/patcher-configs/mozDevedition-branch-patcher2.cfg b/release/patcher-configs/mozDevedition-branch-patcher2.cfg --- a/release/patcher-configs/mozDevedition-branch-patcher2.cfg +++ b/release/patcher-configs/mozDevedition-branch-patcher2.cfg @@ -331,12 +331,14 @@ <57.0b5> checksumsurl http://archive.mozilla.org/pub/devedition/candidates/57.0b5-candidates/build1/%platform%/%locale%/firefox-57.0b5.checksums completemarurl http://archive.mozilla.org/pub/devedition/candidates/57.0b5-candidates/build1/update/%platform%/%locale%/firefox-57.0b5.complete.mar <exceptions> ja linux-i686, linux-x86_64, win32, win64 ja-JP-mac mac + en-ZA win32 + eo win32 </exceptions> extension-version 57.0 locales ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ja-JP-mac ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW <platforms> linux-i686 20171002181526 linux-x86_64 20171002181526
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Even though the patch in comment 5 landed, we still got failures. That makes me wonder if we shouldn't have: > en-ZA mac, linux-i686, linux-x86_64, win64, > eo mac, linux-i686, linux-x86_64, win64, instead. If I understand the the japanese lines correctly, ja-JP-mac is for mac. Then, "en-ZA win32" may mean: this locale only exist for win32. Reopening the bug.  https://hg.mozilla.org/build/tools/rev/b1437b16de6b  https://archive.mozilla.org/pub/devedition/tinderbox-builds/mozilla-beta-win32/release-mozilla-beta_devedition_win32_update_verify-bm91-build1-build186.txt.gz
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago → 2 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1406403
You need to log in before you can comment on or make changes to this bug.