1153242 - update verify fails for first few beta releases after RC

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[[e-mail] (:bhearsum)]

We got errors like:
PATCH DIRECTORY ../../update
INSTALLATION DIRECTORY .
WORKING DIRECTORY .
failed: 22
calling QuitProgressUI
FAIL: update status was not succeeded: failed: 22

Which stands for mar channel mismatch, according to https://github.com/mozilla/gecko-dev/blob/master/toolkit/mozapps/update/common/errors.h.

This is probably fallout from the balrog rc submission work, I'll look into this soon.

Comment 1

[[e-mail] (:bhearsum)]

Turns out this has happened for 38.0b2 as well. Curiously, I can't find record of it happening for 38.0b1 even though it should have...

In any case, the problem here is that we need mar_channel_IDs set for the RC build lines of the update verify configs, so that ACCEPTED_MAR_CHANNEL_IDS gets adjusted before the update gets applied.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #1)
> Turns out this has happened for 38.0b2 as well. Curiously, I can't find
> record of it happening for 38.0b1 even though it should have...

Looks like we don't run RC -> beta 1 update verify checks. This is the patcher config from 38.0b1: https://github.com/mozilla/build-tools/blob/2856d3af42babfd5f1237ee35f9ec57afeb313f6/release/patcher-configs/mozBeta-branch-patcher2.cfg, and we only run update verify checks for the first version in each past-update line, and the "from" version of the current-update section. 37.0 doesn't end up in either for 38.0b1. This might be related to bug 1151755...

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Yes indeed, we missed checks for 38.0b1 because of that bug.

IIRC we need to make sure a line
   mar-channel-ids   firefox-mozilla-beta,firefox-mozilla-release
is added to any RC block in the beta patcher config, preferably when we bump it for that RC. eg http://hg.mozilla.org/build/tools/file/54d6610ac462/release/patcher-configs/mozBeta-branch-patcher2.cfg#l1196

Then that'll get passed down to update verify, and it will fix up the update-settings.ini file between unpacking an RC and testing the update.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: add code to patcher config bump that can create mar channel ids

Needs more testing, but seems to work for the one, somewhat contrived, scenario I tested:
perl ../patcher-config-bump.pl -p firefox -r Firefox -v 37.0 -a 37.0 -o 37.0b7 -b 1 -c mozBeta-branch-patcher2.cfg -t stage.mozilla.org -f ftp.mozilla.org -d download.mozilla.org -l shipped-locales --partial-version 37.0b7 --platform linux --platform linux64 --platform macosx64 --platform win32 --mar-channel-id firefox-mozilla-beta --mar-channel-id firefox-mozilla-release
Error running wget: Modification of a read-only value attempted at /usr/lib/x86_64-linux-gnu/perl/5.20/IO/Handle.pm line 207.

Error running wget: Modification of a read-only value attempted at /usr/lib/x86_64-linux-gnu/perl/5.20/IO/Handle.pm line 207.

Error running wget: Modification of a read-only value attempted at /usr/lib/x86_64-linux-gnu/perl/5.20/IO/Handle.pm line 207.

Error running wget: Modification of a read-only value attempted at /usr/lib/x86_64-linux-gnu/perl/5.20/IO/Handle.pm line 207.

@@ -1523,29 +1512,30 @@
                     mac   20150212154903
                     win32   20150212154903
                 </platforms>
                 prettyVersion   36.0 Beta 9
                 schema   2
                 version   36.0
             </36.0b9>
             <37.0>
-                checksumsurl   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/37.0-candidates/build2/%platform%/%locale%/firefox-37.0.checksums
-                completemarurl   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/37.0-candidates/build2/update/%platform%/%locale%/firefox-37.0.complete.mar
+                checksumsurl   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/37.0-candidates/build1/%platform%/%locale%/firefox-37.0.checksums
+                completemarurl   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/37.0-candidates/build1/update/%platform%/%locale%/firefox-37.0.complete.mar
                 <exceptions>
                     ja   linux-i686, linux-x86_64, win32
                     ja-JP-mac   mac
                 </exceptions>
                 extension-version   37.0
                 locales   ach af an ar as ast az be bg bn-BD bn-IN br bs ca cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gu-IN he hi-IN hr hsb hu hy-AM id is it ja ja-JP-mac kk km kn ko lij lt lv mai mk ml mr ms nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk uz vi xh zh-CN zh-TW
+                mar-channel-ids   firefox-mozilla-beta,firefox-mozilla-release
                 <platforms>
-                    linux-i686   20150326190726
-                    linux-x86_64   20150326190726
-                    mac   20150326190726
-                    win32   20150326190726
+                    linux-i686   20150324194038
+                    linux-x86_64   20150324194038
+                    mac   20150324194038
+                    win32   20150324194038
                 </platforms>
                 prettyVersion   37.0
                 schema   2
                 version   37.0
             </37.0>
             <37.0b1>
                 checksumsurl   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/37.0b1-candidates/build1/%platform%/%locale%/firefox-37.0b1.checksums
                 completemarurl   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/37.0b1-candidates/build1/update/%platform%/%locale%/firefox-37.0b1.complete.mar


(I redacted the current-update parts of the diff).

I'm pretty sure that wget warnings above are some weird issue on my laptop, going to dig into that more though.


I was thinking of hooking this into the automation by adding a "updateVerifyMarChannelIds" (or something) option to the release config and only setting it for the beta channel on mozilla-release. When set, ReleaseUpdatesFactory will pass it along to patcher-config-bump.pl. Leaving it unset for release on mozilla-release and beta on mozilla-beta means they will inherit whatever is already set - which is what we want, I think.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] FIx up 37.0

One-time fix for <37.0> block.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8591674 [details] [diff] [review]
add code to patcher config bump that can create mar channel ids

Review of attachment 8591674 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine to me. A cleaner diff would be 'actual 37.0 build1 config' vs '37.0b7 config bumped with new code', should just be one line difference.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 8592435 [details] [diff] [review]
[tools] FIx up 37.0

Review of attachment 8592435 [details] [diff] [review]:
-----------------------------------------------------------------

Oops, thanks.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8592435 [details] [diff] [review]
[tools] FIx up 37.0

https://hg.mozilla.org/build/tools/rev/c3ae84c332de

Comment 9

[[e-mail] (:bhearsum)]

Attachment: add support for passing mar channel ids down

This should do the trick. I haven't tested this because I didn't think it was worth the effort of getting a staging release environment set-up. If you think this is too risky to land without it, I can do that though.

Passes checkconfig.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: add mar channel ids to beta channel of release configs

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 8593424 [details] [diff] [review]
add mar channel ids to beta channel of release configs

This is all landed now. Going to leave this open until it's verified with 39.0b1/b2.

Comment 12

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/a4454c7a60b3

Comment 13

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/b63222955c94

Comment 14

[[e-mail] (:bhearsum)]

39.0b1 didn't get partials from any 39.0 RC, so I couldn't verify this as part of that. Hopefully 39.0b2 will.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Bug 1163465 is biting us, we have to leave the RC builds out of the partial list.

Comment 16

[[e-mail] (:bhearsum)]

We didn't end up doing any Betas this cycle which had partials from an RC, so we'll have to until 40.0b1 to verify this.

Comment 17

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #15)
> Bug 1163465 is biting us, we have to leave the RC builds out of the partial
> list.

Whoops, forgot to read before replying. Thanks!

Comment 18

[[e-mail] (:bhearsum)]

Still haven't had any betas with RCs as their partials. It's more likely after 41.0 ships, since we'll (surely) have shipped 64-bit Windows with that, and can WONTFIX bug 1163465.

Comment 19

[[e-mail] (:bhearsum)]

Unassigning from myself because surely this will be verified by the time I get back, since we're shipping win64 w/ 42.0?

Comment 20

[Rail Aliiev [:rail]]

Haven't seen this for a while