1391718 - Add LZMA/SHA384 watershed when 56.0 ships

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

Not sure how this would look like, but when 56.0 ships, we'll need to deal with the LZMA/Sha384 for the release as well.

So far that implies:
1. watersheds in Balrog. Let's make sure we test this as good as we can on release-cdntest
2. nuking some patcher-configs? See [1][2][3] and bug 1391297 for that.
3. Double check we use the mar_sha384 in releasetasks. See bug 1391680 for more details.

This is the bare minimum, could be other things as well.

[1]: https://hg.mozilla.org/build/tools/rev/85763394a956
[2]: https://hg.mozilla.org/build/tools/rev/82950f5cdae8
[3]: https://hg.mozilla.org/build/tools/file/tip/release/patcher-configs/mozRelease-branch-patcher2.cfg

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Might be helpful for later:

@catlee> I'm to create a bug about the lzma repacks?
20:35:46 <mtabara|afk> catlee: I think so - I added an action item in the post mortem for you with details
20:40:07 <mtabara|afk> catlee: so let me get this straight: 1. we backout lzma stuff. 2. we build RC with old format but with updater changed for both formats (like 56.0b3 is). 3, we don't ship it to beta channel since they only know the new format 4. we release 56. 5. we reland stuff for 57.0b1 and have beta users skip 56.RC and update from last 56.bX directly to 57.0b1. 6. in
20:40:07 <mtabara|afk> november 57.0RC only knows new format and release that. At that point we need a watershed for 56.0. Is this accurate at all?
20:41:50 <mtabara|afk> as in: use 56.0 as common ground between releases the same way 56.0b3 is. with the caveat we can't offer 56.0 to beta population during RC week
20:43:10 <@catlee> 1. yes 2. yes (except the updater only ever supports 1 format - in this case the 56.0 updater will only accept new format mars). 3. yes - and we want to repack the old format mars into the new format for beta channel. 4. yes 5. beta users will be getting RC as part of step 3 repacks
20:43:34 <@catlee> 56.0.1 will be fun
20:43:55 <@catlee> I guess we force a watershed on 56.0
20:44:15 <@catlee> we could do without the watershed if we wanted though
20:44:28 <@catlee> and continue generating both formats

Comment 2

[Kim Moir [:kmoir] ET]

Nick did this - thanks Nick!

Comment 3

[Kim Moir [:kmoir] ET]

Actually, he updated the release blob, we need to add a watershed too I think that is separate from the default rule

Comment 4

[Kim Moir [:kmoir] ET]

Attachment: Screen Shot 2017-10-02 at 3.07.09 PM.png

I added this rule on release-cdntest and tested an upgrade

I set a priority of 97 which is higher than the default rule on release (90), the 56.0 WNP (95) and the special WNP pages for the untranslated locales (96)

I looked at the patcher configs but I'm not sure what/if anything needs to be changed.  

If this rule looks sane some tests could be added to the QE test plan for 56.0.1 to ensure it works as expected, as well as adding the rule on the release channel

Comment 5

[Kim Moir [:kmoir] ET]

See

https://bugzilla.mozilla.org/show_bug.cgi?id=1397339#c9

and

https://bugzilla.mozilla.org/show_bug.cgi?id=1397339#c10

I'm not sure if this watershed should have 
* show WNP for 56.0* --> 56.0.y with mobile promotion

so I've asked for clarification on bug 1397339

Comment 6

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1391718 - Set 56.0 as general watershed for update verification tests

Review commit: https://reviewboard.mozilla.org/r/186736/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/186736/

Comment 7

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8914476 [details]
Screen Shot 2017-10-02 at 3.07.09 PM.png

Was discussed on IRC and resulted to comment 5. For reference, we also need to exclude some WNP locales. :flod gave the list on IRC.

Comment 8

[Johan Lorenzo [:jlorenzo]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #6)
I think we may get rid of versions <56.0 in UV configs, after we shipped 56.0.1. In fact, once this watershed is fully set up, UV tests will keep running against the same data. What do you all think?

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8915520 [details]
Bug 1391718 - Set 56.0 as general watershed for update verification tests

https://reviewboard.mozilla.org/r/186736/#review191830

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

A couple of comments

1, those update verify configs are generated from the patcher config each time we do a build, so the changes will be removed on the next release. IIRC we don't have a way storing watersheds in the patcher config, so there are a couple of options
  a, stop testing updates prior to the watershed (our usual approach), by removing past-update lines from mozRelease-branch-patcher2.cfg. We lose test coverage this way, the risks are bouncer stops working or the balrog rules are busted for older releases
  b, fork the patcher config, eg mozRelease-branch-patcher2-watersheds.cfg from mozRelease-branch-patcher2-watersheds.cfg at rev from 56.0 build6. The updates builder generates a separate set of u.v. configs, which we have to schedule to run

2, The lzma watershed on release-cdntest (rule 655) essentially undoes the WNP rules while targeting the same builds. Can we just make the WNP rules the lzma watershed, perhaps by adjusting the comments to make sure we don't remove it by mistake later ?

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

It sounds like we are heading towards using bz2 updates on release users with < 56.0, via bug 1395697, which could make this a WONTFIX.

Comment 12

[Kim Moir [:kmoir] ET]

closing.