1395697 - Generate lzma and bz2 compressed mar files as a part of release automation

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Description

[Rail Aliiev [:rail]]

We want to generate both LZMA and BZ2 compressed complete mars to avoid unnecessary watersheds. These mars should be signed with the old signmar binary (sha1, not sha384). The metadata should be submitted to a separate Balrog blob ($regular_blob_name-bz2?). The latter will be done by funsize.

Comment 1

[Rail Aliiev [:rail]]

One of the options would be:

Extend releasetasks to generate more tasks:

* create a script similar to https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/funsize-update-generator/scripts/funsize.py, which will be using https://dxr.mozilla.org/mozilla-central/source/tools/update-packaging/change_mar_compression.pl:

  ** convert complete MARs to BZ2 compressed ones
  ** generate BZ2 compressed partials

* the MARs should be signed using the old signing method ("mar", not "mar_sha384")

* modify https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/funsize-balrog-submitter/scripts/funsize-balrog-submitter.py so we can specify a different release blob as a part of submission (maybe modify the --dummy option to accept a suffix)

* Modify https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/release/updates.py to submit the top level info to a different blob. Also it shouldn't bump/tag the config, because we don't have any good way to run update verification.

* We shouldn't generate BZ2-compressed partials nor serve completes for versions that do not support BZ2 compression.


Another option would be to create a tool (similar to https://github.com/mozilla/releasewarrior/blob/master/how-tos/manually-generate-partials.md) to do the same steps, but without integrating into release automation and run it manually.

Both cases require manual rule tweaking every time we generate special updates.

Comment 2

[Rail Aliiev [:rail]]

Attachment: recompress_completes.diff

catlee's script from bug 1393789 made me think that we can simplify the required modifications if we recompress and sign the completes as a part of release automation. In this case we don't need to modify funsize.py and signingworker logic to handle completes specially.

The idea is to add more tasks:

1) repackage the completes
2) sign the completes
3) beetmove them

This patch adds a new script to handle repackaging. The rest is done via releasetasks: https://github.com/mozilla-releng/releasetasks/compare/master...rail:repackage_completes?expand=1

For partials we would need to:

1) No need to change funsize.py. It detects the "to" compression format and generates proper partials
2) no need to modify signing. We still should make sure that we specify "mar", not "mar-sha384" as the signing format.
3) publish to balrog should publish to a separate release blob (TBD)
4) updates (top level data) should be published to the release blob used in 3)
5) update verification is still to decided. As a possible scenario, we can generate required update verify configs without patcher configs somehow...

Comment 3

[Rail Aliiev [:rail]]

Attachment: Bug 1395697 - Update jamun release configs to match mozilla-beta

Review commit: https://reviewboard.mozilla.org/r/180450/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/180450/

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 8908830 [details]
Bug 1395697 - Update jamun release configs to match mozilla-beta

https://reviewboard.mozilla.org/r/180450/#review185628

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8908830 [details]
Bug 1395697 - Update jamun release configs to match mozilla-beta

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/1f81643788a2c7ee656db6bb33256590c249d0f1
remote:   https://hg.mozilla.org/build/buildbot-configs/rev/e695a86201b4b233ce87e560386e615e0976076e

Comment 6

[Rail Aliiev [:rail]]

Attachment: Bug 1395697 - Do not generate bouncer entries for BZ2-compressed MAR files for beta and devedition.

Review commit: https://reviewboard.mozilla.org/r/180568/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/180568/

Comment 7

[Rail Aliiev [:rail]]

Attachment: buildbot-configs-lzma_to_bz2.diff

Comment 8

[Rail Aliiev [:rail]]

Attachment: tools-lzma_to_bz2.diff

Comment 9

[Rail Aliiev [:rail]]

Attachment: PR for releasetask

This worked as expected in staging. There are some directories with recompressed complete MARs:

http://ftp.stage.mozaws.net/pub/firefox/candidates/56.0b9-candidates/build30/update/win64/sv-SE/

http://ftp.stage.mozaws.net/pub/firefox/candidates/56.0b9-candidates/build30/update/linux-x86_64/en-US/

Comment 10

[Chris AtLee [:catlee]]

Comment on attachment 8908960 [details]
Bug 1395697 - Do not generate bouncer entries for BZ2-compressed MAR files for beta and devedition.

https://reviewboard.mozilla.org/r/180568/#review185726

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 8908960 [details]
Bug 1395697 - Do not generate bouncer entries for BZ2-compressed MAR files for beta and devedition.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/180568/diff/1-2/

Comment 12

[Rail Aliiev [:rail]]

I also added https://reviewboard.mozilla.org/r/180568/diff/1-2/ to add bouncer entries automatically.

Comment 13

[Rail Aliiev [:rail]]

I can land these later today

Comment 14

[Chris AtLee [:catlee]]

\o/

thanks rail!

Comment 15

[Chris AtLee [:catlee]]

Comment on attachment 8908961 [details] [diff] [review]
buildbot-configs-lzma_to_bz2.diff

Review of attachment 8908961 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ +2447,5 @@
>      },
>      # TODO: fennec
>  }
> +# Recompress complete MARs from LZMA to BZ2 for versions >= 56.0
> +BRANCHES['mozilla-beta']['lzma_to_bz2'] = True

It's probably not worth doing this for Beta.

We already have a watershed in place for the LZMA updates, and another one for the win32->win64 migration.

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 8908961 [details] [diff] [review]
buildbot-configs-lzma_to_bz2.diff

Disabled for beta: https://hg.mozilla.org/build/buildbot-configs/rev/9f0d50cf0cff3fe2d2a9ec1b840b375a762acf85

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 8908962 [details] [diff] [review]
tools-lzma_to_bz2.diff

https://hg.mozilla.org/build/tools/rev/6798845e667adbb84270a0ca7a18f6600d2b534b

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a7aee415666d
Add scripts to handle MAR recompression in release automation r=catlee

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8908960 [details]
Bug 1395697 - Do not generate bouncer entries for BZ2-compressed MAR files for beta and devedition.

Inbound: https://hg.mozilla.org/integration/mozilla-inbound/rev/a7aee415666d33258894a3c48f9435978f7c8eef

Beta: https://hg.mozilla.org/releases/mozilla-beta/rev/55d0a2aa99de0d2f646cb6d2ab9026630de2d16b

Release: https://hg.mozilla.org/releases/mozilla-release/rev/7232d6879f930edf1361abe33ec5088ed9e373b7

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/a7aee415666d

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

The changes to testing/mozharness/configs/releases/bouncer_firefox_{beta,devedition,esr}.py are an issue with uptake monitoring, as it checks all products in this file. This bit us on deved 57.0b1 build3 and I'm going to drop a 0 byte key to work around it. 

We should either backout the bouncer changes where lzma_to_bz2 is False, or figure out some way to teach the bouncer script about that.

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 8908960 [details]
Bug 1395697 - Do not generate bouncer entries for BZ2-compressed MAR files for beta and devedition.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/180568/diff/2-3/

Comment 23

[Rail Aliiev [:rail]]

Attachment: disable-beta-devedition-bouncer.diff

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 8909266 [details] [diff] [review]
disable-beta-devedition-bouncer.diff

https://hg.mozilla.org/integration/mozilla-inbound/rev/7ee0accd38eefc04b056b83562685aa472300be3

Comment 25

[Rail Aliiev [:rail]]

Ryan, I need this change to be on beta for 56+. What landing strategy would you suggest?

Comment 26

[Ryan VanderMeulen [:RyanVM]]

I have other uplifts to land for 56 before the RC build. Given that this grafts cleanly, I can get it where it needs to go from here :)

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/69932f66faf2 (FIREFOX_56b13_RELBRANCH)
https://hg.mozilla.org/releases/mozilla-release/rev/73002c440952

Comment 28

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/7ee0accd38ee

Comment 29

[Rail Aliiev [:rail]]

Attachment: typo.diff

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5fb440d47ab9
Syntax fix r=nthomas

Comment 31

[Rail Aliiev [:rail]]

Comment on attachment 8909553 [details] [diff] [review]
typo.diff

https://hg.mozilla.org/integration/mozilla-inbound/rev/5fb440d47ab9ebfa9633ff88274f221924f3cc3a
https://hg.mozilla.org/releases/mozilla-release/rev/28fadaace7bd2189a6c13db6765bf04048a41789

Comment 32

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1395697 - Deactivate partial updates on non-LZMA releases

Review commit: https://reviewboard.mozilla.org/r/181188/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/181188/

Comment 33

[Johan Lorenzo [:jlorenzo]]

Deactivating partial updates was necessary because of [1]: generated partial mars were lzma instead of bz2. I manually updated the balrog blob, but then new failures came up[2]: no partials updates were found. The latest patch removes partials from expected updates paths. Once it's merged, I'll rerun the failed jobs.


[1] https://tools.taskcluster.net/groups/aUpBDSw5S2etpFHqxKSCYA/tasks/ZdiZxLNMRweZ1C8ZWYPDCw/runs/0/logs/public%2Flogs%2Flive.log
[2] https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-win32/release-mozilla-release_firefox_win32_update_verify-bm91-build1-build215.txt.gz

Comment 34

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8909668 [details]
Bug 1395697 - Deactivate partial updates on non-LZMA releases

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181188/diff/1-2/

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8909668 [details]
Bug 1395697 - Deactivate partial updates on non-LZMA releases

https://reviewboard.mozilla.org/r/181188/#review186348

r+ in order to test the completes are all working. FWIW I think we'll want to fix partials before we ship 56.0 on release.

Comment 36

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8909668 [details]
Bug 1395697 - Deactivate partial updates on non-LZMA releases

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181188/diff/2-3/

Comment 37

[Rail Aliiev [:rail]]

Hmm, reading https://hg.mozilla.org/releases/mozilla-release/file/tip/taskcluster/docker/funsize-update-generator/scripts/funsize.py#l245

            if mar_type == 'to' and not is_lzma_compressed_mar(dest):
                use_old_format = True

Sounds like we use the "to" compression format to decide wether we want lzma or bz2. Shouldn't it be "from"?

Comment 38

[Rail Aliiev [:rail]]

Attachment: from.diff

Probaly we need this.

Comment 39

[Simon Fraser [:sfraser] ⌚️GMT]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #37)
> Hmm, reading
> https://hg.mozilla.org/releases/mozilla-release/file/tip/taskcluster/docker/
> funsize-update-generator/scripts/funsize.py#l245
> 
>             if mar_type == 'to' and not is_lzma_compressed_mar(dest):
>                 use_old_format = True
> 
> Sounds like we use the "to" compression format to decide wether we want lzma
> or bz2. Shouldn't it be "from"?

This was done for nightly - Just to make sure, let's work it through. 

There's a watershed, so watershed-N will all have updaters that need bz2, but will only ever get sent data to take them to the watershed. 

The watershed itself has to be bz2, but the updater in it only supports lzma. So anything after the watershed needs lzma.

From              To                UpdaterNeeds  Checking 'to'  Checking 'from'
Watershed-2(bz2)  Watershed-1(bz2)  bz2           bz2            bz2
Watershed-1(bz2)  Watershed(bz2)    bz2           bz2            bz2
Watershed(bz2)    Watershed+1(lzma) lzma          lzma           bz2
Watershed+1(lzma) Watershed+2(lzma) lzma          lzma           lzma

So this worked for nightly - but presumably you're having issues because there isn't a similar watershed now? 

I wonder if we could modify this just because we've built all the relevant nightly partials now.

Comment 40

[Rail Aliiev [:rail]]

Comment on attachment 8909695 [details] [diff] [review]
from.diff

<sfraser> rail-away: have responded. Short version is the code is correct as-is for nightly, am unsure if what you're doing with 54 is in the same situation, or will have the same updater change
<•rail-away> I see what you mean, having the MAR bz2-comressed tells you nothing about the updater... hmmm
<•rail-away> right... we can't use the patch as is

Comment 41

[Rail Aliiev [:rail]]

Attachment: from_less_than_56.diff

Maybe something like this?

Comment 42

[Rail Aliiev [:rail]]

Attachment: Use proper signing format for partials

Apparently we use wrong signature format for partials that suppose to use BZ2 compression and thus SHA1 ("mar") as the signature format.

Comment 43

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8909707 [details] [diff] [review]
from_less_than_56.diff

I think that should work - basing it on what the updater will understand given from the version. Go for it

Comment 44

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6c0e0f992f33
Use BZ2 MAR compression for versions less than 56.0 r=sfraser

Comment 45

[Rail Aliiev [:rail]]

Comment on attachment 8909707 [details] [diff] [review]
from_less_than_56.diff

https://hg.mozilla.org/integration/mozilla-inbound/rev/6c0e0f992f335b37ad70f6818b3e528c106455ee

https://hg.mozilla.org/releases/mozilla-release/rev/2a8f5f48a4b43b6ba44669a8df9b398057c7880f

Comment 46

[Rail Aliiev [:rail]]

Comment on attachment 8909833 [details] [review]
Use proper signing format for partials

r+ed in the PR.

Comment 47

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/5fb440d47ab9
https://hg.mozilla.org/mozilla-central/rev/6c0e0f992f33

Comment 48

[Rail Aliiev [:rail]]

Attachment: tools-partials-not-required.diff

in 56.0.2 we had no partials for versions less than 56.0 so we ended up not passing --partial-update to this script.

Comment 49

[Rail Aliiev [:rail]]

Comment on attachment 8921607 [details] [diff] [review]
tools-partials-not-required.diff

https://hg.mozilla.org/build/tools/rev/75799ef19de8dc7ed676b826ae96540b52bfb388

Comment 50

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.