Generate lzma and bz2 compressed mar files as a part of release automation

RESOLVED FIXED

Status

enhancement
RESOLVED FIXED
2 years ago
Last year

People

(Reporter: rail, Assigned: rail)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox56 fixed)

Details

Attachments

(12 attachments, 1 obsolete attachment)

10.53 KB, patch
Details | Diff | Splinter Review
59 bytes, text/x-review-board-request
aki
: review+
Details
59 bytes, text/x-review-board-request
catlee
: review+
Details
3.50 KB, patch
catlee
: review+
Details | Diff | Splinter Review
4.94 KB, patch
catlee
: review+
Details | Diff | Splinter Review
55 bytes, patch
catlee
: review+
Details | Diff | Splinter Review
4.70 KB, patch
catlee
: review+
Details | Diff | Splinter Review
1.30 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
59 bytes, text/x-review-board-request
nthomas
: review+
Details
2.08 KB, patch
sfraser
: review+
Details | Diff | Splinter Review
55 bytes, text/x-github-pull-request
rail
: review+
Details | Review
2.21 KB, patch
jlund
: review+
Details | Diff | Splinter Review
Assignee

Description

2 years ago
We want to generate both LZMA and BZ2 compressed complete mars to avoid unnecessary watersheds. These mars should be signed with the old signmar binary (sha1, not sha384). The metadata should be submitted to a separate Balrog blob ($regular_blob_name-bz2?). The latter will be done by funsize.
Assignee

Updated

2 years ago
Assignee: nobody → rail
Assignee

Comment 1

2 years ago
One of the options would be:

Extend releasetasks to generate more tasks:

* create a script similar to https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/funsize-update-generator/scripts/funsize.py, which will be using https://dxr.mozilla.org/mozilla-central/source/tools/update-packaging/change_mar_compression.pl:

  ** convert complete MARs to BZ2 compressed ones
  ** generate BZ2 compressed partials

* the MARs should be signed using the old signing method ("mar", not "mar_sha384")

* modify https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/funsize-balrog-submitter/scripts/funsize-balrog-submitter.py so we can specify a different release blob as a part of submission (maybe modify the --dummy option to accept a suffix)

* Modify https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/release/updates.py to submit the top level info to a different blob. Also it shouldn't bump/tag the config, because we don't have any good way to run update verification.

* We shouldn't generate BZ2-compressed partials nor serve completes for versions that do not support BZ2 compression.


Another option would be to create a tool (similar to https://github.com/mozilla/releasewarrior/blob/master/how-tos/manually-generate-partials.md) to do the same steps, but without integrating into release automation and run it manually.

Both cases require manual rule tweaking every time we generate special updates.
Assignee

Comment 2

2 years ago
catlee's script from bug 1393789 made me think that we can simplify the required modifications if we recompress and sign the completes as a part of release automation. In this case we don't need to modify funsize.py and signingworker logic to handle completes specially.

The idea is to add more tasks:

1) repackage the completes
2) sign the completes
3) beetmove them

This patch adds a new script to handle repackaging. The rest is done via releasetasks: https://github.com/mozilla-releng/releasetasks/compare/master...rail:repackage_completes?expand=1

For partials we would need to:

1) No need to change funsize.py. It detects the "to" compression format and generates proper partials
2) no need to modify signing. We still should make sure that we specify "mar", not "mar-sha384" as the signing format.
3) publish to balrog should publish to a separate release blob (TBD)
4) updates (top level data) should be published to the release blob used in 3)
5) update verification is still to decided. As a possible scenario, we can generate required update verify configs without patcher configs somehow...
Comment hidden (mozreview-request)

Comment 4

2 years ago
mozreview-review
Comment on attachment 8908830 [details]
Bug 1395697 - Update jamun release configs to match mozilla-beta

https://reviewboard.mozilla.org/r/180450/#review185628
Attachment #8908830 - Flags: review?(aki) → review+
Assignee

Comment 7

2 years ago
Attachment #8908961 - Flags: review?(catlee)
Assignee

Comment 8

2 years ago
Attachment #8908962 - Flags: review?(catlee)
Assignee

Updated

2 years ago
Keywords: leave-open

Comment 10

2 years ago
mozreview-review
Comment on attachment 8908960 [details]
Bug 1395697 - Do not generate bouncer entries for BZ2-compressed MAR files for beta and devedition.

https://reviewboard.mozilla.org/r/180568/#review185726
Attachment #8908960 - Flags: review?(catlee) → review+
Comment hidden (mozreview-request)
Attachment #8908961 - Flags: review?(catlee) → review+
Attachment #8908962 - Flags: review?(catlee) → review+
Attachment #8908964 - Attachment is patch: true
Attachment #8908964 - Attachment mime type: text/x-github-pull-request → text/plain
Attachment #8908964 - Flags: review?(catlee) → review+
Assignee

Comment 12

2 years ago
I also added https://reviewboard.mozilla.org/r/180568/diff/1-2/ to add bouncer entries automatically.
Assignee

Comment 13

2 years ago
I can land these later today
\o/

thanks rail!
Comment on attachment 8908961 [details] [diff] [review]
buildbot-configs-lzma_to_bz2.diff

Review of attachment 8908961 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ +2447,5 @@
>      },
>      # TODO: fennec
>  }
> +# Recompress complete MARs from LZMA to BZ2 for versions >= 56.0
> +BRANCHES['mozilla-beta']['lzma_to_bz2'] = True

It's probably not worth doing this for Beta.

We already have a watershed in place for the LZMA updates, and another one for the win32->win64 migration.

Comment 18

2 years ago
Pushed by raliiev@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a7aee415666d
Add scripts to handle MAR recompression in release automation r=catlee
Assignee

Updated

2 years ago
Attachment #8908964 - Flags: checked-in+
The changes to testing/mozharness/configs/releases/bouncer_firefox_{beta,devedition,esr}.py are an issue with uptake monitoring, as it checks all products in this file. This bit us on deved 57.0b1 build3 and I'm going to drop a 0 byte key to work around it. 

We should either backout the bouncer changes where lzma_to_bz2 is False, or figure out some way to teach the bouncer script about that.
Comment hidden (mozreview-request)
Assignee

Comment 23

2 years ago
Attachment #8909266 - Flags: review?(catlee)
Attachment #8909266 - Flags: review?(catlee) → review+
Assignee

Comment 25

2 years ago
Ryan, I need this change to be on beta for 56+. What landing strategy would you suggest?
Flags: needinfo?(ryanvm)
I have other uplifts to land for 56 before the RC build. Given that this grafts cleanly, I can get it where it needs to go from here :)
Flags: needinfo?(ryanvm)
Assignee

Comment 29

2 years ago
Posted patch typo.diffSplinter Review
Attachment #8909553 - Flags: review?(nthomas)
Attachment #8909553 - Flags: review?(nthomas) → review+
Comment hidden (mozreview-request)
Deactivating partial updates was necessary because of [1]: generated partial mars were lzma instead of bz2. I manually updated the balrog blob, but then new failures came up[2]: no partials updates were found. The latest patch removes partials from expected updates paths. Once it's merged, I'll rerun the failed jobs.


[1] https://tools.taskcluster.net/groups/aUpBDSw5S2etpFHqxKSCYA/tasks/ZdiZxLNMRweZ1C8ZWYPDCw/runs/0/logs/public%2Flogs%2Flive.log
[2] https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-win32/release-mozilla-release_firefox_win32_update_verify-bm91-build1-build215.txt.gz
Comment hidden (mozreview-request)

Comment 35

2 years ago
mozreview-review
Comment on attachment 8909668 [details]
Bug 1395697 - Deactivate partial updates on non-LZMA releases

https://reviewboard.mozilla.org/r/181188/#review186348

r+ in order to test the completes are all working. FWIW I think we'll want to fix partials before we ship 56.0 on release.
Attachment #8909668 - Flags: review+
Comment hidden (mozreview-request)
Assignee

Comment 37

2 years ago
Hmm, reading https://hg.mozilla.org/releases/mozilla-release/file/tip/taskcluster/docker/funsize-update-generator/scripts/funsize.py#l245

            if mar_type == 'to' and not is_lzma_compressed_mar(dest):
                use_old_format = True

Sounds like we use the "to" compression format to decide wether we want lzma or bz2. Shouldn't it be "from"?
Flags: needinfo?(sfraser)
Assignee

Comment 38

2 years ago
Posted patch from.diff (obsolete) — Splinter Review
Probaly we need this.
Attachment #8909695 - Flags: review?(sfraser)
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #37)
> Hmm, reading
> https://hg.mozilla.org/releases/mozilla-release/file/tip/taskcluster/docker/
> funsize-update-generator/scripts/funsize.py#l245
> 
>             if mar_type == 'to' and not is_lzma_compressed_mar(dest):
>                 use_old_format = True
> 
> Sounds like we use the "to" compression format to decide wether we want lzma
> or bz2. Shouldn't it be "from"?

This was done for nightly - Just to make sure, let's work it through. 

There's a watershed, so watershed-N will all have updaters that need bz2, but will only ever get sent data to take them to the watershed. 

The watershed itself has to be bz2, but the updater in it only supports lzma. So anything after the watershed needs lzma.

From              To                UpdaterNeeds  Checking 'to'  Checking 'from'
Watershed-2(bz2)  Watershed-1(bz2)  bz2           bz2            bz2
Watershed-1(bz2)  Watershed(bz2)    bz2           bz2            bz2
Watershed(bz2)    Watershed+1(lzma) lzma          lzma           bz2
Watershed+1(lzma) Watershed+2(lzma) lzma          lzma           lzma

So this worked for nightly - but presumably you're having issues because there isn't a similar watershed now? 

I wonder if we could modify this just because we've built all the relevant nightly partials now.
Flags: needinfo?(sfraser)
Assignee

Comment 40

2 years ago
Comment on attachment 8909695 [details] [diff] [review]
from.diff

<sfraser> rail-away: have responded. Short version is the code is correct as-is for nightly, am unsure if what you're doing with 54 is in the same situation, or will have the same updater change
<•rail-away> I see what you mean, having the MAR bz2-comressed tells you nothing about the updater... hmmm
<•rail-away> right... we can't use the patch as is
Attachment #8909695 - Flags: review?(sfraser)
Assignee

Comment 41

2 years ago
Maybe something like this?
Attachment #8909695 - Attachment is obsolete: true
Attachment #8909707 - Flags: review?(sfraser)
Depends on: 1401176
Assignee

Comment 42

2 years ago
Apparently we use wrong signature format for partials that suppose to use BZ2 compression and thus SHA1 ("mar") as the signature format.
Attachment #8909833 - Flags: review?(jlorenzo)
Comment on attachment 8909707 [details] [diff] [review]
from_less_than_56.diff

I think that should work - basing it on what the updater will understand given from the version. Go for it
Attachment #8909707 - Flags: review?(sfraser) → review+

Comment 44

2 years ago
Pushed by raliiev@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6c0e0f992f33
Use BZ2 MAR compression for versions less than 56.0 r=sfraser
Assignee

Comment 46

2 years ago
Comment on attachment 8909833 [details] [review]
Use proper signing format for partials

r+ed in the PR.
Attachment #8909833 - Flags: review?(jlorenzo)
Attachment #8909833 - Flags: review+
Attachment #8909833 - Flags: checked-in+
Assignee

Updated

2 years ago
Depends on: 1402011
Assignee

Updated

2 years ago
Depends on: 1402015
Assignee

Comment 48

2 years ago
in 56.0.2 we had no partials for versions less than 56.0 so we ended up not passing --partial-update to this script.
Attachment #8921607 - Flags: review?
Assignee

Updated

2 years ago
Attachment #8921607 - Flags: review? → review?(jlund)
Attachment #8921607 - Flags: review?(jlund) → review+
Assignee

Updated

2 years ago
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.