Closed Bug 1395697 Opened 3 years ago Closed 3 years ago

Generate lzma and bz2 compressed mar files as a part of release automation

Categories

(Release Engineering :: Release Automation: Other, enhancement)

enhancement
Not set
normal

Tracking

(firefox56 fixed)

RESOLVED FIXED
Tracking Status
firefox56 --- fixed

People

(Reporter: rail, Assigned: rail)

References

Details

Attachments

(12 files, 1 obsolete file)

10.53 KB, patch
Details | Diff | Splinter Review
59 bytes, text/x-review-board-request
aki
: review+
Details
59 bytes, text/x-review-board-request
catlee
: review+
Details
3.50 KB, patch
catlee
: review+
Details | Diff | Splinter Review
4.94 KB, patch
catlee
: review+
Details | Diff | Splinter Review
55 bytes, patch
catlee
: review+
Details | Diff | Splinter Review
4.70 KB, patch
catlee
: review+
Details | Diff | Splinter Review
1.30 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
59 bytes, text/x-review-board-request
nthomas
: review+
Details
2.08 KB, patch
sfraser
: review+
Details | Diff | Splinter Review
55 bytes, text/x-github-pull-request
rail
: review+
Details | Review
2.21 KB, patch
jlund
: review+
Details | Diff | Splinter Review
We want to generate both LZMA and BZ2 compressed complete mars to avoid unnecessary watersheds. These mars should be signed with the old signmar binary (sha1, not sha384). The metadata should be submitted to a separate Balrog blob ($regular_blob_name-bz2?). The latter will be done by funsize.
Assignee: nobody → rail
One of the options would be:

Extend releasetasks to generate more tasks:

* create a script similar to https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/funsize-update-generator/scripts/funsize.py, which will be using https://dxr.mozilla.org/mozilla-central/source/tools/update-packaging/change_mar_compression.pl:

  ** convert complete MARs to BZ2 compressed ones
  ** generate BZ2 compressed partials

* the MARs should be signed using the old signing method ("mar", not "mar_sha384")

* modify https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/funsize-balrog-submitter/scripts/funsize-balrog-submitter.py so we can specify a different release blob as a part of submission (maybe modify the --dummy option to accept a suffix)

* Modify https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/release/updates.py to submit the top level info to a different blob. Also it shouldn't bump/tag the config, because we don't have any good way to run update verification.

* We shouldn't generate BZ2-compressed partials nor serve completes for versions that do not support BZ2 compression.


Another option would be to create a tool (similar to https://github.com/mozilla/releasewarrior/blob/master/how-tos/manually-generate-partials.md) to do the same steps, but without integrating into release automation and run it manually.

Both cases require manual rule tweaking every time we generate special updates.
catlee's script from bug 1393789 made me think that we can simplify the required modifications if we recompress and sign the completes as a part of release automation. In this case we don't need to modify funsize.py and signingworker logic to handle completes specially.

The idea is to add more tasks:

1) repackage the completes
2) sign the completes
3) beetmove them

This patch adds a new script to handle repackaging. The rest is done via releasetasks: https://github.com/mozilla-releng/releasetasks/compare/master...rail:repackage_completes?expand=1

For partials we would need to:

1) No need to change funsize.py. It detects the "to" compression format and generates proper partials
2) no need to modify signing. We still should make sure that we specify "mar", not "mar-sha384" as the signing format.
3) publish to balrog should publish to a separate release blob (TBD)
4) updates (top level data) should be published to the release blob used in 3)
5) update verification is still to decided. As a possible scenario, we can generate required update verify configs without patcher configs somehow...
Comment on attachment 8908830 [details]
Bug 1395697 - Update jamun release configs to match mozilla-beta

https://reviewboard.mozilla.org/r/180450/#review185628
Attachment #8908830 - Flags: review?(aki) → review+
Attachment #8908961 - Flags: review?(catlee)
Attachment #8908962 - Flags: review?(catlee)
Keywords: leave-open
Comment on attachment 8908960 [details]
Bug 1395697 - Do not generate bouncer entries for BZ2-compressed MAR files for beta and devedition.

https://reviewboard.mozilla.org/r/180568/#review185726
Attachment #8908960 - Flags: review?(catlee) → review+
Attachment #8908961 - Flags: review?(catlee) → review+
Attachment #8908962 - Flags: review?(catlee) → review+
Attachment #8908964 - Attachment is patch: true
Attachment #8908964 - Attachment mime type: text/x-github-pull-request → text/plain
Attachment #8908964 - Flags: review?(catlee) → review+
I also added https://reviewboard.mozilla.org/r/180568/diff/1-2/ to add bouncer entries automatically.
I can land these later today
Comment on attachment 8908961 [details] [diff] [review]
buildbot-configs-lzma_to_bz2.diff

Review of attachment 8908961 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ +2447,5 @@
>      },
>      # TODO: fennec
>  }
> +# Recompress complete MARs from LZMA to BZ2 for versions >= 56.0
> +BRANCHES['mozilla-beta']['lzma_to_bz2'] = True

It's probably not worth doing this for Beta.

We already have a watershed in place for the LZMA updates, and another one for the win32->win64 migration.
Pushed by raliiev@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a7aee415666d
Add scripts to handle MAR recompression in release automation r=catlee
Attachment #8908964 - Flags: checked-in+
The changes to testing/mozharness/configs/releases/bouncer_firefox_{beta,devedition,esr}.py are an issue with uptake monitoring, as it checks all products in this file. This bit us on deved 57.0b1 build3 and I'm going to drop a 0 byte key to work around it. 

We should either backout the bouncer changes where lzma_to_bz2 is False, or figure out some way to teach the bouncer script about that.
Attachment #8909266 - Flags: review?(catlee)
Attachment #8909266 - Flags: review?(catlee) → review+
Ryan, I need this change to be on beta for 56+. What landing strategy would you suggest?
Flags: needinfo?(ryanvm)
I have other uplifts to land for 56 before the RC build. Given that this grafts cleanly, I can get it where it needs to go from here :)
Flags: needinfo?(ryanvm)
Attached patch typo.diffSplinter Review
Attachment #8909553 - Flags: review?(nthomas)
Attachment #8909553 - Flags: review?(nthomas) → review+
Deactivating partial updates was necessary because of [1]: generated partial mars were lzma instead of bz2. I manually updated the balrog blob, but then new failures came up[2]: no partials updates were found. The latest patch removes partials from expected updates paths. Once it's merged, I'll rerun the failed jobs.


[1] https://tools.taskcluster.net/groups/aUpBDSw5S2etpFHqxKSCYA/tasks/ZdiZxLNMRweZ1C8ZWYPDCw/runs/0/logs/public%2Flogs%2Flive.log
[2] https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-win32/release-mozilla-release_firefox_win32_update_verify-bm91-build1-build215.txt.gz
Comment on attachment 8909668 [details]
Bug 1395697 - Deactivate partial updates on non-LZMA releases

https://reviewboard.mozilla.org/r/181188/#review186348

r+ in order to test the completes are all working. FWIW I think we'll want to fix partials before we ship 56.0 on release.
Attachment #8909668 - Flags: review+
Hmm, reading https://hg.mozilla.org/releases/mozilla-release/file/tip/taskcluster/docker/funsize-update-generator/scripts/funsize.py#l245

            if mar_type == 'to' and not is_lzma_compressed_mar(dest):
                use_old_format = True

Sounds like we use the "to" compression format to decide wether we want lzma or bz2. Shouldn't it be "from"?
Flags: needinfo?(sfraser)
Attached patch from.diff (obsolete) — Splinter Review
Probaly we need this.
Attachment #8909695 - Flags: review?(sfraser)
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #37)
> Hmm, reading
> https://hg.mozilla.org/releases/mozilla-release/file/tip/taskcluster/docker/
> funsize-update-generator/scripts/funsize.py#l245
> 
>             if mar_type == 'to' and not is_lzma_compressed_mar(dest):
>                 use_old_format = True
> 
> Sounds like we use the "to" compression format to decide wether we want lzma
> or bz2. Shouldn't it be "from"?

This was done for nightly - Just to make sure, let's work it through. 

There's a watershed, so watershed-N will all have updaters that need bz2, but will only ever get sent data to take them to the watershed. 

The watershed itself has to be bz2, but the updater in it only supports lzma. So anything after the watershed needs lzma.

From              To                UpdaterNeeds  Checking 'to'  Checking 'from'
Watershed-2(bz2)  Watershed-1(bz2)  bz2           bz2            bz2
Watershed-1(bz2)  Watershed(bz2)    bz2           bz2            bz2
Watershed(bz2)    Watershed+1(lzma) lzma          lzma           bz2
Watershed+1(lzma) Watershed+2(lzma) lzma          lzma           lzma

So this worked for nightly - but presumably you're having issues because there isn't a similar watershed now? 

I wonder if we could modify this just because we've built all the relevant nightly partials now.
Flags: needinfo?(sfraser)
Comment on attachment 8909695 [details] [diff] [review]
from.diff

<sfraser> rail-away: have responded. Short version is the code is correct as-is for nightly, am unsure if what you're doing with 54 is in the same situation, or will have the same updater change
<•rail-away> I see what you mean, having the MAR bz2-comressed tells you nothing about the updater... hmmm
<•rail-away> right... we can't use the patch as is
Attachment #8909695 - Flags: review?(sfraser)
Maybe something like this?
Attachment #8909695 - Attachment is obsolete: true
Attachment #8909707 - Flags: review?(sfraser)
Depends on: 1401176
Apparently we use wrong signature format for partials that suppose to use BZ2 compression and thus SHA1 ("mar") as the signature format.
Attachment #8909833 - Flags: review?(jlorenzo)
Comment on attachment 8909707 [details] [diff] [review]
from_less_than_56.diff

I think that should work - basing it on what the updater will understand given from the version. Go for it
Attachment #8909707 - Flags: review?(sfraser) → review+
Pushed by raliiev@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6c0e0f992f33
Use BZ2 MAR compression for versions less than 56.0 r=sfraser
Comment on attachment 8909833 [details] [review]
Use proper signing format for partials

r+ed in the PR.
Attachment #8909833 - Flags: review?(jlorenzo)
Attachment #8909833 - Flags: review+
Attachment #8909833 - Flags: checked-in+
Depends on: 1402011
Depends on: 1402015
in 56.0.2 we had no partials for versions less than 56.0 so we ended up not passing --partial-update to this script.
Attachment #8921607 - Flags: review?
Attachment #8921607 - Flags: review? → review?(jlund)
Attachment #8921607 - Flags: review?(jlund) → review+
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.