1393789 - Repack updates for 56.0 RCs

Status: RESOLVED WORKSFORME

(Release Engineering :: Release Requests, enhancement, P1)

Description

[Chris AtLee [:catlee]]

The 56.0 release candidates will be served to two groups of users:

- beta channel users. these users will be forced through the watershed of 56.0b3, and will only accept LZMA compressed MAR files signed with the new SHA384 format.

- release channel users. these users will only accept BZ2 compressed MAR files signed with the old SHA1 format.

As a result, we will need to generate both types of updates for the 56.0 release candidates.

We're currently planning to let release automation run as normal and produce new-format MARs to serve to the beta population.

We need to create process to repack these into the old format and publish them via S3, bouncer, balrog, etc.

Comment 1

[Chris AtLee [:catlee]]

downloading/repacking and resigning all 1900 updates from 56.0b9 took 2,950s on my test instance.

I've parallelized the process 8-ways, and am caching the compressed bz2 contents of files so we don't need to recompress them for each mar file. The biggest bottleneck is signing. Just downloading / repacking all the mar files takes about 10 minutes.

Comment 2

[Chris AtLee [:catlee]]

Attachment: repack_mars.py

Comment 3

[Chris AtLee [:catlee]]

Need to figure out a way to update checksums too.

Comment 4

[Chris AtLee [:catlee]]

Attachment: Firefox-56.0b11-build1-bz2.json

New release blob for 56.0b11 w/ URLs changed for bz2 files.

Comment 5

[Chris AtLee [:catlee]]

Attachment: Firefox-56.0b11-build1-bz2.json

stripped out partials that won't work

Comment 6

[Chris AtLee [:catlee]]

I repacked the 56.0b11 complete updates only, and uploaded them to the mozilla-releng-test bucket.

There's no point repacking the complete updates 56.0b11, since none of them will apply correctly using the beta's updater.

oremj: would you be able to copy the following path from mozilla-releng-test bucket into the product delivery bucket: /pub/firefox/candidates/56.0b11-candidates/build1/update-bz2

no keys should be overwritten.

Comment 7

[Jeremy Orem [:oremj]]

Which account does mozilla-releng-test belong to? I don't think I have access.

Comment 8

[Chris AtLee [:catlee]]

It's in the releng account. I've made the bucket public, so you should be able to read objects from it.

Comment 9

[Chris AtLee [:catlee]]

Attachment: Firefox-56.0b11-build1-bz2.json

fixed blob name, and updated sizes/hashes

Comment 10

[Chris AtLee [:catlee]]

Attachment: transform_balrog_blob.py

script for transforming balrog blobs

Comment 11

[Chris AtLee [:catlee]]

Attachment: transform_balrog_blob.py

updated for 56.0-build1

Comment 12

[Chris AtLee [:catlee]]

Attachment: Firefox-56.0-build1-bz2.json

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8909603 [details]
Firefox-56.0-build1-bz2.json

Looks good. I've added 
  "actions": "showURL",
  "openURL": "https://www.mozilla.org/%LOCALE%/firefox/56.0/whatsnew/?oldversion=%OLD_VERSION%",
and renamed it to Firefox-56.0-build1-bz2-WNP, to satisfy bug 1397339. Uploaded to Barlog, and made rule 628 point to it release-localtest, and 627 for release-cdntest.

Comment 14

[Chris AtLee [:catlee]]

Attachment: transform_balrog_blob.py

Comment 15

[Chris AtLee [:catlee]]

Attachment: transform_balrog_blob.py

Comment 16

[Chris AtLee [:catlee]]

Attachment: transform_balrog_blob.py

Comment 17

[Chris AtLee [:catlee]]

This latest version grabs the sha512sums from the beetmover artifacts. It validates those against the signed SHA512SUMs in the root of the release directory. It should run much faster, and not require downloading all the mar files!

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: transform_balrog_blob.py

Now strips away 56.0b12 partials from locale and fileUrls data, which isn't strictly necessary but avoids serving bogus updates if we mix up the blobs and channels. 

Optionally adds what's new page with --wnp arg.

Doc update - https://github.com/mozilla/releasewarrior/commit/da6b8005090d25b59baed8af63f14bf0a80de52b

Comment 19

[Firefox Bug Husbandry Bot]

Bulk change of QA Contact to :jlund, per https://bugzilla.mozilla.org/show_bug.cgi?id=1428483

Comment 20

[Chris AtLee [:catlee]]

I suspect that whatever we wanted to do here is done :)