1492033 - Generate partner repacks on esr60

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Description

[Nick Thomas [:nthomas] (UTC+12)]

mkaply requests we generate partner repacks based on esr60, using the manifest in https://github.com/mozilla-partners/esr-repack-manifests. 

We'll have to do a bit of rejigging to support this as we didn't build in branch support.
https://hg.mozilla.org/releases/mozilla-esr60/file/default/taskcluster/ci/config.yml#l194
and explicitly disable partners on esr,
https://dxr.mozilla.org/build-central/source/tools/buildfarm/release/release-runner3.py#144

Probably other places too so testing in staging will be necessary.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I've got some patches for this in gecko, tools (releaserunner), and braindump. Next step is taking them for a spin in a staging release.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Gecko WIP

Untested except for taskgraph-gen/diff, but
* sets different manifest repo for esr
* cleans up arguments on get_partner_url_config()

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Braindump WIP

Adjusts parameters for mozilla-esr releases.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Tools WIP

This is for releases being started with ship-it v1 & releaserunner3, which is mozilla-release and mozilla-esr60 as of now.

There's something I don't understand going on here and with the gecko patch, because the ACTION_PARAMETERS on the promote decision tasks typically have
   release_enable_partners: false
   release_partner_config: {}
and we still get partner repacks done; eg late beta [1] and 62.0.3 [2]. They're using ship-it v2 and v1 respectively.


[1] https://tools.taskcluster.net/groups/SpHRT9cdTLusYVxlI9g-WA/tasks/d8IYPTiNRHmkHjw0IsfIvQ/details
[2] https://tools.taskcluster.net/groups/S-6-JjCzSBeDvbGOzIVlow/tasks/Iel3zFH9Sc64pTH2zqgEog/details

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 9013485 [details] [diff] [review]
Tools WIP

No longer needed now that ship-it v2 handles esr60.

Comment 6

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/release-services/pull/1743

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to GitHub Bugzilla PR Linker from comment #6)

This is to adjust ship-it v2 so that it sets release_enable_partners to True in the action input.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Create partner repacks during esr60 releases

Supports partner builds on esr60 by re-arranging how we specify the manifest 
repositories, which are different for esr60 compared to beta+release. Also simplifies 
the call to get_partner_url_config because everything can be determined from the project 
once the manifest information is nearby. 

This patch also adds support for partner repacks in try staging releases, for all the 
various release branches. It makes release_level available early in the decision task 
(for the release promotion action) where a Parameters object is not available. Signing 
worker type is no longer hard-coded, and we use level-1 secrets for querying Github (via 
bug 1513375). The defaults for enabling partners and EME-free now depend on the product, 
except for excluding EME-free for the esr60 release_type.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a09c3829ea4b
Create partner repacks during esr60 releases r=tomprince

Comment 10

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/a09c3829ea4b

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/56ba482ab2c2
Port bug 1492033 - Create partner repacks during esr60 releases. rs=bustage-fix

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Apologies for that comm-central bustage. Reopening to track uplift, shipit v2 patch etc.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Uplifted to esr60 repo: https://hg.mozilla.org/releases/mozilla-esr60/rev/ac44dea415b7202fc83ed63ff3b8f613111f1f23

The ship-it v2 patch is tested and ready to merge; there's a deploy on January 3 where it should go live. Then we'll get partner repacks in the first release build after that.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Rail deployed ship-it v2 in early January, so we're ready to go. I've got some tidy up to do around staging to do before closing this.

Comment 15

[Jorg K (CEST = GMT+2)]

https://hg.mozilla.org/releases/comm-esr60/rev/8e5256de94803e1c95a45530cdedd9792f52eabc

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Firefox 60.5.0esr build1 was created today, and the partner repacks are in
https://archive.mozilla.org/pub/firefox/candidates/60.5.0esr-candidates/build1/partner-repacks/mozillaonline/

Hector, could you please double check they're working as you expected.

Comment 17

[Hector Zhao [:hectorz]]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #16)

Firefox 60.5.0esr build1 was created today, and the partner repacks are in
https://archive.mozilla.org/pub/firefox/candidates/60.5.0esr-candidates/build1/partner-repacks/mozillaonline/

Hector, could you please double check they're working as you expected.

Thanks for making this happen!

Our desktop QA Yanfang will verify them.

Comment 18

[Yanfang Liu]

Verified!

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

I'm not sure yet. On a beta EME task (YkBO2xynRZO6uTEw8mBnAw) I see a scope
docker-worker:cache:gecko-level-3-mozilla-beta-build-macosx64-nightly-opt-workspace-v3-33ea6ead87f10b63cd64
providing a cache at /builds/worker/workspace, which makes me think we might be sharing with genuine repackage jobs. If so, clobbering is unhelpful to those, and a cleanup action at the end of the partner repacks would be better.

Do you know where I can read about docker-worker and how working directories are treated ?