Closed Bug 1442545 Opened 6 years ago Closed 6 years ago

Scheduling partner repacks on taskcluster

Categories

(Release Engineering :: Release Automation: Other, enhancement)

enhancement
Not set
normal

Tracking

(firefox61 fixed)

RESOLVED FIXED
Tracking Status
firefox61 --- fixed

People

(Reporter: nthomas, Assigned: mozilla)

References

Details

Attachments

(5 files, 8 obsolete files)

5.92 KB, text/plain
Details
1.54 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
1.41 KB, patch
mozilla
: review+
Details | Diff | Splinter Review
2.43 KB, patch
rail
: review+
Details | Diff | Splinter Review
2.24 KB, patch
rail
: review+
Details | Diff | Splinter Review
Rough outline - Task generation for partner repacks:
- pull in partner configs when running the action task (r/o github creds via TC secret service, with graceful fallback for unscoped and testing)
- create tasks as necessary per platform: partner repack, signing, repackage, repackage-signing

We can start with EME-free builds to prove this out with all public artifacts.
This script uses the Github API (v4, GraphQL) to grab the repack manifest from the specified repo, parse it for partners, and then walk those repos to retrieve all repack.cfg files. It works for both public and private manifests (given sufficient privs on the Github Personal Access Token (PAT) in the latter case). We'd likely have a token in the TC secret service (this is not provisioned yet).

Example output for my modified mozilla-EME-free setup:
2018-03-02 22:36:00,088 - DEBUG - __main__.<module>#146: Found configs: {
    "mozilla-EME-free:dummy": "foo bar baz\n",
    "mozilla-EME-free:mozilla-EME-free": "aus=\"mozilla-EMEfree\"\ndist_id=\"mozilla-EMEfree\"\ndist_version=\"1.0\"\nlinux-i686=true\nlinux-x86_64=true\nlocales=\"ach af de en-US\"\nmac=true\nwin32=true\nwin64=true\noutput_dir=\"%(platform)s-EME-free/%(locale)s\"\n\n# Upload params\nbucket=\"net-mozaws-prod-delivery-firefox\"\nupload_to_candidates=true\n"
}

You can see it's not parsing the repack.cfg content yet, which we'll need to do to pick out upload_to_candidates, and possibly bucket & output_dir for beetmover config; and maybe enabled platforms for the graph gen.

The idea with this approach is to avoid requirements for git and repo at graph generation time. Only need requests, which should already be available given what we do in nightly action tasks on m-c to look up data for partial generation.
Attachment #8955431 - Attachment mime type: text/x-python-script → text/plain
I've been looking into the details of generating the tasks we'll need for this a bit. My idea is to make as much use of the existing transforms for signing, repackage, and repackage-signing as possible. My hope is that if we can get the right bits fed into signing, the downstream jobs should mostly "just work". Some random notes:
* We'll almost certainly need new kinds for each of the above steps, because the upstream dependencies start with release-partner-repack, not builds or repacks.
* Some of these steps are no-ops, depending on platform. Linux does not need to run any of them AFAIK, Mac only needs signing & repackage, while Windows only needs repackage & repackage-signing. All platforms need beetmover, which sounds like will be handled elsewhere.
* We may need some hacks to remove these jobs from Treeherder. The current "release_deps" transform requires each side of a dependency to be in the same treeherder "tier", and the existing signing, repackage, etc. transforms set tier 1 (which is forcing me to set partner repacks to tier 1, at the moment).
I've gotten as far as I can on this for the week. I've managed to get signing tasks generated with some tweaks to transforms that look correct-ish, specifically:
* Not setting any upstreamArtifacts for Windows & Linux. Dunno if this will work. We might need to modify the signing tasks to bail on this, or somehow remove them from the graph entirely. (I think the latter will be tough, because of the way kind dependencies/release_deps works.)
* Set multiple upstreamArtifacts for Mac. This is hardcoded in the transform right now, but based on what Nick has said, I think this will end up coming from a loop over something in config.params.
* Set some routes. This probably need to change once we figure out private artifacts.

I've not figured out how the heck I can test this yet - I think I'll need the partner repack tasks to actually produce artifacts, and ideally - some way to run partner repacks + downstreams without running an entire release.

https://github.com/mozilla/gecko-dev/compare/master...mozbhearsum:partner-repack-scheduling?expand=1 is my current code, which needs a ton of refactoring and hack removal.
(In reply to Ben Hearsum (:bhearsum) from comment #3)
> https://github.com/mozilla/gecko-dev/compare/master...mozbhearsum:partner-
> repack-scheduling?expand=1 is my current code, which needs a ton of
> refactoring and hack removal.

I made a bunch of progress on this in the past week. I've now got a graph that includes (and probably fans out for each partner+locale):
* partner repack
* signing
* repackage
* repackage signing
* beetmover

I've done a lot of checks of the output of signing, repackage, and repackage signing - and I think they're in pretty shape. Beetmover is currently only running for Windows, and is currently busted. More work is needed to get the Linux and Mac jobs scheduled, and it looks like some changes to the beetmover mainfest are needed to make them green.
Attachment #8955658 - Attachment is obsolete: true
(In reply to Ben Hearsum (:bhearsum) from comment #4)
> (In reply to Ben Hearsum (:bhearsum) from comment #3)
> > https://github.com/mozilla/gecko-dev/compare/master...mozbhearsum:partner-
> > repack-scheduling?expand=1 is my current code, which needs a ton of
> > refactoring and hack removal.
> 
> I made a bunch of progress on this in the past week. I've now got a graph
> that includes (and probably fans out for each partner+locale):
> * partner repack
> * signing
> * repackage
> * repackage signing
> * beetmover
> 
> I've done a lot of checks of the output of signing, repackage, and repackage
> signing - and I think they're in pretty shape. Beetmover is currently only
> running for Windows, and is currently busted. More work is needed to get the
> Linux and Mac jobs scheduled, and it looks like some changes to the
> beetmover mainfest are needed to make them green.

I've now got what I think to be a graph that's shaped properly. Beetmover is still failing, but I think it's time to get real feedback on this work before proceeding any further. I'll post a series of patches shortly.
Attachment #8958460 - Attachment description: * Create eme-free repack kind* Dummy out eme-free & partner repack tasks to allow downstreams to work* Add partner repack support to name_sanity transform* Create stubbed out version of partner_config → Create eme free kind & dummy out tasks to allow downstreams to work
Attachment #8958463 - Attachment description: * Add eme-free and partner repack repackage (heh) kinds. * Add transform to chunk partners. → Create repackage kinds for eme-free/partners
Attachment #8958466 - Attachment description: * Add beetmover kinds for eme-free and partner repacks * Add chunking dummy to deal with Linux, with doesn't run repackage (so it can't get chunked there) → add eme-free and partner repack beetmover tasks
Depends on: 1451481
Attachment #8958443 - Attachment is obsolete: true
Attachment #8958460 - Attachment is obsolete: true
Attachment #8958462 - Attachment is obsolete: true
Attachment #8958463 - Attachment is obsolete: true
Attachment #8958465 - Attachment is obsolete: true
Attachment #8958466 - Attachment is obsolete: true
Attachment #8958467 - Attachment is obsolete: true
Attached patch partner-toolsSplinter Review
Attachment #8968982 - Flags: review?(bhearsum)
Attachment #8968982 - Flags: review?(bhearsum) → review+
Attached patch py3.diffSplinter Review
Attachment #8969103 - Flags: review?(aki)
Attachment #8969103 - Flags: review?(aki) → review+
Depends on: 1440546
If we need jamun releases soon for esr, we probably want to skip partner configs until that code is merged in.
Assignee: nobody → aki
Attachment #8969105 - Flags: review?(rail)
We may want to enable tc partner repacks for beta tomorrow...
Attachment #8969106 - Flags: review?(rail)
Attachment #8969106 - Flags: review?(rail) → review+
Attachment #8969105 - Flags: review?(rail) → review+
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2e43f41eb784
Add max-run-time to signing workers; r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/fc930141faf9
Propogate shipping-product/phase in signing; r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/e2a4ff5c10f6
Allow forcing artifact access to be via private URLs; r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/dd7db84b4f11
[partner-repack] Add parameters needed for doing partner repacks; r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/90fddb648f30
[partner-repack] Don't display partner repacks on treeherder; r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/991f58ef782b
[partner-repack] Add docker image for partner repacks; r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/00b9d353a6b7
[partner-repack] Add repack partner builds; r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/3773a90e8607
[partner-repack] Sign partner builds; r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/e5b269778841
[partner-repack] Repackage signed partner builds; r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/e4aa73efed4d
[partner-repack] Beetmove the partner repacks; r=Callek
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/mozilla-inbound/rev/355bcc77b538
[partner-repack] Sign repacked partner builds; r=Callek CLOSED TREE
You need to log in before you can comment on or make changes to this bug.