Closed Bug 1440546 Opened 2 years ago Closed Last year

Rework partner-repacks.py for taskcluster

Categories

(Release Engineering :: Release Automation: Other, enhancement)

enhancement
Not set

Tracking

(firefox61 fixed)

RESOLVED FIXED
Tracking Status
firefox61 --- fixed

People

(Reporter: nthomas, Assigned: nthomas)

References

Details

Attachments

(2 files, 1 obsolete file)

Steps to complete:
1, prove it's possible to repack all platforms on taskcluster linux workers, where the job needs to only do the partner repacking, and not signing, uploading etc
2, polish up the code from 1, make it ready to be scheduled
For 1, here's a subset of locales for EME-free builds
  mac:     https://tools.taskcluster.net/groups/b2g5xXFVSS2l1QdZXNSR_g/tasks/b2g5xXFVSS2l1QdZXNSR_g/runs/0/artifacts
  win32:   https://tools.taskcluster.net/groups/G9QiaePQRkaevoKGd69tCA/tasks/N4QkU1XiSYyM_pyCCPZKhQ/runs/0/artifacts
  linux32: https://tools.taskcluster.net/groups/P-42mBdIRqebSmHEkXKZJA/tasks/P-42mBdIRqebSmHEkXKZJA/runs/0/artifacts
running on the same image as we use for l10n repacks.

Code to do this:
 mozharness diff: https://hg.mozilla.org/users/nthomas_mozilla.com/mozilla-unified/rev/4b41b28b2928
 repack script:   https://github.com/nthomas-mozilla/repack-scripts-2/blob/e8b1430/tc-partner-repacks.py
 manifests:       https://github.com/nthomas-mozilla/mozilla-EME-free-manifest/blob/0c69010/default.xml
 repack config:   https://github.com/nthomas-mozilla/mozilla-EME-free/blob/7930b04/desktop/mozilla-EME-free/repack.cfg

Notes:
* taskIds for upstream tasks are the Ns tasks of the vanilla en-US and l10n repacks, which emit target.{zip,tar.gz,tar.bz2} for {win*,mac,linux*}. Passing them in manually is a temporary measure until we hook that up via scheduling
* the builder image doesn't have any issues with older TLS deprecation on github (bug 1435441), so I've removed the support I added for that. For expediency I'm cloning repos via https, but we'll do that via ssh using a new r/o key

Known issues:
* tc-partner-repacks.py hangs if an artifact download fails
* not convinced it's actually doing any multiprocessing
Blocks: 1398803
https://tools.taskcluster.net/groups/DznFEIoHS5mpWs6b6csULQ/tasks/DznFEIoHS5mpWs6b6csULQ is a successful eme-free job for mac, which has closer-to-final artifact definitions, and passes in the upstreams tasks via an UPSTREAM_TASKIDS env var.

Known TODOs
* for windows, copy setup.exe for each repack and attach it to the partner repack job, do pass on to the repackage task downstream
* more massaging of artifacts definitions, especially for partners which go to the private s3 bucket, see also bug 1447673
* support cloning via ssh, including provisioning the ssh key via the secret service
(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #2)
> * support cloning via ssh, including provisioning the ssh key via the secret
> service

We've used repo to clone the code and configs from github for a long time, via git-over-ssh to see the private repos. But in TC the image for linux builds doesn't have ssh available. My job in comment #2 uses https:// cloning with public repos to sidestep this.

Ben, would you be able to look at the best solution ? There seem to be quite a few images in taskcluster/docker already, so that might be the normal way of doing things ? Saves convincing someone to accept changes to their image I suppose. There aren't a lot of requirements for the script: openssh-client, git, tar, bzip2, gzip, and zip; I don't mind what distro it's based on. 

To avoid ssh prompting to accept github's host sig (and hanging the job) I think we'll need to add a known_hosts file, or set StrictHostKeyChecking=accept-new (or off, eg via GIT_SSH_COMMAND in mozharness). Any thoughts on the best trade-off there for maintenance vs security ? There's the hgfingerprint in the secret service already for precedence. The actual ssh key I'm thinking will come in at runtime via mozharness's SecretsMixin.

BTW, I had a look at using https://<username>:<personal_access_token>@github.com/<org>/<repo> instead of ssh, but didn't find a path which didn't involve a lot of munging to keep using repo, or rewriting repo in mozharness.
Flags: needinfo?(bhearsum)
(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #3)
> (In reply to Nick Thomas [:nthomas] (UTC+13) from comment #2)
> > * support cloning via ssh, including provisioning the ssh key via the secret
> 
> Ben, would you be able to look at the best solution ? There seem to be quite
> a few images in taskcluster/docker already, so that might be the normal way
> of doing things ? Saves convincing someone to accept changes to their image
> I suppose. There aren't a lot of requirements for the script:
> openssh-client, git, tar, bzip2, gzip, and zip; I don't mind what distro
> it's based on. 

I'll look into this.
Flags: needinfo?(bhearsum)
My WIP is on maple. I ended up embedding the host key right in the image - we can definitely change/improve this, but maybe it's okay for now?
......but it's probably worth the trade off.)

I've embedded the Github host key right in this image - we can change this up if it's a bad idea.

curl is being installed, but may not be needed in the end. Right now we need it for the hacky "fake out artifacts" stuff that's happening on maple.
The latest staging release (https://tools.taskcluster.net/groups/M-tvfF1XTDSKHyh7S5ZPnQ/tasks/ZkpiQnIAQW6sXSPgei52UQ/details) seemed to work OK-ish (modulo issues unrelated to the new docker image). I tried out the image locally, too, and was able to "ssh github.com" without getting prompted for the ssh key. I haven't done any work trying to sort out the ssh key deployment/usage. If you want me to look at that too, just let me know.
Thanks very much for working on that! I moved the known_hosts as the interactive session connects as root, while the job runs as 'worker', and repackage.sh insists on calling xvfb.sh.
Attachment #8965814 - Attachment is obsolete: true
(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #9)
> Created attachment 8967154 [details] [review]
> Add new repacking script to existing repack-scripts repo

This is merged, and we're using it for staging releases on maple. There are other changes to mozharness code and configs which are in maple.
Comment on attachment 8969073 [details]
Bug 1440546 - Mozharness support for new partner repack script

Justin Wood (:Callek) has approved the revision.

https://phabricator.services.mozilla.com/D973
Attachment #8969073 - Flags: review+
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0a202b97948b
[partner-repack] Mozharness support for new partner repack script; r=Callek
https://hg.mozilla.org/mozilla-central/rev/0a202b97948b
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.