Create Windows signing worker in Taskcluster

RESOLVED FIXED

Status

task
RESOLVED FIXED
3 years ago
a year ago

People

(Reporter: coop, Assigned: kmoir)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

We need a signing worker that can handle signing Windows internals and packages.
Assignee

Comment 1

3 years ago
So my thinking is that I would just have to setup a worker to have the same environment etc using 
https://tools.taskcluster.net/aws-provisioner/#win2012/

I've looked at the docs/puppet configuration for the existing signing servers and it looks like it's created with a virtual env here

http://hg.mozilla.org/build/puppet/file/tip/modules/signingworker/manifests/init.pp

and then is started like this + credentials
http://hearsum.ca/blog/signing-software-at-scale.html

under "Running your own signing server"

:bhearsum is this still accurate?  I don't have access to the signing servers to look at them.
Flags: needinfo?(bhearsum)
(In reply to Kim Moir [:kmoir] from comment #1)
> So my thinking is that I would just have to setup a worker to have the same
> environment etc using 
> https://tools.taskcluster.net/aws-provisioner/#win2012/
> 
> I've looked at the docs/puppet configuration for the existing signing
> servers and it looks like it's created with a virtual env here
> 
> http://hg.mozilla.org/build/puppet/file/tip/modules/signingworker/manifests/
> init.pp
> 
> and then is started like this + credentials
> http://hearsum.ca/blog/signing-software-at-scale.html
> 
> under "Running your own signing server"
> 
> :bhearsum is this still accurate?  I don't have access to the signing
> servers to look at them.

I just checked on a signing server and that's still the command we use to start them.
Flags: needinfo?(bhearsum)
(In reply to Kim Moir [:kmoir] from comment #1)
> So my thinking is that I would just have to setup a worker to have the same
> environment etc using 
> https://tools.taskcluster.net/aws-provisioner/#win2012/

This will probably be on signing scriptworker.  If `mach repackage` can run on the same instances, we might reuse the same workers.  If `mach repackage` ends up with different requirements, we might spin up a second pool of signing scriptworkers.

> I've looked at the docs/puppet configuration for the existing signing
> servers and it looks like it's created with a virtual env here
> 
> http://hg.mozilla.org/build/puppet/file/tip/modules/signingworker/manifests/
> init.pp

The signingworker in puppet today is python2, and will be obsoleted by the signing scriptworkers that I'm going to deploy soon.  https://github.com/escapewindow/build-puppet/tree/scriptworker

You'll need the passwords to connect to the signing servers to request tokens, but you don't necessarily need access to the signing servers otherwise.  I pared down https://github.com/escapewindow/build-puppet/blob/scriptworker/modules/signing_scriptworker/templates/passwords.json.erb to only have access to the dep signing keys, until we want to flip the switch to tier1.

https://github.com/escapewindow/docker-signing-server is a dev signing server we can test against, but it only supports gpg right now.  To add other signing format support, we need to add developer (self-signed?) keys, and then change the startup to enter in blank passphrases for all the keys.
(In reply to Aki Sasaki [:aki] from comment #3)
> If `mach repackage` can run
> on the same instances, we might reuse the same workers.  If `mach repackage`
> ends up with different requirements, we might spin up a second pool of
> signing scriptworkers.

To be explicit:
* `mach repackage`, or something like it, will be needed to do windows signing as a separate step from builds.
** unsigned windows packages need to be exploded, all the internal binaries signed separately, then repackaged, and the external package also signed.
* `mach repackage` is not written yet
* writing `mach repackage` will likely be the long pole for this bug.
* I'm not certain who is assigned the task of writing `mach repackage`.  Its absence blocks this bug.  In my mind, the actual windows signingscript + scriptworker implementations/deployment are relatively easy once linux signing scriptworker is deployed.  The main difference is `mach repackage`.
Assignee

Updated

3 years ago
Depends on: 1287881
in bug 1287881 we implemented actions that perform the repackage

- https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/7z_exe_extract.py
- https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/7z_exe_archive.py

i think (please correct if wrong) this doesn't completely fulfil the requirement to implement `mach repackage`, but i'd like to continue with the implementation and am looking for pointers on what the next step would be.

eg: where does the call to the signing servers come in or where can i add that?
Flags: needinfo?(aki)
removing the ni because i've just found a bunch of reading material on adding a mach command. anyone cruising past, please feel free to drop pointers here as i'm a complete newb to adding mach commands.
Flags: needinfo?(aki)
some questions (for anyone passing who knows):

- what's a good mach_commands.py template/scaffold for the `mach repackage` implementation to copy/build off of?
  i'm looking at: python/mozbuild/mozbuild/mach_commands.py. am i warm?

- is there a taskcluster linux/osx mach-based signing implementation or bug that will give me some ideas about where in the build process the call to unpackage, sign, repackage belongs?

- where is buildbot windows signing implemented?
  (i'm looking for the command that calls the signing server asking for a specific binary to be signed)
just found Kim's osx patch which contains a lot of what i need at https://bugzilla.mozilla.org/attachment.cgi?id=8845653&action=diff
(In reply to Rob Thijssen (:grenade - GMT) from comment #8)
> just found Kim's osx patch which contains a lot of what i need at
> https://bugzilla.mozilla.org/attachment.cgi?id=8845653&action=diff

Windows will largely match OSX, so that's a good approach.  I think Windows will differ in one regard: we need a 2nd signing pass to sign the repackaged executable.  If you need a hand with that, holler; a number of us in #tcmigration have dealt with that type of problem.  And if we get 3/4 of the way there (repackaged exe with signed-internals), we're winning.
See Also: → 1323732
Depends on: 1360525
Depends on: 1357112
Depends on: 1361402
Depends on: 1265537
Blocks: 1362511
Blocks: 1357125
Blocks: 1357136
Blocks: 1362494
Blocks: 1267427
Blocks: 1362497
We're now signing on date, including mach repackage of the installer/complete.mar
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED

Updated

a year ago
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.