1346354 - Create a script worker to obtain transparency certificates

Status: RESOLVED INCOMPLETE

(Release Engineering :: Release Automation, enhancement, P2)

Description

[Richard Barnes [:rbarnes]]

For binary transparency, we will need release automation to create a "transparency certificate" with information that summarizes the release:

* The tree head from SHA256SUMMARY (cf. Bug 1342974)
* The version label for the release
* The product (for now, just "firefox")

It looks like the simplest way to do this right now is to use Let's Encrypt, via the lego tool [1] and Route53 control of the transparency domain (Bug 1346351).  So this script worker will need access to Route53 credentials.

It should not need access to any other secrets.  The Let's Encrypt interactions can be done with ephemeral account credentials, and the private key for the certificate can be deleted immediately.

So the script in this worker should just gather the relevant data, form the domain name, and call out to lego to get the cert.

[1] https://github.com/xenolf/lego

Comment 1

[Richard Barnes [:rbarnes]]

Attachment: Bug 1346354 - Create a script worker to obtain transparency certificates

Review commit: https://reviewboard.mozilla.org/r/120618/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/120618/

Comment 2

[Richard Barnes [:rbarnes]]

Comment on attachment 8847690 [details]
Bug 1346354 - Create a script worker to obtain transparency certificates

Rail: Does this look like the beginnings of something you could wrap into a ScriptWorker? I've verified that it works, in that it properly parses the summary file, issues a certificate, and outputs something that CT logs will accept.  It's a little rough around the edges, e.g., with the `shell=True` bits at the end.

> python scripts/release/issue-transparency-cert.py \
>        --stage-product firefox \
>        --version 53.0b5 \
>        --contact [e-mail] \
>        --summary https://ipv.sx/tmp/SHA256SUMMARY \
>        --chain TRANSPARENCY.pem

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8847690 [details]
Bug 1346354 - Create a script worker to obtain transparency certificates

I'm not yet familiar with the scriptworker codebase. Aki probably can help here.

Comment 4

[Aki Sasaki (not active)]

What timeframe are we looking at?  By the ask, I'm guessing you want someone in releng to do this work?  If so, we'll need to get this prioritized against our other work.  Otherwise there are lots of people willing to help whoever does take this on.

Tl;dr:
- for scriptworker, we'll have to pass the commandline options via config.json or task.json
- we're going to need to define the task definition schema
- we're going to need to decide whether to use chain of trust; this is a big one
- there's an "adding new scriptworker instance types" doc that may need some updates [0]
- we may want to write this in python3

# commandline options

- Currently we have a puppet limitation of |$executable $script $config| as the commandline [1].  (This is a releng scriptworker module limitation, which is potentially fixable; we also have the optionof setting up this scriptworker type without using that puppet module.)  The config looks something like [2] ... configurable per script, but we generally want to specify scriptworker's `work_dir` and `artifact_dir`.  In this case, we're specifying where the signing server config is in `signing_server_config` and otherwise adding some signingscript config items that are true for every signing task.

- Scriptworker will always run the same commandline for every task sent to it.  Currently, we definethe behavior of each task via the task definition, e.g. [3].  Scriptworker will populate this on disk in `$work_dir/task.json` before launching the script.

- For security reasons, it's best to hardcode anything that remains true across all runs in either the script or the script's config.json... the fewer things that a malicious/broken task definition canbreak, the better.

# task definition schema

Referencing [3] again:

The provisionerId / workerType / schedulerId help specify what workers should pick this task up.  We're using scopes to both act as ACLs and as behavior: in this case, we're specifying what class of signing certs to use (release) and which signing formats we'll be using (gpg and mar).

```
project:releng:signing:cert:release-signing
project:releng:signing:format:gpg
project:releng:signing:format:mar
```

The `upstreamArtifacts` specify which artifact `paths` we want to sign with which `formats` from which `taskId`s.  The `taskType` is for chain of trust verification.  More on `upstreamArtifacts` in thechain of trust section below.  When `verify_chain_of_trust` is True, scriptworker pre-downloads these into disk, and verifies their shas against the upstream signed chain of trust artifacts.  Otherwise, the script has to deal with artifact download + verification.

We currently use jsonschema to verify the task definition schema [4] [5].  This reuses some scriptworker code, but that requires python>=3.5 for async.  We're already running a py2 script in scriptworker (balrogscript requires some libraries that haven't been ported to py3 yet), so it's doable; I intentionally chose to have scriptworker launch a standalone script to allow for different languages.

# Chain of Trust

Scriptworker currently lets you set 3 different booleans in the `scriptworker.yaml`: `sign_chain_of_trust`, `verify_chain_of_trust`, and `verify_cot_signature` [6]

It's easiest to turn them off, and you don't have to deal with anything gpg.  However, then the burden of verifying request and artifact validity is entirely on the script.

Consider:
- anyone with the right scopes can submit a task that passes the above jsonschema check.
- many more people will have the right scopes than you will be comfortable with.  this is a general rule.
- chain of trust was written specifically as a second factor to mitigate scope compromise or misuse
- what is the risk if someone is able to run this script with malicious intent?

## Checks to add, especially if disabling chain of trust

Most likely, the fact that the checksums files are release-signed is a good check.  Hardcoding or allowlisting certain options is a good thing.  These are checks that are good whether chain of trust ison or not.

For funsize we are using a hack where we sign something and embed it in the `task.payload`; the funsize workers verify the signature and signed request before proceeding.  (This means submitting these requests is either manual or requires a locked down machine that has the private key and passphrase.) This is a decent fallback option if chain of trust is disabled.

## Considerations before enabling chain of trust

The biggie is probably that we require the task definition of this task, and all upstream tasks, to be generated from an in-tree decision task, i.e. `mach taskgraph`.  We're looking at moving our 2nd releasetasks graph (push-to-releases) to an in-tree chain-of-trust enabled graph, but we're not there yet.

We're working around this with pushapk by scheduling it as part of the build graph, but with a breakpoint task dependency that can be manually resolved to run pushapk (push to google play).  We could potentially do something similar with binary transparency until we have the second graph in tree.

Adding a new task type to the chain of trust isn't trivial, but we have a number of people in #tcmigration who have gone through it and can help.

[0] http://scriptworker.readthedocs.io/en/latest/new_instance_types.html
[1] https://hg.mozilla.org/build/puppet/file/f84e0afe760d/modules/scriptworker/templates/scriptworker.yaml.erb#l13
[2] https://github.com/mozilla-releng/signingscript/blob/master/config_example.json
[3] https://queue.taskcluster.net/v1/task/Her_cJYgR1eeULRRSfi2hw
[4] https://github.com/mozilla-releng/signingscript/blob/master/signingscript/data/signing_task_schema.json
[5] https://github.com/mozilla-releng/signingscript/blob/master/signingscript/task.py#L58
[6] https://github.com/mozilla-releng/scriptworker/blob/master/scriptworker/constants.py#L62-L64

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 8847690 [details]
Bug 1346354 - Create a script worker to obtain transparency certificates

https://reviewboard.mozilla.org/r/120618/#review122640

feedback+ : comments at https://bugzilla.mozilla.org/show_bug.cgi?id=1346354#c4

Comment 6

[Aki Sasaki (not active)]

(We could certainly write a wrapper transparencyscript.py that does the above scriptworker-specific things, then calls the mozharness script with the appropriate options.)

Comment 7

[Aki Sasaki (not active)]

I know comment 4 is a lot of text, so here's a simpler starting point:

1. Write a script that scriptworker can call, that does what you need it to.

   This script should be take one commandline argument, the path to a config file.

1a. Create the config file.  At some point we'll need to separate the per-run config from the per-instance config, but at first it's easy to put everything in the config file.

   The config file should contain these two key/value pairs: `work_dir` and `artifact_dir`, which should point to two different directories on disk.  Looking at rbarnes' script, it seems like these additional items might be good to put into config:

    - the path to `lego`
    - AWS config
    - stage_product
    - version
    - contact
    - summary
    - chain

Version and stage_product may vary per-run, which means they'll move to `work_dir`/task.json at some point (likely in `task.payload` or `task.extra`), though we can worry about that later.  Once we define the task.json schema, we'll want to check that as well, but we can worry about that in a later step.

We probably want to hardcode the contact in the config, or have an allowlist of acceptable contacts.  The summary url should probably be a hardcoded template string that expands via stage_product and version.  Since we have an `artifact_dir`, if we want to upload the resulting output .crt file, let's copy the `chain` into that directory.  The convention is to put it into a `public/` directory if it's intended to be public, and `private/` if it's intended to be private.  So possibly we'll want to copy it into `artifact_dir`/public/build/SOME_NAME.crt, and it'll be uploaded as public/build/SOME_NAME.crt in the task.  We probably don't have to have this change per-run, so we can hardcode that or put it in the config file.

1b. Running the script with the config file as its argument should (1) read the config file, (2) do any sort of verification (this will expand later), (3) create the cert (this can call the mozharness script, or we can pull the logic from the mozharness script into this script), and (4) copy any artifacts to upload into `artifact_dir`.  On success, exit 0.  On failure, use this error-to-exitcode map: https://github.com/mozilla-releng/scriptworker/blob/68699020a62715acc5f663d9db98935df29b8504/scriptworker/constants.py#L262-L272

Does that simplify the starting point?

Comment 8

[Christoph Kerschbaumer [:ckerschb}]

Thanks Aki. Still having trouble to allocate the required time to get up to speed on those things. I would really hope to get some help from releng folks. Maybe next quarter? Or potentially a great intern project?

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

https://phabricator.services.mozilla.com/D32532 removes support for generating binary transparency tasks in the release task graphs. We can revert that if BT becomes a priority in future.

Comment 10

[Julien Cristau [:jcristau]]

As far as I can tell this effort died down some years ago; closing.