1492617 - add ed25519 support to scriptworker

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Description

[Aki Sasaki (not active)]

We need to:

a) be able to generate ecdsa-signed cot artifacts, and
b) be able to verify ecdsa-signed cot artifacts.

To allow for a rollout of ecdsa support, we need to support both gpg-signed and ecdsa-signed cot artifacts until ecdsa is supported everywhere.

Comment 1

[Aki Sasaki (not active)]

Johan,

I'm looking at this block [1], where we allow for optional artifacts in `scriptworker.cot.verify.download_cot`. IIRC, we did this for bug 1385401 in this PR [2] because you needed to support some alternate configuration to download Google Play strings.

Do we still need this support? Ideally we have a signed cot artifact everywhere in all production release/nightly graphs at this stage. If not, is there something we can fix to get us to that point?

I need to change how we download cot artifacts to both support rolling out a detached ecdsa signature and de-support gpg signatures, and this logic complicates that. Also, if we don't need this exception anymore, we should be more secure if we remove it.

[1] https://github.com/escapewindow/scriptworker/blob/ecdsa/scriptworker/cot/verify.py#L613
[2] https://github.com/mozilla-releng/scriptworker/pull/169

Comment 2

[Aki Sasaki (not active)]

We probably want to take this opportunity to rename the artifact. `chainOfTrust.json.asc` is sometimes a misnomer (it keeps the .asc suffix, even if it's not gpg-signed), and is camelCase when most taskcluster artifacts are kebab-case.

I'm currently thinking we should deprecate `chainOfTrust.json.asc`, and move to using `chain-of-trust.json` and `chain-of-trust.json.sig` for the detached ecdsa signature. Scriptworker will need to support both file naming conventions until docker-worker and generic-worker are using the new names. We can drop the `chainOfTrust.json.asc` filename when we drop gpg support.

For the most part, scriptworker is the only thing that needs to handle this renaming. I also found this [1], however; there may be other locations.

[1] https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/mach_commands.py#1375

Comment 3

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #1)
> I'm looking at this block [1], where we allow for optional artifacts in
> `scriptworker.cot.verify.download_cot`. IIRC, we did this for bug 1385401 in
> this PR [2] because you needed to support some alternate configuration to
> download Google Play strings.
> 
> Do we still need this support?

I don't see any instances of `Continuing CoT verifications` in papertrail logs back to Dec 24. It's possible we have a less common occurrence of this; it's also possible we can drop support for optionally downloading cot artifacts.

Comment 4

[Johan Lorenzo [:jlorenzo]]

(In reply to Aki Sasaki [:aki] from comment #1)
Thanks for looping me in, Aki! I think we still need it, even though this event remains unlikely. The root motive is to prevent bug 1383642 comment 4 from happening again. I see this bug hasn't been fixed yet. So we either need a new way to mitigate downtime, or we keep optional artifacts. How complex would the ECSDA implementation be?

Comment 5

[Aki Sasaki (not active)]

(In reply to Johan Lorenzo [:jlorenzo] from comment #4)
> (In reply to Aki Sasaki [:aki] from comment #1)
> Thanks for looping me in, Aki! I think we still need it, even though this
> event remains unlikely. The root motive is to prevent bug 1383642 comment 4
> from happening again. I see this bug hasn't been fixed yet. So we either
> need a new way to mitigate downtime, or we keep optional artifacts.

I might be reading this wrong, but in my understanding,

1. if we hit bug 1383642 comment 4 again, we won't publish any task artifacts
  1a. but the worker will still publish a chainOfTrust.json.asc artifact
2. if the chainOfTrust.json.asc artifact exists, we can download it during cot verification, and just not check the task artifact's sha if it's optional

Is that incorrect? I believe I'm keeping optional artifact support, but making a chain of trust artifact mandatory.

> How
> complex would the ECSDA implementation be?

I've already written code to support ecdsa cot generation and verification. `download_cot` looks like this [1], although I made a chainOfTrust.json.asc mandatory. It's technically possible to make chainOfTrust.json.asc optional again, but a) I want to make sure we actually need this fix (with the above assumptions, it looks like we don't), and b) if we need this fix, if this is the best place to fix it.

[1] https://github.com/escapewindow/scriptworker/blob/ecdsa/scriptworker/cot/verify.py#L589-L649

Comment 6

[Johan Lorenzo [:jlorenzo]]

(In reply to Aki Sasaki [:aki] from comment #5)
> Is that incorrect? I believe I'm keeping optional artifact support, but
> making a chain of trust artifact mandatory.

That's correct. I now remember why I made chainOfTrust.json.asc optional, back then. There are some cases where this artifact isn't uploaded. For instance: the task gets cancelled before it even runs. This might happen in the (unlikely) case where we surely don't want to update the strings by killing the task off. This has never happened over the past year and a half. I see some other ways to achieve the same goal[1].

Therefore, let's make chainOfTrust.json.sig mandatory :)

[1] By, for instance, letting stores-l10n bet down while the task runs.

Comment 7

[Aki Sasaki (not active)]

Attachment: ecdsa github PR

Comment 8

[Aki Sasaki (not active)]

Attachment: ed25519 github PR

Comment 9

[Aki Sasaki (not active)]

Attachment: puppet pr