1526419 - easy button to test autograph-stage mar signing

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Description

[Aki Sasaki (not active)]

Our current dep, nightly, and release mar configs all hit autograph-prod. This is by design: we don't want the availability or configuration of autograph-stage to cause tree closures.

However, we'd like a way to test autograph config or code changes before they hit autograph-prod. Our current solution involves a lot of manual steps by Ben or me, or waiting until the changes hit prod and rerunning a nightly mar signing task.

Let's add an easy button: a task that exists for add task, but doesn't run without an explicit request. This will hit the depsigning scriptworkers with a special autograph-stage signing format. This task should also verify the signature matches the autograph-stage mar signing key.

• [X] add the autograph-stage mar pubkey in-tree
• [ ] add the autograph-stage mar pubkey in mardor, to allow for signature verification (this PR)
• [X] add the autograph-stage creds to the dep signing passwords file
• [ ] add the capability to verify mar signatures, via mardor, in signingscript. This could be optional or on by default.
• [ ] add the scopes for the new nightly task in ci-configuration
• [ ] add a nightly task [that doesn't run by default] that signs a mar through depsigning scriptworkers->autograph-stage, and verifies it through mardor, and turns green if successful
  • [ ] the plan is for anyone with the right scopes to be able to add task this task against the most recent nightly graph, and check its status.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Possible dupe bug 1510778?

Comment 2

[Aki Sasaki (not active)]

Aha! https://bugzilla.mozilla.org/show_bug.cgi?id=1510778#c1 is where I documented how to do testing on signing-linux-dev1. I might dup that bug forward to this one, since this one has an actionable plan.

Comment 4

[Aki Sasaki (not active)]

Attachment: Bug 1526419 - add autograph_stage.der mar signing pubkey. r=catlee

Comment 5

[Aki Sasaki (not active)]

Attachment: Bug 1526419 - add autograph_stage.pem mar signing pubkey. r=catlee

Comment 6

[Aki Sasaki (not active)]

Attachment: puppet - add autograph_stage_mar384 format

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/91ac960f7b59
add autograph_stage.pem mar signing pubkey. r=catlee

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/91ac960f7b59

Comment 9

[Aki Sasaki (not active)]

https://github.com/mozilla-releng/build-puppet/commit/28162e6163098ff0c75c7601012d1620526e78a3
https://github.com/mozilla-releng/build-puppet/commit/947934e4f34f1d3de7fe8bae5970b2f5040e6efc
https://github.com/mozilla/build-mar/pull/50

Comment 10

[Aki Sasaki (not active)]

Attachment: mardor - update sha384 revision

Comment 11

[Aki Sasaki (not active)]

signingscript wip

Comment 12

[Aki Sasaki (not active)]

It looks like I got that to work in this try push: log

Comment 13

[Aki Sasaki (not active)]

I still need to fix up the signingscript PR (test coverage, fix tests) and write the in-tree task. We also probably need a mardor release.

We'll need to get the keyids in-tree and deal with that in signingscript+mardor for key rotation, but we can leave that for a followup bug :)

Comment 14

[Aki Sasaki (not active)]

Attachment: signingscript - verify mar signatures

Comment 15

[Intermittent Failures Robot]

13 failures in 2832 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-central: 2

Platform breakdown:

• windows2012-32: 3
• windows2012-64: 4
• linux32: 3
• linux64-noopt: 1
• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1526419&startday=2019-02-11&endday=2019-02-17&tree=all

Comment 16

[Aki Sasaki (not active)]

Attachment: bump mardor+signingscript in puppet

Comment 17

[Aki Sasaki (not active)]

in-tree task wip

Comment 18

[Aki Sasaki (not active)]

Attachment: Bug 1526419 - add mar-signing-autograph-stage task r=tomprince

We use autograph-prod for our ci, nightly, and release signing. Autograph-stage doesn't have the same guarantees for availability, so pointing, say, dep-signing at autograph-stage would have resulted in occasional tree closures whenever autograph-stage changes configuration or is down.

However, we also want a way to verify autograph-stage is still valid, after the autograph team makes changes. This task is meant to be add-task'ed; a green result means autograph-stage has signed the mar file correctly.

Comment 19

[Aki Sasaki (not active)]

My test try push is here.

The first signing task failed because I was on scriptworker 20.0.0 without the ci-configuration patch in 20.0.1.
The first mar-signing-autograph-stage-linux64-nightly/opt task failed because it looks like the queue or artifact mirror didn't have the artifacts ready to download until the 2nd run.

All we need to do to test autograph-stage is to add the ms-stage(N) task, aka mar-signing-autograph-stage-linux64-nightly/opt . It will go green if autograph-stage's mar signing provides a valid mar signature.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/811caa480654
add mar-signing-autograph-stage task r=Callek

Comment 21

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/811caa480654

Comment 22

[Aki Sasaki (not active)]

Attachment: relengdocs PR

Comment 23

[Aki Sasaki (not active)]

We have stage and prod testing docs at https://moz-releng-docs.readthedocs.io/en/latest/procedures/Testing_Autograph.html now.