Port Bug 1501878 - create a second promote phase graph for deferred mar signing

RESOLVED FIXED in Thunderbird 65.0

Status

enhancement
--
critical
RESOLVED FIXED
7 months ago
5 months ago

People

(Reporter: rjl, Assigned: rjl)

Tracking

Thunderbird 65.0
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

Assignee

Description

7 months ago
This is for the Comm port.


+++ This bug was initially created as a clone of Bug #1501878 +++

Currently, this looks like:

- in the partial kind, download target.complete.mar from the repackage kinds instead of repackage-signing kinds.

  - (we don't require that the complete mar is signed to create a partial, and
     if we depend on the signed complete mar, and defer mar signing until signoff,
     we won't generate partials until after signoff.)

  - this was complicated by single_dep dependency crawling. I believe resolving
    bug 1471202 has removed this complication.

- split the repackage-signing into repackage-signing (installer signing) and complete-mar-signing kinds. This will require downstream beetmover and balrog changes.

- allow for another release promotion flavor for the promote graph without mar signing.

for q4. In the future, we'll probably want to:

- add support in ship-it v2 to support the new release promotion flavor
- add a shipitscript task to generate and submit a manifest of unsigned mar files
- add support in ship-it v2 to sign off on the manifest
- add support in signingscript to send the url of the signoffs to autograph along with the signing request
- add support in autograph to require and verify the signoffs for some set of mar signing

I'll probably spawn this second set of tasks out into other bugs and focus on the first set in this one.

Updated

7 months ago
Duplicate of this bug: 1508057
Assignee

Updated

7 months ago
Assignee: nobody → rob
Status: NEW → ASSIGNED

Comment 2

7 months ago
From bug 1508057 - There's also a WIP patch in attachment 9025883 [details] [diff] [review].

===

Running the magic |mach taskgraph ...| locally I get:

ParameterMismatch: missing parameters: signoff_urls, required_signoffs

Grepping through M-C's taskcluster/ci there is no signoff_urls anywhere, and only three mentions of required_signoffs in mar-signing* and partials-signing. I've included the mar-signing* in the patch.

Updated

7 months ago
Blocks: 1501878

Comment 3

7 months ago
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/46fb423d7ced
Port bug 1501878 [create a second promote phase graph for deferred mar signing]. rs=bustage-fix CLOSED TREE
Status: ASSIGNED → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED

Comment 4

7 months ago
I've landed the patch from bug 1508057, attachment 9025883 [details] [diff] [review], which is pretty much what Rob came up with independently here:
https://hg.mozilla.org/try-comm-central/rev/6e5c75b9c379974adf5b45bd2b16f07a59aa5d35
(which also contains some undesired hunks in taskcluster/ci/release-balrog-*).

All that said, the local magic:
mach taskgraph full --root comm/taskcluster/ci -p https://index.taskcluster.net/v1/task/comm.v2.comm-central.latest.taskgraph.decision/artifacts/public/parameters.yml
failed until a few minutes ago with
  ParameterMismatch: missing parameters: signoff_urls, required_signoffs
Now it works again. Who understands this stuff :-( - I guess Rob, since he said on IRC:

11:31:58 - rjl-: but the reason it didn't work actually had nothing to do with your patch.. it because those signoff_urls and required_signoffs actually have to be added to parameters.yml
11:32:21 - rjl-: which is automagic if a job runs normally, but not so much with the mach taskgraph thing
Assignee

Comment 5

7 months ago
Error log from a mar-signing task on comm-central.

I suspect there is a misconfiguration in the tb-signing-v1 workers, possibly in the passwords.json file or else in some Taskcluster secrets that I don't have access to.
Will check in with the Taskcluster team.
So should this bug be open until the complete fix is done?

Comment 7

7 months ago
Well, we see the problem on the tree each day, so we won't forget. If no further patches for M-C or C-C are required, it may as well stay resolved since the porting is done as per the summary. In the end, it doesn't matter. Reopen it if you please.
Maybe best to file a separate issue then. Just wanted it to be tracked somewhere
Assignee

Comment 9

7 months ago
Per Tom, work needs to happen around https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/mar_signing.py#134 to add support to prefix format:autograph_hash_only_mar384 with the correct project:comm:thunderbird:releng prefix.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Is this the bug that's preventing nightlies from getting generated? No nightlies since 2018-11-21
Severity: major → critical
Assignee

Comment 11

7 months ago
The deferred mar signing feature in Bug 1501878 has a taskcluster scope
value in mar_signing.py hardcoded to something Firefox specific. This
patch introduces a new function, get_autograph_format_scope, that will
produce the right value for Firefox and Thunderbird.

Updated

6 months ago
Blocks: 1512127
Aki, can the patch on this bug be landed? I assume Rob meant to do so, but he's been away for some time, and meanwhile we haven't had a nightly build for three weeks. :-/
Flags: needinfo?(aki)

Comment 13

6 months ago
Yes, please land.
Flags: needinfo?(aki)

Comment 14

6 months ago
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/integration/autoland/rev/09176a21664e
Create function for determining autographing scope. r=aki

Comment 15

6 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/09176a21664e
Status: REOPENED → RESOLVED
Closed: 7 months ago6 months ago
Resolution: --- → FIXED

Comment 16

6 months ago
Aki, Tom, who can approve a beta uplift for https://hg.mozilla.org/integration/autoland/rev/09176a21664e?

This should have landed at the end of November :-(
Flags: needinfo?(mozilla)
Flags: needinfo?(aki)
You need to log in before you can comment on or make changes to this bug.