1453273 - taskgraph: Make signing tasks support esr60

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Johan Lorenzo [:jlorenzo]]

Generating a taskgraph with esr60 configs[1] leads to some discrepancies. The things I've noticed are:

* checksums-signing-ach-linux-nightly/opt, release-generate-checksums-firefox-signing, release-source-signing-linux54-source, repackage-signing-ach-linux-nightly/opt =>
  * Bad scope "project:releng:signing:cert:dep-signing".
  * Bad worker-type "depsigning"


There are probably other issues I didn't see.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1434889#c4

Comment 1

[Johan Lorenzo [:jlorenzo]]

Hello Aki! 60.0esr goes to build on April 30th. Would you feel comfortable looking into this bug by then?

Comment 2

[Aki Sasaki (not active)]

On my radar. If partner repacks go long, this may be fodder for someone else to tackle.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Working on similar stuff in bug 1453255 so I can take this along.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: in-tree signing scope tweaks for esr60

Double-checked for all signing jobs, over all promote/push/ship:
* build-signing
* checksums-signing
* nightly-l10n-signing
* partials-signing 
* release-beetmover-signed-langpacks
* release-generate-checksums-signing
* release-sign-and-push-langpacks (are mozilla-esr60 langpacks listed publicly)
* release-source-signing
* repackage-signing

A subset of the hunks already landed on jamun here[1]. I'm going to land the rest tomorrow before we start a staging release.
For future use of this (next esr bump or w\e), I'm stating the patch here just so we have a good record of what changed overall for signing. 

[1]: https://hg.mozilla.org/projects/jamun/diff/f200dea1d257/taskcluster/taskgraph/util/scriptworker.py

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

00:58:19 <mtabara> are mozilla-esr60 langpacks supposed to be listed publicly?
00:58:39 <aki> c.allek knows best, but i'd guess probably not until esr60 is live
01:01:27 <mtabara> sounds reasonable to me, thanks

@Callek: can you please confirm we don't need to add (yet) esr60 here[1]

[1]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/ci/release-sign-and-push-langpacks/kind.yml#l34

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

The above patch is what we need to make sure esr60 works as expected (the actual repo, not jamun).
As for jamun, out of the list of *signing* jobs above, there's one that doesn't have the scopes/get-worker_type_by_scope, hence they are hardcoded. 

Out of the signing types above, that's: 
* `partials-signing `[1]

Easy fix is to:
* hardcode the below lines with the dev worker and make sure we don't uplift that to beta/esr when we're done

Correct fix is to:
* rewrite all that logic to better handle the scopes via scopes/get-worker_type_by_scope

[1]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/taskgraph/transforms/partials_signing.py#l86

Comment 7

[Justin Wood (:Callek)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #5)
> 00:58:19 <mtabara> are mozilla-esr60 langpacks supposed to be listed
> publicly?
> 00:58:39 <aki> c.allek knows best, but i'd guess probably not until esr60 is
> live
> 01:01:27 <mtabara> sounds reasonable to me, thanks
> 
> @Callek: can you please confirm we don't need to add (yet) esr60 here[1]
> 
> [1]:
> https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/ci/release-
> sign-and-push-langpacks/kind.yml#l34

Langpacks for esr60 *must* be submitted as "unlisted"

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Justin Wood (:Callek) from comment #7)
> (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #5)
> > 00:58:19 <mtabara> are mozilla-esr60 langpacks supposed to be listed
> > publicly?
> > 00:58:39 <aki> c.allek knows best, but i'd guess probably not until esr60 is
> > live
> > 01:01:27 <mtabara> sounds reasonable to me, thanks
> > 
> > @Callek: can you please confirm we don't need to add (yet) esr60 here[1]
> > 
> > [1]:
> > https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/ci/release-
> > sign-and-push-langpacks/kind.yml#l34
> 
> Langpacks for esr60 *must* be submitted as "unlisted"

Cool, good to know. Nothing to do there then. Thanks!

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Dropping more lines here for posterity:

01:41:47 <Callek> aki: mtabara|afk: I answered in bug, but even once esr60 is live, we can't list langpacks in amo as "listed" and must stay "unlisted"
01:41:58 <aki> makes sense
01:42:27 <Callek> that is no difference on AMO side vs today, (with esr52) and is primarily because "most recent langpack is what users are presented with while browsing" which if min/max gecko versions don't match then those users will get a "this addon is not compatible with your version" type of error message
01:42:49 <Callek> though they could find a compat version in "Other Versions" its just not worth it for ESR with todays AMO

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Landed rest of the hunks in https://hg.mozilla.org/projects/jamun/rev/cf2e01720435b19d7fb1320c48a91728b39bddaa

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Pushed the temp hack to unblock staging releases until I add the proper fixes: 
https://hg.mozilla.org/projects/jamun/rev/c4b233b286a443549cbf88820b3dd9c93de70ef6

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: in-tree {signing,beetmover,balrog} worker-type tweaks for esr60

We have a bunch of tasks across {signing,beetmover,balrog} that have had the worker-type hardcoded. This is to address this in one single patch.

Signing (current bug)
* partials-signing

Beetmover (bug 1453255)
* beetmover-checksums
* beetmover-repackage
* beetmover-source (which actually uses beetmover kind to perform this)

Balrog (bug 1453253)
* balrog

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 8969709 [details] [diff] [review]
in-tree {signing,beetmover,balrog} worker-type tweaks for esr60

Thanks! I think we landed at least some of this patch on mozilla-beta yesterday.

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

https://hg.mozilla.org/projects/jamun/rev/185083f7f079

Comment 15

[Justin Wood (:Callek)]

mihai, do you have further state here, what is required to land to central/beta?

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

Status update: 

* for signing, fixes are part of the following patches that landed on jamun, in this order:
https://hg.mozilla.org/projects/jamun/rev/cf2e01720435
https://hg.mozilla.org/projects/jamun/rev/c4b233b286a4
https://hg.mozilla.org/projects/jamun/rev/8082db5de26e
https://hg.mozilla.org/projects/jamun/rev/185083f7f079

I already checked the esr60 graphs with these patches produce the right bits.
Leftover: test a staging esr60 release on jamun to make sure all the signing jobs are using the dep-workers. 

After that, I'll graft these to central and beta.

@Callek: if you can help me with the staging release test today (if not already), I'll take care of the grafting by EOD.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1527cfbcd067
1453255,1453253 - add esr60 fixes in automation. r=aki

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #16)
> Status update: 
> 
> * for signing, fixes are part of the following patches that landed on jamun,
> in this order:
> https://hg.mozilla.org/projects/jamun/rev/cf2e01720435
> https://hg.mozilla.org/projects/jamun/rev/c4b233b286a4
> https://hg.mozilla.org/projects/jamun/rev/8082db5de26e
> https://hg.mozilla.org/projects/jamun/rev/185083f7f079
> 
> I already checked the esr60 graphs with these patches produce the right bits.
> Leftover: test a staging esr60 release on jamun to make sure all the signing
> jobs are using the dep-workers. 
> 
> After that, I'll graft these to central and beta.
> 
> @Callek: if you can help me with the staging release test today (if not
> already), I'll take care of the grafting by EOD.


I landed this on inbound. Some of the stuff was already there from other grafts I suppose.
https://hg.mozilla.org/integration/mozilla-inbound/rev/1527cfbcd067b594afc71fc2a7673f5bffed1a94

Letting this ride the sheriffingh to central + merge tomorrow to get to beta and eventually to esr60.

Comment 19

[Cristina Coroiu [:ccoroiu]]

Backed out changeset 1527cfbcd067 (bug 1453273) for failing Geko decision task on a CLOSED TREE

Backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/2db2fd52dd668ee4f168a716f21755c61b8966c6

Failure push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=1527cfbcd067b594afc71fc2a7673f5bffed1a94

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=175569942&repo=mozilla-inbound&lineNumber=1699

[task 2018-04-25T18:14:56.909Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/release_notifications.py", line 39, in add_notifications
[task 2018-04-25T18:14:56.909Z]     for job in jobs:
[task 2018-04-25T18:14:56.910Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/beetmover_cdns.py", line 100, in make_beetmover_cdns_worker
[task 2018-04-25T18:14:56.910Z]     for job in jobs:
[task 2018-04-25T18:14:56.910Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/beetmover_cdns.py", line 62, in make_beetmover_cdns_description
[task 2018-04-25T18:14:56.910Z]     for job in jobs:
[task 2018-04-25T18:14:56.910Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/beetmover_cdns.py", line 56, in validate
[task 2018-04-25T18:14:56.910Z]     "In cdns-signing ({!r} kind) task for {!r}:".format(config.kind, label))
[task 2018-04-25T18:14:56.910Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 31, in validate_schema
[task 2018-04-25T18:14:56.911Z]     raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
[task 2018-04-25T18:14:56.911Z] Exception: In cdns-signing (u'beetmover-cdns' kind) task for 'firefox-push-to-cdns':
[task 2018-04-25T18:14:56.911Z] extra keys not allowed @ data['worker-type']
[task 2018-04-25T18:14:56.911Z] {u'dependencies': {u'release-generate-checksums-firefox-beetmover': u'release-generate-checksums-firefox-beetmover'},
[task 2018-04-25T18:14:56.911Z]  u'job-from': u'kind.yml',
[task 2018-04-25T18:14:56.911Z]  'name': 'firefox-push-to-cdns',
[task 2018-04-25T18:14:56.911Z]  'product': 'firefox',
[task 2018-04-25T18:14:56.911Z]  'run-on-projects': [],
[task 2018-04-25T18:14:56.912Z]  'shipping-phase': 'push',
[task 2018-04-25T18:14:56.912Z]  'shipping-product': 'firefox',
[task 2018-04-25T18:14:56.912Z]  'treeherder-platform': 'Linux64/opt',
[task 2018-04-25T18:14:56.912Z]  'worker-type': {'by-project': {'default': 'scriptworker-prov-v1/beetmoverworker-dev',
[task 2018-04-25T18:14:56.912Z]                                 'mozilla-beta': 'scriptworker-prov-v1/beetmoverworker-v1',
[task 2018-04-25T18:14:56.912Z]                                 'mozilla-esr60': 'scriptworker-prov-v1/beetmoverworker-v1',
[task 2018-04-25T18:14:56.912Z]                                 'mozilla-release': 'scriptworker-prov-v1/beetmoverworker-v1'}}}
[taskcluster 2018-04-25 18:14:57.418Z] === Task Finished ===

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/db04d03e8418
1453255,1453253 - add esr60 fixes in automation. r=aki CLOSED TREE

Comment 21

[Mihai Tabara [:mtabara]⌚️GMT]

Tried again after backout https://hg.mozilla.org/integration/mozilla-inbound/rev/db04d03e84185756c423bd40141fa45e7004d05aa

Comment 22

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/db04d03e8418

Comment 23

[Mihai Tabara [:mtabara]⌚️GMT]

We failed to graft this patch in time on beta for esr60 mergeduty. 
We'll wait until the merge is completed and then we'll land this patch to release & esr60.
Work for this is tracked in bug 1457090.

Comment 24

[Johan Lorenzo [:jlorenzo]]

Comment 22 was grafted to mozilla-esr60 as https://hg.mozilla.org/releases/mozilla-esr60/rev/28107b91f58a

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/42223172397c