1535056 - Taskgraph - supply schema for "Parameters"

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Mitchell Hentges [:mhentges] 🦀]

As part of adding git support for taskgraph, we can no longer uniformly define all taskgraph "parameters" as required (for example, pushlog_id isn't relevant for git clones).

This will leverage taskgraph's schema tooling to validate parameters and allow conditionally required values.

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Tom, would you recommend making this change to the extracted taskgraph, then porting it to in-tree, or the other way around?

Comment 2

[Tom Prince [:tomprince]]

My general workflow has to get the version targeted at mozilla-central reviewed and landed first, and then land on ci/taskgraph. At the moment, the mozilla-central copy is critical code, and ci/taskgraph less so, so the review there is more critical, and there are more stakeholders.

For development, I've sometime worked on ci/taskgraph first, but ported to mozilla-central for review, before porting back to ci/taskgraph to land.

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1535056 - Specifies schema for taskgraph parameters

I've tested this with ./mach python-test taskcluster, as well as playing with ./mach taskgraph tasks.
I'm very open to recommendations for further testing or workflow :)

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f5eea3706636
Validate taskgraph parameters using a schema r=tomprince

Comment 5

[Stefan Hindli [:stefan_hindli]]

Backed out 2 changesets (Bug 1535056, Bug 1513854) for linux build bustages and failed Gecko Decision Task CLOSED TREE

https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&selectedJob=244115091

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=244115091&repo=autoland&lineNumber=59

Comment 6

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4b50c89ff7cf
Backed out 2 changesets (bug 1535056, bug 1513854) for linux build bustages and failed Gecko Decision Task CLOSED TREE

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

I had left my patch sitting and it got stale - I should've re-tested it before landing.
I'll re-test and re-land this ticket later (next week, hopefully)

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

The failure was caused by pushlog_id being a basestring, not unicode

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/48687a76aec5
Validate taskgraph parameters using a schema r=tomprince

Comment 10

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/48687a76aec5

Comment 11

[Alexandru Michis [:malexandru]]

Backed out changeset 48687a76aec5 for causing nightly builds to fail.

Backout link: https://hg.mozilla.org/mozilla-central/rev/5cc220ddf028de011a922042ee9ba691b94d055d

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&revision=900ff508219cddc29c6f6307c65f0e90c3422c24&selectedJob=248684134

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248684134&repo=mozilla-central&lineNumber=79

Comment 12

[Mitchell Hentges [:mhentges] 🦀]

Ahh, I tested with taskgraph decision, but I was using an old head-ref that didn't contain recent payload changes.
I'll update, learn how to run against try, and land again

Comment 13

[Mitchell Hentges [:mhentges] 🦀]

Associated try has passed the decision task

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/50beca34e939
Validate taskgraph parameters using a schema r=tomprince

Comment 15

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/50beca34e939

Comment 16

[Noemi Erli[:noemi_erli]]

Backed out changeset 50beca34e939 (Bug 1535056) for causing cron tasks bustages a=backout

Push with issues: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&selectedJob=248921207&revision=3c26311b3d8df20c4c0b00eda34b932df121b65c&searchStr=gecko%2Cdecision%2Ctask%2Copt

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248921207&repo=mozilla-central&lineNumber=48

Backout: https://hg.mozilla.org/mozilla-central/rev/8d86cc081fe5d16980a0a610af6b967cc03bf814

Comment 17

[Mitchell Hentges [:mhentges] 🦀]

I've resolved the specific error (--optimize-target-tasks was being parsed to an int, not a bool).
I'll land again (after checking with Tom that my solution is OK), and I'll continue to iterate if it fails

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f0f79e047a47
Specifies schema for taskgraph parameters r=tomprince

Comment 19

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/f0f79e047a47