1759030 - Merge generator.py between the two taskgraphs

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - Convert TransformConfig.kind_dependencies_task to a list, r?#releng-reviewers!

While a dict is a better data structure to hold this information (as we can do
quick containment checks), the standalone version uses a list. If we decided to
switch over to a dict there instead, we'd be introducing a new backwards
incompatible change and need to make a new major version of Taskgraph.

So for now it's best to make 'gecko_taskgraph' follow standalone's convention.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - Refactor util/verify.py to support non-task verifications, r?#releng-reviewers!

Currently all verifications loop over tasks in a taskgraph. Though there are
some miscellaneous verifications in generator.py for things like parameters and
attributes. These verifications are currently specific to Gecko and cannot be
ported to other repositories.

This patch refactors the verifications logic to enable adding other arbitrary
verifications that don't use the same parameter values that a normal task
verification does. This will allow us to move the Gecko specific verifications
out of 'generator.py', which will allow us to use the standalone taskgraph
version of it.

Depends on D142022

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - Move TaskGraphGenerator.verify_* to a verifications in util/verify.py, r?#releng-reviewers!

Standalone taskgraph doesn't have a 'verify_parameters' function as we can't
assume that parameters are documented in every consumer repo. By moving these
to proper verifications in util/verify.py, each consumer repo in standalone may
optional implement something for them or not (as they see fit).

Depends on D142023

Comment 4

[Phabricator Automation]

Comment on attachment 9269330 [details]
Bug 1759030 - Refactor util/verify.py to support non-task verifications, r?#releng-reviewers!

Revision D142023 was moved to bug 1748929. Setting attachment 9269330 [details] to obsolete.

Comment 5

[Phabricator Automation]

Comment on attachment 9269331 [details]
Bug 1759030 - Move TaskGraphGenerator.verify_* to a verifications in util/verify.py, r?#releng-reviewers!

Revision D142024 was moved to bug 1748929. Setting attachment 9269331 [details] to obsolete.

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - Use target_task method registration from standalone taskgraph, r?#taskgraph-reviewers

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - Use 'filter_tasks' from standalone taskgraph, r?#taskgraph-reviewers

Depends on D152211

Comment 8

[Phabricator Automation]

Comment on attachment 9286112 [details]
Bug 1759030 - Use target_task method registration from standalone taskgraph, r?#taskgraph-reviewers

Revision D152211 was moved to bug 1780273. Setting attachment 9286112 [details] to obsolete.

Comment 9

[Phabricator Automation]

Comment on attachment 9286113 [details]
Bug 1759030 - Use 'filter_tasks' from standalone taskgraph, r?#taskgraph-reviewers

Revision D152212 was moved to bug 1780273. Setting attachment 9286113 [details] to obsolete.

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - Use 'load_graph_config' from vendored taskgraph, r?#releng-reviewers

This function is called by generator.py, so we need to replace this before we
can replace that.

Comment 11

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - Use 'taskgraph.register_morph' in gecko_taskgraph, r?#releng-reviewers

Depends on D156697

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0c04847a9e46
Use 'load_graph_config' from vendored taskgraph, r=releng-reviewers,jlorenzo

Comment 13

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/0c04847a9e46

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9695f66aa0fe
Use 'taskgraph.register_morph' in gecko_taskgraph, r=releng-reviewers,gbrown

Comment 15

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/9695f66aa0fe

Comment 16

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - [gecko_taskgraph] Move 'always_target' implementation outside of generator.py, r?#releng-reviewers

The always_target attribute is a feature which I believe I previously
mis-implemented. It doesn't belong in generator.py and should instead be
implemented somewhere in target_tasks.py. I chose to wrap the registration
decorator to guarantee backwards compatibility within Gecko. Though we may wish
to move it out to a filter in the future.

I'm making this change now to facilitate merging standalone and Gecko
taskgraphs. The always_target feature will be removed from standalone (as it
isn't being used and should consumers need something like it, they can roll
their own implementations).

With the feature removed from both Gecko and standalone in generator.py, this
file will now be identical across both Taskgraphs and can therefore be removed
from Gecko.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b7380df3b9a8
[gecko_taskgraph] Move 'always_target' implementation outside of generator.py, r=releng-reviewers,gbrown

Comment 18

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/b7380df3b9a8

Comment 20

[Alex Hochheiden [:ahochheiden]]

Attachment: vendor_taskcluster-taskgraph_3_5_0.patch

Patch to vendor taskcluster-taskgraph at 3.5.0 as requested by :ahal.

Comment 21

[Alex Hochheiden [:ahochheiden]]

Attachment: vendor_taskcluster-taskgraph_3_5_1.patch

Patch to vendor taskcluster-taskgraph at 3.5.1 as requested by :ahal.

Comment 22

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - Vendor `taskcluster-taskgraph` at `3.5.1`, r?ahochheiden!

Comment 23

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - [gecko_taskgraph] Use 'generator.py' from upstream Taskgraph, r?#releng-reviewers!

Depends on D161056

Comment 24

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1759030 - Use 'verifications' from upstream taskgraph module, r?#releng-reviewers!

This is the last place that in gecko_taskgraph/generator.py that calls into
gecko_taskgraph code. By removing it, the two generator.py files will be
identical and the gecko_taskgraph copy can be removed.

Depends on D161056

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3f2b2e009c00
Vendor `taskcluster-taskgraph` at `3.5.1`, r=ahochheiden
https://hg.mozilla.org/integration/autoland/rev/9940b1cf6881
Use 'verifications' from upstream taskgraph module, r=releng-reviewers,gbrown
https://hg.mozilla.org/integration/autoland/rev/9c52daf6117c
[gecko_taskgraph] Use 'generator.py' from upstream Taskgraph, r=releng-reviewers,jlorenzo

Comment 26

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/3f2b2e009c00
https://hg.mozilla.org/mozilla-central/rev/9940b1cf6881
https://hg.mozilla.org/mozilla-central/rev/9c52daf6117c

Comment 27

[Serban Stanca [:SerbanS]]

Backed out for causing for causing Bug 1799074.

• Backout link

Comment 28

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/d5fdedfaefcb
Backed out 3 changesets for causing Bug 1799074. CLOSED TREE

Comment 29

[Andrew Halberstadt [:ahal]]

The problem was simply because we were validating against taskgraph.config.graph_config_schema instead of gecko_taskgraph.config.graph_config_schema. This got regressed in this patch due to changing:

- from gecko_taskgraph.generator import load_tasks_for_kind
+ from taskgraph.generator import load_tasks_for_kind

in python/mozbuild/mozbuild/toolchains.py.

Previously the gecko_taskgraph import was triggering this line which made everything work. Now, that import was no longer happening. A simple (somewhat hacky) fix is to add an unused import gecko_taskgraph directly prior.

I have ideas around better support for extending Taskgraph so we don't need to do all this hacky monkeypatching in gecko_taskgraph. But that is a larger project which likely won't be prioritized for awhile, so for now I think the simple fix is the way to go.

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/965dd7af8b21
Vendor `taskcluster-taskgraph` at `3.5.1`, r=ahochheiden
https://hg.mozilla.org/integration/autoland/rev/952c83a44302
Use 'verifications' from upstream taskgraph module, r=releng-reviewers,gbrown
https://hg.mozilla.org/integration/autoland/rev/b9c288fda96d
[gecko_taskgraph] Use 'generator.py' from upstream Taskgraph, r=releng-reviewers,jlorenzo

Comment 31

[Cristian Tuns]

Backout merged to central: https://hg.mozilla.org/mozilla-central/rev/decaca6eacd7

Comment 32

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/965dd7af8b21
https://hg.mozilla.org/mozilla-central/rev/952c83a44302
https://hg.mozilla.org/mozilla-central/rev/b9c288fda96d