1356952 - Move as much tooltool manifest definitions as possible to taskcluster job definitions

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Mike Hommey [:glandium]]

There are two main reasons to want this:
- Jobs (like toolchain builds) can be optimized out and not run even when the tooltool manifest they use changes, because taskcluster can't know about them.
- The mach artifact toolchain additions from bug 1356529 want to find the tooltool manifest easily and currently has to find the info in mozharness configuration, which only really works for mozharness-based jobs, which is not all of them.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Since bug 1321847, mozharness tooltool manifests can be overriden from
the environment. We use that possibility to now define tooltool
manifests from taskcluster job definitions. Ideally, we'd also remove
the definitions from the mozharness configs, but with things still
running on buildbot, it's not clear what things might break because of
that. We'll do it in a separately back-out-able followup.

Review commit: https://reviewboard.mozilla.org/r/146400/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146400/

Comment 2

[Mike Hommey [:glandium]]

Attachment: Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Hazard jobs use a specific tooltool-manifest field in their definition.
Since there is no post-processing happening on those definitions, and
since generalizing it would require adding the field to a bunch of
validation schemas, and the same code to various transforms, it's just
simpler to move to use environment variable definitions here too.

Eventually, tooltool manifests won't be necessary anyways, and those
environment variables will go away.

Review commit: https://reviewboard.mozilla.org/r/146402/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146402/

Comment 3

[Mike Hommey [:glandium]]

Attachment: Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Like hazard jobs.

Review commit: https://reviewboard.mozilla.org/r/146404/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146404/

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

The toolchain build scripts are currently defining the tooltool
manifest they use on their own. We move the definitions to the
taskcluster job definitions to normalize on everything using that.

Review commit: https://reviewboard.mozilla.org/r/146406/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146406/

Comment 5

[Mike Hommey [:glandium]]

Attachment: Validation script

FWIW, this is the script I used to validate that the manifests taken out of mozharness do match (except for the base-toolchains jobs, because that was exactly why they had their tooltool manifest defined in taskcluster definitions in the first place).

It requires bug 1356958 to be applied first to work for the Windows jobs.

(The script assumes it's in $topsrcdir, and runs with the virtualenv python)

Comment 6

[Mike Hommey [:glandium]]

Attachment: Validation script

I had forgotten to account for variants.

Comment 7

[Mike Hommey [:glandium]]

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146400/diff/1-2/

Comment 8

[Mike Hommey [:glandium]]

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146402/diff/1-2/

Comment 9

[Mike Hommey [:glandium]]

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146404/diff/1-2/

Comment 10

[Mike Hommey [:glandium]]

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146406/diff/1-2/

Comment 11

[Mike Hommey [:glandium]]

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146400/diff/2-3/

Comment 12

[Mike Hommey [:glandium]]

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146402/diff/2-3/

Comment 13

[Mike Hommey [:glandium]]

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146404/diff/2-3/

Comment 14

[Mike Hommey [:glandium]]

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146406/diff/2-3/

Comment 15

[Mike Hommey [:glandium]]

Attachment: Validation script

And another adjustment for variants.

Comment 16

[Mike Hommey [:glandium]]

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146400/diff/3-4/

Comment 17

[Mike Hommey [:glandium]]

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146402/diff/3-4/

Comment 18

[Mike Hommey [:glandium]]

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146404/diff/3-4/

Comment 19

[Mike Hommey [:glandium]]

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146406/diff/3-4/

Comment 20

[Michael Shal [:mshal]]

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

https://reviewboard.mozilla.org/r/146400/#review150892

These all look fine to me. Thanks for providing the validation script!

Comment 21

[Michael Shal [:mshal]]

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

https://reviewboard.mozilla.org/r/146402/#review150894

Comment 22

[Michael Shal [:mshal]]

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

https://reviewboard.mozilla.org/r/146404/#review150896

Comment 23

[Michael Shal [:mshal]]

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

https://reviewboard.mozilla.org/r/146406/#review150898

Comment 24

[Mike Hommey [:glandium]]

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146400/diff/4-5/

Comment 25

[Mike Hommey [:glandium]]

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146402/diff/4-5/

Comment 26

[Mike Hommey [:glandium]]

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146404/diff/4-5/

Comment 27

[Mike Hommey [:glandium]]

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146406/diff/4-5/

Comment 28

[Mike Hommey [:glandium]]

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146400/diff/5-6/

Comment 29

[Mike Hommey [:glandium]]

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146402/diff/5-6/

Comment 30

[Mike Hommey [:glandium]]

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146404/diff/5-6/

Comment 31

[Mike Hommey [:glandium]]

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146406/diff/5-6/

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bde62d05afec
Add tooltool manifests defined in mozharness configurations to taskcluster job definitions. r=mshal
https://hg.mozilla.org/integration/autoland/rev/0376305933bf
Use environment variable to define tooltool manifest in hazard builds. r=mshal
https://hg.mozilla.org/integration/autoland/rev/3d6e3b030c4f
Use environment variable to define tooltool manifest in spidermonkey builds. r=mshal
https://hg.mozilla.org/integration/autoland/rev/29d01be1fca2
Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions. r=mshal

Comment 33

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/bde62d05afec
https://hg.mozilla.org/mozilla-central/rev/0376305933bf
https://hg.mozilla.org/mozilla-central/rev/3d6e3b030c4f
https://hg.mozilla.org/mozilla-central/rev/29d01be1fca2