Closed Bug 1356952 Opened 7 years ago Closed 7 years ago

Move as much tooltool manifest definitions as possible to taskcluster job definitions

Tracking

(firefox55 fixed)

Status:

RESOLVED FIXED

Milestone:

mozilla55

Tracking Flags:

Tracking

Status

firefox55

---

fixed

People

(Reporter: glandium, Assigned: glandium)

References

Details

Attachments

(5 files, 2 obsolete files)

Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions. 7 years ago Mike Hommey [:glandium] 59 bytes, text/x-review-board-request	mshal : review+	Details
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds. 7 years ago Mike Hommey [:glandium] 59 bytes, text/x-review-board-request	mshal : review+	Details
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds. 7 years ago Mike Hommey [:glandium] 59 bytes, text/x-review-board-request	mshal : review+	Details
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions. 7 years ago Mike Hommey [:glandium] 59 bytes, text/x-review-board-request	mshal : review+	Details
Validation script 7 years ago Mike Hommey [:glandium] 1.61 KB, text/plain		Details
Validation script 7 years ago Mike Hommey [:glandium] 2.17 KB, text/plain		Details
Validation script 7 years ago Mike Hommey [:glandium] 2.26 KB, text/x-python		Details

Mike Hommey [:glandium]

Assignee

Description

•

7 years ago

There are two main reasons to want this:
- Jobs (like toolchain builds) can be optimized out and not run even when the tooltool manifest they use changes, because taskcluster can't know about them.
- The mach artifact toolchain additions from bug 1356529 want to find the tooltool manifest easily and currently has to find the info in mozharness configuration, which only really works for mozharness-based jobs, which is not all of them.

Mike Hommey [:glandium]

Assignee

Updated

•

7 years ago

Assignee: nobody → mh+mozilla

Comment hidden (mozreview-request)

Since bug 1321847, mozharness tooltool manifests can be overriden from
the environment. We use that possibility to now define tooltool
manifests from taskcluster job definitions. Ideally, we'd also remove
the definitions from the mozharness configs, but with things still
running on buildbot, it's not clear what things might break because of
that. We'll do it in a separately back-out-able followup.

Review commit: https://reviewboard.mozilla.org/r/146400/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146400/

Comment hidden (mozreview-request)

Hazard jobs use a specific tooltool-manifest field in their definition.
Since there is no post-processing happening on those definitions, and
since generalizing it would require adding the field to a bunch of
validation schemas, and the same code to various transforms, it's just
simpler to move to use environment variable definitions here too.

Eventually, tooltool manifests won't be necessary anyways, and those
environment variables will go away.

Review commit: https://reviewboard.mozilla.org/r/146402/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146402/

Comment hidden (mozreview-request)

Like hazard jobs.

Review commit: https://reviewboard.mozilla.org/r/146404/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146404/

Comment hidden (mozreview-request)

The toolchain build scripts are currently defining the tooltool
manifest they use on their own. We move the definitions to the
taskcluster job definitions to normalize on everything using that.

Review commit: https://reviewboard.mozilla.org/r/146406/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146406/

Mike Hommey [:glandium]

Assignee

Comment 5

•

7 years ago

Attached file Validation script (obsolete) — Details

FWIW, this is the script I used to validate that the manifests taken out of mozharness do match (except for the base-toolchains jobs, because that was exactly why they had their tooltool manifest defined in taskcluster definitions in the first place).

It requires bug 1356958 to be applied first to work for the Windows jobs.

(The script assumes it's in $topsrcdir, and runs with the virtualenv python)

Mike Hommey [:glandium]

Assignee

Updated

•

7 years ago

Attachment #8875049 - Attachment mime type: text/x-python → text/plain

Mike Hommey [:glandium]

Assignee

Comment 6

•

7 years ago

Attached file Validation script (obsolete) — Details

I had forgotten to account for variants.

Attachment #8875049 - Attachment is obsolete: true

Comment hidden (mozreview-request)

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146400/diff/1-2/

Comment hidden (mozreview-request)

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146402/diff/1-2/

Comment hidden (mozreview-request)

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146404/diff/1-2/

Comment hidden (mozreview-request)

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146406/diff/1-2/

Mike Hommey [:glandium]

Assignee

Updated

•

7 years ago

Attachment #8875054 - Attachment mime type: text/x-python → text/plain

Comment hidden (mozreview-request)

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146400/diff/2-3/

Comment hidden (mozreview-request)

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146402/diff/2-3/

Comment hidden (mozreview-request)

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146404/diff/2-3/

Comment hidden (mozreview-request)

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146406/diff/2-3/

Mike Hommey [:glandium]

Assignee

Comment 15

•

7 years ago

Attached file Validation script — Details

And another adjustment for variants.

Attachment #8875054 - Attachment is obsolete: true

Comment hidden (mozreview-request)

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146400/diff/3-4/

Comment hidden (mozreview-request)

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146402/diff/3-4/

Comment hidden (mozreview-request)

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146404/diff/3-4/

Comment hidden (mozreview-request)

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146406/diff/3-4/

Mike Hommey [:glandium]

Assignee

Updated

•

7 years ago

Depends on: 1370815

Michael Shal [:mshal]

Comment 20

•

7 years ago

mozreview-review

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

https://reviewboard.mozilla.org/r/146400/#review150892

These all look fine to me. Thanks for providing the validation script!

Attachment #8875039 - Flags: review?(mshal) → review+

Michael Shal [:mshal]

Comment 21

•

7 years ago

mozreview-review

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

https://reviewboard.mozilla.org/r/146402/#review150894

Attachment #8875040 - Flags: review?(mshal) → review+

Michael Shal [:mshal]

Comment 22

•

7 years ago

mozreview-review

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

https://reviewboard.mozilla.org/r/146404/#review150896

Attachment #8875041 - Flags: review?(mshal) → review+

Michael Shal [:mshal]

Comment 23

•

7 years ago

mozreview-review

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

https://reviewboard.mozilla.org/r/146406/#review150898

Attachment #8875042 - Flags: review?(mshal) → review+

Comment hidden (mozreview-request)

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146400/diff/4-5/

Comment hidden (mozreview-request)

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146402/diff/4-5/

Comment hidden (mozreview-request)

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146404/diff/4-5/

Comment hidden (mozreview-request)

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146406/diff/4-5/

Comment hidden (mozreview-request)

Comment on attachment 8875039 [details]
Bug 1356952 - Add tooltool manifests defined in mozharness configurations to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146400/diff/5-6/

Comment hidden (mozreview-request)

Comment on attachment 8875040 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in hazard builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146402/diff/5-6/

Comment hidden (mozreview-request)

Comment on attachment 8875041 [details]
Bug 1356952 - Use environment variable to define tooltool manifest in spidermonkey builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146404/diff/5-6/

Comment hidden (mozreview-request)

Comment on attachment 8875042 [details]
Bug 1356952 - Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146406/diff/5-6/

Pulsebot

Comment 32

•

7 years ago

Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/bde62d05afec
Add tooltool manifests defined in mozharness configurations to taskcluster job definitions. r=mshal
https://hg.mozilla.org/integration/autoland/rev/0376305933bf
Use environment variable to define tooltool manifest in hazard builds. r=mshal
https://hg.mozilla.org/integration/autoland/rev/3d6e3b030c4f
Use environment variable to define tooltool manifest in spidermonkey builds. r=mshal
https://hg.mozilla.org/integration/autoland/rev/29d01be1fca2
Move tooltool manifests defined in toolchain build scripts to taskcluster job definitions. r=mshal

Mike Hommey [:glandium]

Assignee

Updated

•

7 years ago

Blocks: 1371093

Carsten Book [:Tomcat]

Comment 33

•

7 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/bde62d05afec
https://hg.mozilla.org/mozilla-central/rev/0376305933bf
https://hg.mozilla.org/mozilla-central/rev/3d6e3b030c4f
https://hg.mozilla.org/mozilla-central/rev/29d01be1fca2

Status: NEW → RESOLVED

Closed: 7 years ago

status-firefox55: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → mozilla55

BMO Automation

Updated

•

6 years ago

Product: Core → Firefox Build System

You need to log in before you can comment on or make changes to this bug.