1313111 - Hook up toolchain tasks to the build and test task graph

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

We have "toolchain" tasks defined in-tree currently:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/toolchain/kind.yml

These can be used to build some of our toolchains, like clang/gcc/binutils, so that developers don't have to build them locally. This is good for making repeatable builds and removing single points of failure on people knowing how to build these things.

We should hook these up to the existing build and test task graph--the task definitions should specify how to build a tool (a script plus an external repository and revision to build), and just like the tasks that build docker images toolchain tasks should be scheduled when those inputs change. We should be able to specify that certain types of tasks have a dependency on certain tools, so that the taskgraph will be generated appropriately. Finally, tasks will need to be provided with the artifacts from the toolchain tasks as inputs somehow. We could bake this into a task definition somewhere, specifying that a certain tool should be unpacked into a certain directory in the worker. Dustin says this last part has been discussed as "pre-filled caches" in the past, but not yet implemented.

This would make the testing cycle for testing new toolchains and other tools much simpler--you'd just change the revision in the task definition or the build script and push to try and you'd get a toolchain job and then build jobs using that toolchain.

I'd like this to support use cases that aren't just the toolchains we have defined now. Specifically I've been testing sccache2 on try (bug 1286934) and it's a hassle to build it for all 3 Tier-1 platforms, put it in tooltool, update the in-tree manifests and then push to try. It'd be much easier if it was defined as a toolchain task and I could just tweak the rev and push to try.

Comment 1

[Mike Hommey [:glandium]]

Sadly, until buildbot jobs are replaced by taskcluster jobs, we'll still have to deal with tooltool :(

Comment 2

[Pete Moore [:pmoore][:pete]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #0)
> Dustin says this last part has been discussed as
> "pre-filled caches" in the past, but not yet implemented.

For windows tasks, this is already implemented in generic worker (the worker we currently run on windows worker types). See the "mounts" section of the generic worker payload here:

https://docs.taskcluster.net/manual/execution/workers/generic-worker

Comment 3

[(not currently active) Ted Mielczarek]

(In reply to Mike Hommey [:glandium] from comment #1)
> Sadly, until buildbot jobs are replaced by taskcluster jobs, we'll still
> have to deal with tooltool :(

Yeah. I don't know what the timeline looks like on that, or if we could figure out something clever to work around this in the meantime.

Comment 4

[Gregory Szorc [:gps]]

Random thought: having the toolchain in the task graph makes it considerably easier for local builds to download and use toolchain artifacts from automation in local builds. One could imagine a configure mode that looks at the VCS commit (like artifact builds), finds toolchains built in automation, and automatically uses them. If you are bisecting locally, the toolchain would "just work." That could be really convenient.

Comment 5

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #4)
> Random thought: having the toolchain in the task graph makes it considerably
> easier for local builds to download and use toolchain artifacts from
> automation in local builds. One could imagine a configure mode that looks at
> the VCS commit (like artifact builds), finds toolchains built in automation,
> and automatically uses them. If you are bisecting locally, the toolchain
> would "just work." That could be really convenient.

You can already do that with tooltool manifests (except for non-public files, but those wouldn't be built on taskcluster anyways).

Comment 6

[Mike Hommey [:glandium]]

Assigning this to myself, since I'm currently actively working on most parts of this.

Comment 7

[Mike Hommey [:glandium]]

Attachment: Migration helper

I just opened a bunch of bugs for individual toolchains. This is a helper patch for the migration, not meant to land.

The way it works is:
- Apply the patch
- Add an entry to KNOWN_TOOLCHAINS, for example `'linux64-sccache': '7bbe53cda3f6'`
- Run the command line from a decision task and add the missing toolchains until it doesn't complain anymore.
- Remove the corresponding tooltool manifest entries.

Comment 8

[Mike Hommey [:glandium]]

Attachment: Migration helper

This is a revised version that takes into account that some toolchain jobs produce an artifact with the same name. That prevented either win32-clang-cl or win64-clang-cl to be found missing.

Comment 9

[(not currently active) Ted Mielczarek]

Despite bug 1356529 still being open this seems pretty fixed. Should we punt additional work to followup bugs?

Comment 10

[Mike Hommey [:glandium]]

I'm keeping this as a tracking bug. The android toolchains and rust are the main remaining things.

Comment 11

[Gregory Szorc [:gps]]

glandium: is this bug serving any useful role these days?

Comment 12

[Mike Hommey [:glandium]]

I guess not, since most things are now hooked up. The remainder is things like msvc or old js shell.