Hook up toolchain tasks to the build and test task graph

NEW
Assigned to

Status

Taskcluster
General
a year ago
28 days ago

People

(Reporter: ted, Assigned: glandium)

Tracking

(Depends on: 1 bug, Blocks: 3 bugs)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

a year ago
We have "toolchain" tasks defined in-tree currently:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/toolchain/kind.yml

These can be used to build some of our toolchains, like clang/gcc/binutils, so that developers don't have to build them locally. This is good for making repeatable builds and removing single points of failure on people knowing how to build these things.

We should hook these up to the existing build and test task graph--the task definitions should specify how to build a tool (a script plus an external repository and revision to build), and just like the tasks that build docker images toolchain tasks should be scheduled when those inputs change. We should be able to specify that certain types of tasks have a dependency on certain tools, so that the taskgraph will be generated appropriately. Finally, tasks will need to be provided with the artifacts from the toolchain tasks as inputs somehow. We could bake this into a task definition somewhere, specifying that a certain tool should be unpacked into a certain directory in the worker. Dustin says this last part has been discussed as "pre-filled caches" in the past, but not yet implemented.

This would make the testing cycle for testing new toolchains and other tools much simpler--you'd just change the revision in the task definition or the build script and push to try and you'd get a toolchain job and then build jobs using that toolchain.

I'd like this to support use cases that aren't just the toolchains we have defined now. Specifically I've been testing sccache2 on try (bug 1286934) and it's a hassle to build it for all 3 Tier-1 platforms, put it in tooltool, update the in-tree manifests and then push to try. It'd be much easier if it was defined as a toolchain task and I could just tweak the rev and push to try.
(Assignee)

Comment 1

a year ago
Sadly, until buildbot jobs are replaced by taskcluster jobs, we'll still have to deal with tooltool :(
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #0)
> Dustin says this last part has been discussed as
> "pre-filled caches" in the past, but not yet implemented.

For windows tasks, this is already implemented in generic worker (the worker we currently run on windows worker types). See the "mounts" section of the generic worker payload here:

https://docs.taskcluster.net/manual/execution/workers/generic-worker
(Reporter)

Comment 3

a year ago
(In reply to Mike Hommey [:glandium] from comment #1)
> Sadly, until buildbot jobs are replaced by taskcluster jobs, we'll still
> have to deal with tooltool :(

Yeah. I don't know what the timeline looks like on that, or if we could figure out something clever to work around this in the meantime.
Random thought: having the toolchain in the task graph makes it considerably easier for local builds to download and use toolchain artifacts from automation in local builds. One could imagine a configure mode that looks at the VCS commit (like artifact builds), finds toolchains built in automation, and automatically uses them. If you are bisecting locally, the toolchain would "just work." That could be really convenient.
(Assignee)

Comment 5

a year ago
(In reply to Gregory Szorc [:gps] from comment #4)
> Random thought: having the toolchain in the task graph makes it considerably
> easier for local builds to download and use toolchain artifacts from
> automation in local builds. One could imagine a configure mode that looks at
> the VCS commit (like artifact builds), finds toolchains built in automation,
> and automatically uses them. If you are bisecting locally, the toolchain
> would "just work." That could be really convenient.

You can already do that with tooltool manifests (except for non-public files, but those wouldn't be built on taskcluster anyways).
(Assignee)

Updated

10 months ago
Depends on: 1335651
(Assignee)

Comment 6

10 months ago
Assigning this to myself, since I'm currently actively working on most parts of this.
Assignee: nobody → mh+mozilla
(Assignee)

Updated

9 months ago
Depends on: 1343713
(Assignee)

Updated

9 months ago
Depends on: 1343716
(Assignee)

Updated

9 months ago
Depends on: 1343718
(Assignee)

Updated

8 months ago
Depends on: 1355731
(Assignee)

Updated

7 months ago
Depends on: 1356529
(Assignee)

Updated

7 months ago
Depends on: 1356926
(Assignee)

Updated

7 months ago
Depends on: 1356927
(Assignee)

Updated

7 months ago
Depends on: 1356929
(Assignee)

Updated

7 months ago
Depends on: 1356931
(Assignee)

Updated

7 months ago
Depends on: 1356932
(Assignee)

Updated

7 months ago
Depends on: 1341934
(Assignee)

Updated

7 months ago
Depends on: 1356951
(Assignee)

Updated

7 months ago
Depends on: 1356955
(Assignee)

Updated

7 months ago
Depends on: 1356974
(Assignee)

Updated

7 months ago
Blocks: 1356989
(Assignee)

Updated

7 months ago
Depends on: 1356994
(Assignee)

Updated

5 months ago
Depends on: 1360609
(Assignee)

Updated

5 months ago
Depends on: 1359838
(Assignee)

Updated

5 months ago
No longer depends on: 1359838
(Assignee)

Updated

5 months ago
Depends on: 1374940
(Assignee)

Updated

4 months ago
Depends on: 1382564
(Assignee)

Updated

4 months ago
Depends on: 1382571
(Assignee)

Updated

4 months ago
Depends on: 1383993
(Assignee)

Updated

4 months ago
Blocks: 1383996
(Assignee)

Updated

4 months ago
Depends on: 1384417
(Assignee)

Updated

4 months ago
Depends on: 1384418
(Assignee)

Updated

4 months ago
Depends on: 1384419
(Assignee)

Updated

4 months ago
Depends on: 1384420
(Assignee)

Updated

4 months ago
Depends on: 1384421
(Assignee)

Updated

4 months ago
Depends on: 1384422
(Assignee)

Comment 7

4 months ago
Created attachment 8890195 [details] [diff] [review]
Migration helper

I just opened a bunch of bugs for individual toolchains. This is a helper patch for the migration, not meant to land.

The way it works is:
- Apply the patch
- Add an entry to KNOWN_TOOLCHAINS, for example `'linux64-sccache': '7bbe53cda3f6'`
- Run the command line from a decision task and add the missing toolchains until it doesn't complain anymore.
- Remove the corresponding tooltool manifest entries.
(Assignee)

Comment 8

4 months ago
Created attachment 8892217 [details] [diff] [review]
Migration helper

This is a revised version that takes into account that some toolchain jobs produce an artifact with the same name. That prevented either win32-clang-cl or win64-clang-cl to be found missing.
Attachment #8890195 - Attachment is obsolete: true
(Assignee)

Updated

4 months ago
Depends on: 1386028
(Assignee)

Updated

4 months ago
Depends on: 1386149
(Assignee)

Updated

4 months ago
Depends on: 1386519
(Assignee)

Updated

4 months ago
Depends on: 1386539
(Assignee)

Updated

4 months ago
Depends on: 1386588
(Assignee)

Updated

4 months ago
Depends on: 1386589
(Reporter)

Comment 9

3 months ago
Despite bug 1356529 still being open this seems pretty fixed. Should we punt additional work to followup bugs?
(Reporter)

Updated

3 months ago
Blocks: 1391408
(Assignee)

Comment 10

3 months ago
I'm keeping this as a tracking bug. The android toolchains and rust are the main remaining things.
Depends on: 1391427
Blocks: 1405408
Blocks: 1405412
Blocks: 1405413
Blocks: 1412006
You need to log in before you can comment on or make changes to this bug.