1650239 - Patches to support a parallel CI for clang trunk

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[(Away)]

Previous compiler upgrades have shown that we lose enormous amounts of engineering time dealing with fallout from newer compiler versions. When our "last known good" point is months in the past, it takes tons of time to narrow down a problem on the Gecko side, then narrow it down on the LLVM side, and then try to get a fix from upstream when the original developer has already swapped out that code from their mind, or so much other code has landed on top that it's harder to fix. I want to see us get to a point where the time between our test runs is on the order of days, not weeks or months.

On the ash branch I'm attempting to set up a prototype of a system that will frequently test new LLVM revisions against new Gecko revisions. I'd like to land as much of the supporting work as possible on m-c though. For example, adding new clang-trunk toolchains to the taskcluster yml files, even if they go unused in our normal m-c automation. Then ideally, the only delta on the ash branch would be a small change to move the toolchain-alias lines from the clang-9 tasks to the clang-trunk tasks. (In practice, ash might also have some short-term workarounds to get builds back to green while investigations are in progress.)

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[(Away)]

Attachment: Rename cctools-clang-10-port to cctools-port-clang-10

Some cleanup before I add more copies of this task.

Since this is based on a repo called cctools-port it seems like it would be better to keep that substring intact.

Comment 3

[(Away)]

Attachment: Name clang-9 toolchains as such explicitly

We already do this for e.g. linux64-clang-9; this patch extends that pattern everywhere.

This will make it easy to do try runs with other clangs: just move the toolchain-alias lines from the 9 tasks to the trunk tasks.

Comment 4

[(Away)]

Attachment: Update cctools-port to make it buildable with clang 11

Fixes the bustage described in https://phabricator.services.mozilla.com/differential/diff/307597/

In theory I could leave this task alone and fork a new toolchain for use by clang 11, but this seems internal-enough that existing builds shouldn't be negatively affected by this.

Comment 5

[(Away)]

Attachment: Allow threads in our Mac clang build

Source history does not give any good clues about why this line was added in the first place. In any case, LLVM trunk currently has build bustage when threads are disabled. We could work around the bustage and/or wait for a fix, but it seems like threads are a good thing to have in general nowadays. Maybe this could help with LTO build times.

Comment 6

[(Away)]

Attachment: Build llvm-dsymutil out of the clang-10 source tree

The current version hits OOM errors when dsymutil-ing files created by clang 10 and 11.

The choice of clang-10 here is somewhat arbitrary in that it's a fetch job that we conveniently already had in the tree. It doesn't have to be exactly version 10 specifically.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f94dd6e92776
Rename cctools-clang-10-port to cctools-port-clang-10  r=firefox-build-system-reviewers,rstewart

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7be0c0fd106b
Allow threads in our Mac clang build r=froydnj

Comment 9

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/f94dd6e92776
https://hg.mozilla.org/mozilla-central/rev/7be0c0fd106b

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d5f4639fbc93
Name clang-9 toolchains as such explicitly r=firefox-build-system-reviewers,rstewart

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/546201cdf1cb
Update cctools-port to make it buildable with clang 11 r=froydnj

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/75384d97c9df
Build llvm-dsymutil out of the clang-10 source tree r=firefox-build-system-reviewers,rstewart

Comment 13

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/d5f4639fbc93
https://hg.mozilla.org/mozilla-central/rev/546201cdf1cb
https://hg.mozilla.org/mozilla-central/rev/75384d97c9df

Comment 14

[(Away)]

I've achieved more or less what I wanted to with this bug. If any new pieces arise, they can be separate tickets.