1614858 - Do regular builds with clang trunk

Status: NEW

(Firefox Build System :: Toolchains, task)

Description

[Mike Hommey [:glandium]]

Most of the pieces to do that automatically are actually available, with a caveat.

• "cron" tasks can be setup, cf. http://firefox-source-docs.mozilla.org/taskcluster/cron.html

• Defining what specific tasks to run via "cron" is rather straightforward, cf. e.g. https://searchfox.org/mozilla-central/rev/7e92a667e3829831c31e8d46aefe7ef67ad5be1c/taskcluster/taskgraph/target_tasks.py#611

• Adding tasks that would use clang-trunk is also straightforward in taskcluster/ci/build, although ideally, we'd derive them from existing tasks to avoid one being updated without the others (but we already have the problem with e.g. dev-edition and reproducible builds, so the ideal would be out of scope here).

• Adding a toolchain for clang trunk is the most tricky part, because fetches do want static definitions in taskcluster/ci/fetches and do want fixed sha1s per https://searchfox.org/mozilla-central/rev/7e92a667e3829831c31e8d46aefe7ef67ad5be1c/taskcluster/taskgraph/transforms/fetch.py#261-263
  I'd rather not lift the restriction because it's a footgun, but I could see us adding an alternative field to revision that would track a branch and disable caching so that the task would run every time it's needed https://searchfox.org/mozilla-central/rev/7e92a667e3829831c31e8d46aefe7ef67ad5be1c/taskcluster/taskgraph/transforms/fetch.py#288 . This will probably require setting run-on-projects to [] on all the involved tasks to avoid them being scheduled on every push. They would still happen when explicitly requested via mach try fuzzy or via cron.

Comment 1

[(Away)]

This is great; thanks for filing, glandium.

My understanding is that cron jobs get added to whatever unsuspecting push happens at the trigger time. I'm not so sure I'd want to add this extra noise to pushes or make people worry about failures that will inevitably happen (even if these are marked tier 3 or whatever). And it would be nice to have a view that shows only the pushes with trunk jobs, one day after the next. Also, how do tests work in this setup? Do we need to duplicate everything like linux64-asan and linux64-asan-with-clang-trunk etc.?

I think my ideal would be to have a bot that does try pushes where the clang fetch task is changed to the latest revision. That avoids duplicating the test definitions, and would let us filter on that bot's username to see the history of pushes from one day to the next. I am not sure how much harder this would be than your proposal though.

Comment 2

[(Away)]

I guess neither of us has mentioned tests explicitly, so for completeness I'll share my position here. In my experience, the bulk of the effort dealing with "clang N+1" problems happens outside of basic builds. New -Werrors (for example) are pretty easy to deal with, versus the runtime problems are the trickiest to repro and debug, and would benefit the most from having a viewable history so that we can minimize bisect windows. So while builds alone would be a great step, in the long term I'd love to have a world where we get full test suite results on a regular basis.

Comment 3

[Tom Ritter [:tjr]]

I don't have a timeframe beyond '2020' but I will be working on a system that will be able to run try pushes with arbitrary sets of builds/tests bumping clang to trunk; as well as (if desired) filing bugs for breakage and patches to land clang bumps for a particular toolchain.

Comment 4

[Nathan Froyd [:froydnj]]

Bastien had mentioned having some experience with cron stuff also.

Comment 5

[Sylvestre Ledru [:Sylvestre]]

I have been doing that work for several years now on https://relman-ci.moz.tools/
So, I am 200% in favor of this.

Are we going to go tier 2 or 3?
What should the action in case of failure?

What about gcc too ? gcc is doing great work in term of warnings/errors detection (better than clang IMHO). In clang, most of the improvements are doing in clang-tidy.

Comment 6

[Eric Rahm [:erahm]]

(In reply to Sylvestre Ledru [:Sylvestre] from comment #5)

I have been doing that work for several years now on https://relman-ci.moz.tools/
So, I am 200% in favor of this.

Are we going to go tier 2 or 3?

What's the difference? If tier 2 means "Sheriffs look at it and file bugs but most developers won't be bothered" that's possibly okay, but I think we're going to expect it to permafail a lot (and possibly fix itself as upstream fixes bugs).

What should the action in case of failure?

I suppose filing a bug. If it just doesn't compile then we can probably ignore it. For things like new -Werrors we can fix them or disable at directory level and file follow ups. For ICE's we'd file upstream bugs. For test failures we'd need a lot of further analysis.

What about gcc too ? gcc is doing great work in term of warnings/errors detection (better than clang IMHO). In clang, most of the improvements are doing in clang-tidy.

I think we're focused on LLVM for now as it's our default compiler toolchain, but ideally whatever we get in place could be a template for testing GCC release branches as well.

Comment 7

[(Away)]

(In reply to Eric Rahm [:erahm] from comment #6)

What's the difference? If tier 2 means "Sheriffs look at it and file bugs but most developers won't be bothered" that's possibly okay, but I think we're going to expect it to permafail a lot (and possibly fix itself as upstream fixes bugs).

(Assuming we're talking about cron jobs getting added to random developers' pushes) I don't want sheriffs to file bugs "against" a certain changeset that wasn't at fault, developers shouldn't think they need to spend time investigating those.

Comment 8

[Mike Hommey [:glandium]]

Bug 1703483 addressed comment 0, let's recycle this bug for the upcoming extended version of this, where we setup a separate branch to run the same things as central using clang-trunk.