Closed Bug 1262729 Opened 4 years ago Closed 4 years ago

Allow to more conveniently build toolchains

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox48 fixed)

RESOLVED FIXED
mozilla48
Tracking Status
firefox48 --- fixed

People

(Reporter: glandium, Unassigned)

References

Details

Attachments

(1 file)

Currently, we have in-tree scripts that allow to build our toolchains. We also have a taskcluster script for clang that can be used in with manually created taskcluster tasks. I wrote a similar script for gcc a while ago, for the same usage.

This bug is about hooking all these up such that one can do a try push with `-p linux64-clang` or `-p linux64-gcc` and get a toolchain built as a result.
Currently, we have in-tree scripts that allow to build our toolchains.
We also have a taskcluster script for clang that can be used in with
manually created taskcluster tasks. I wrote a similar script for gcc a
while ago, for the same usage.

This change hooks all these up such that one can do a try push with
`-p linux64-clang` or `-p linux64-gcc` and get a toolchain built as a
result.

Review commit: https://reviewboard.mozilla.org/r/44725/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/44725/
Attachment #8738872 - Flags: review?(dustin)
This is a try with -p linux64-gcc,linux64-clang:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=af72df9fbcb3b1569c8b4609ca5f7b75a052f0ed

Note the linux64-clang job is busted (patch not applying), but that's not related to this bug.

mach taskcluster-graph suggests that the patch is enough for those jobs to *not* happen on -p all, which is what is wanted, but I don't really know what I'm doing, so...

Also, the extra.locations.* are wrong, and taskcluster complains about them, but without the fields, taskcluster-graph complains, and other jobs that don't build firefox have those dummy values, so I just cargo culted.
Depends on: 1262735
Blocks: 1262781
I was thinking about having mach commands to do this (for things like minidump_stackwalk), but building them on try is pretty sensible too. Ideally we'd wire things up at some point so that you could push to try and get a toolchain build, and then get a Firefox built with that toolchain...
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #3)
> I was thinking about having mach commands to do this (for things like
> minidump_stackwalk), but building them on try is pretty sensible too.
> Ideally we'd wire things up at some point so that you could push to try and
> get a toolchain build, and then get a Firefox built with that toolchain...

We talked about that at Orlando, but that requires everything to be in taskcluster...
Comment on attachment 8738872 [details]
MozReview Request: Bug 1262729 - Allow to more conveniently build toolchains. r?dustin

https://reviewboard.mozilla.org/r/44725/#review41529

I think that treating this as a "platform" is wrong -- we're not building Firefox to run on Clang, which is what -p linux64-clang suggests.  That misfit is reflected concretely in the task.extras strangeness.  As I refactor the task-graph generation process, I don't want to be tied down by one more ill-fitting hack to the try syntax.

I'd prefer something more like what Ted suggests: a mach command that can create the toolchain-building task.

Long-term, the toolchains will be a part of the task graph -- just one that is optimized out almost all the time since no changes have occurred.  But you would be able to update the toolchain script and push to try, either selecting the toolchain-building task as a target or selecting a Firefox build or test as a target.  In either case, the toolchain would be built before anything that depended on it.
Attachment #8738872 - Flags: review?(dustin)
Assignee: mh+mozilla → nobody
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WONTFIX
Note that a mach command would still have to push the local state somewhere, which means it would have to deal with how different people have different setups to push. Using try is still the simpler solution that doesn't require having to deal with idiosyncrasies.

Also, the world in which you can push changes to the toolchain build script and it propagates down to the firefox build using the newly compiled toolchain is far away. Being in the process of compiling new toolchains *right* *now*, this patch has been useful to me, however hacky it is.
WONTFIX saddens me.

While I agree with what dustin said about eventually having toolchain building as a prerequisite task for a build task, I think there is room to shoehorn this into TC/Try today.

The work I did in https://hg.mozilla.org/mozilla-central/rev/623765c2381e adds a Taskcluster only -j/--job argument to Try syntax which I think would be appropriate for producing toolchains in the short term. That allows us to define a Taskcluster mechanism for building toolchains (which we'll need eventually) and a method to invoke them in a consistent environment (as opposed to what you get from a mach command running locally).

We may not run these tasks on every push or use the output for build tasks just yet. But it should get us one step closer to the eventual goal. I could easily see someone building on top of this work and e.g. scheduling daily builds of Firefox against GCC and Clang head/tip/master (a subject that has come up multiple times in the past few weeks, surprisingly).
-j seems like a good fit, yes
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Comment on attachment 8738872 [details]
MozReview Request: Bug 1262729 - Allow to more conveniently build toolchains. r?dustin

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/44725/diff/1-2/
Attachment #8738872 - Flags: review?(dustin)
Comment on attachment 8738872 [details]
MozReview Request: Bug 1262729 - Allow to more conveniently build toolchains. r?dustin

https://reviewboard.mozilla.org/r/44725/#review42557
Attachment #8738872 - Flags: review?(dustin) → review+
https://hg.mozilla.org/mozilla-central/rev/bba8a0264e24
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.