Closed Bug 1135640 (oxidation) Opened 9 years ago Closed 7 years ago

Add rust compiler support

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: larsberg, Assigned: larsberg)

References

(Depends on 8 open bugs, Blocks 2 open bugs)

Details

This bug is to track adding support for building code written in Rust inside of gecko build. The intent is for this to be behind a flag initially.


Notes from Portlandia on this topic:
https://github.com/servo/servo/wiki/Mozlandia-Rust-In-Gecko
Tracking bug for Windows debuginfo (PDB): https://github.com/rust-lang/rust/issues/19533
Can you please clarify what you want the scope of this bug to be? Are we talking simply "invoke rustc from the build system" or are we encompassing "bundle and ship Rust code in libxul/Gecko" as well?
Flags: needinfo?(larsberg)
Just invoking rustc from the build system. I'd like to open separate bugs for feature work that relies on it and ships in Gecko, if that's OK.
Flags: needinfo?(larsberg)
I'm more than OK with a tightly-scoped bug :)

So, I guess we're looking at configure support for detecting rustc and some moz.build foo to define Rust files, extra flags, etc, and some build backend code to turn moz.build into build rules. Sounds simple enough.
I'm not sure if detecting rustc will work, since rustc at the wrong version might be useless.

Preferably:

 - Download rustc as an (optional?) part of ./mach bootstrap
 - See if it exists, and use it if it does.

However I poked around my m-c clone and I don't see any place for downloaded target-agnostic files. Perhaps adding one might not be the best idea.

So, the alternative is the following:

 - Provide a specific dist tarball for rustc
 - Add configure support to detect rustc *and compare its version* (`rustc -V`)
 - Add an optional bootstrap flag to download the tarball and install it via the bundled install shell script.


This, of course, will break it for everyone who already uses Rust. For them, the solution is to use [multirust](https://github.com/brson/multirust), set up the downloaded tarball as a custom toolchain, and use overrides. This might be harder to integrate into the bootstrapping system though. (We can just put up instructions and hope Rustaceans are able to figure it out)
What's wrong with sniffing rustc capabilities (like is currently done for C/C++ compilers) in configure? Is Rust still such a moving target that downloading pre-built binaries instead of relying on system packages is preferred?

Don't get me wrong - I think offering a build mode that downloads exact versions of tools we use is good (deterministic builds, easier to developers to get up and running, etc). Just trying to understand why "download a binary" is your first instinct.
> Is Rust still such a moving target that downloading pre-built binaries instead of relying on system packages is preferred?

Depends on how we write the code. If we write it in the stable subset of Rust, we might be okay. But then we get a limited set of features. If we write it using full Rust features, there's no guarantee that stuff will work.


> Just trying to understand why "download a binary" is your first instinct.

It's what Servo does ;)
Ms2ger has swam in both oceans. I suspect he has insightful context to add.
Flags: needinfo?(Ms2ger)
Alias: oxidation
If you don't write it in the stable subset of Rust, then that code is very unlikely to ever be built and shipped in Linux distros, and other BSDs.
To be clear, when I say "stable" subset I'm talking of features/libraries which will not change. Using unstable Rust features means that upgrading the compiler one is using can break things.

I don't see the correlation with distros; with downloadable compiler snapshots we should be fine, right?

Without snapshots, we can't do this, obviously.
Distros are not going to download a snapshot during a firefox build.
Oh. That way.

No, I was suggesting we  make it a part of `./mach bootstrap`. But this will only work if we make it something that is off by default at compile time.

Perhaps sticking to the stable subset is best.
I suspect we should try to stick with the stable channel; if that doesn't work out, we can still figure out a way to limit the range of compiler versions.
Flags: needinfo?(Ms2ger)
The Rust stable channel is not frozen, either. We can request features we need and they may be added over time.
This is fine, we can require specific rustc versions in configure like we do with C++ compilers and other things.
Ms2ger: Given your knowledge of Rust and the Firefox build system, is this something you would be interested in implementing? If not implementing, would you be willing to propose a moz.build "schema" for defining Rust sources/crates/libraries?
Flags: needinfo?(Ms2ger)
I can help, but I'm not familiar with moz.build aside from having poked it a few times.
Not implementing, no, but I can try to think about a schema.
So, invoking rustc would just be the ability to add .rs files to SOURCES in moz.build.

I'd suggest we also/instead support cargo, rust's native package manager. The issues there are much more complicated as far as in-tree resources, but it would help us leverage rust libraries, unit tests, simplify pulling updates and so on. Maybe something like CRATES = [subdir list] runs 'cargo build' in each subdir and adds the products to the link list.
rillian: We're probably just going to invoke rustc at the start.

There's still a LOT of work to do with Cargo (some listed https://github.com/servo/servo/wiki/Mozlandia-Rust-In-Gecko#compiler-efficiency, more is in tracking bugs https://github.com/servo/servo/issues/2853 and https://github.com/servo/servo/issues/2854) before we could hope to land Cargo support in mozbuild.

That said, I'm tracking it and driving those requirements separately with the Rust/Cargo teams.
(In reply to Ralph Giles (:rillian) from comment #19)
> I'd suggest we also/instead support cargo, rust's native package manager.


Firefox automation is supposed to not require the internet whilst building (or something of the sort), Cargo "phones home" at times and we might have to set up a dummy registry like we do for pypi. Cargo could do with some improvements in this area; making it able to work offline in a way resilient to accidental lockfile clobbers, but we're not quite there yet.
For b2g builds we have git mirrors of the repos we need. I guess you could do the same and configure cargo to use these instead of the "remote" ones.
There's also the issue of reproducibility over time. If you push Firefox 45 to Try 2 years from now, the build should not fail because some random Rust package has been deleted from the Internet. This necessitates running our own mirror (which never deletes files) or "vendoring" 3rd party code in mozilla-central. The latter is preferred because a) it preserves the "you can build Firefox without Internet" property b) it is simpler (no server to run).

FWIW, given what Lars says about Cargo's unfinished state, I support the choice of starting with rustcc and adding Cargo support if/when needed.
Depends on: 1161339
Blocks: 1161350
Depends on: 1163214
Depends on: 1163224
Depends on: 1164109
Depends on: 1164126
Depends on: 1165382
I started a page on MDN to with a brief howto and documenting current limitations.

https://developer.mozilla.org/en-US/Firefox/Building_Firefox_with_Rust_code
Depends on: 1165528
Depends on: 1165531
Depends on: 1166996
Depends on: 1175359
Depends on: 1176294
Depends on: 1177128
Depends on: 1177608
Depends on: 1177791
Depends on: 1177896
Depends on: 1178897
Depends on: 1183850
Depends on: 1184732
Depends on: 1185662
Depends on: 1185664
Depends on: 1188030
Depends on: 1205012
Depends on: 1208566
Depends on: 1211439
Depends on: 1219530
Depends on: 1219983
Depends on: 1220307
Depends on: 1228703
Blocks: 1231711
No longer depends on: 1231711
Depends on: 1231764
Depends on: 1237366
Depends on: 1240982
Depends on: 1243037
Depends on: 1243363
Depends on: 1244261
Depends on: 1178667
Depends on: 1186719
Depends on: 1252964
Depends on: 1261503
Depends on: 1264490
Depends on: 1265893
Depends on: 1266877
Blocks: 1267492
Depends on: 1267845
Depends on: 1268328
Depends on: 1268547
Depends on: 1266368
Depends on: 1268727
Depends on: 1269790
Depends on: 1269808
Depends on: 1273603
Depends on: 1273625
Depends on: 1275659
Depends on: 1285323
Stylo depends on rust compiler support.
Blocks: stylo
Depends on: 1288173
Blocks: 1288280
No longer blocks: stylo
Hardware: x86 → All
Depends on: 1289583
Depends on: 1293253
Depends on: 1294083
Depends on: 1296403
Depends on: 1290972
Depends on: 1299971
Blocks: 1290022
Depends on: 1306438
Depends on: 1314071
Depends on: 1314336
Depends on: 1300835
Depends on: 1316751
Depends on: 1317478
Depends on: 1319690
Depends on: 1320022
Depends on: 1320269
Depends on: 1320710
Depends on: 1320741
Depends on: 1320960
Depends on: 1319156
Depends on: 1321073
Flags: needinfo?(Ms2ger)
Depends on: 1321691
Depends on: 1321696
Blocks: encoding_rs
Depends on: 1322798
Depends on: 1323557
Depends on: 1323901
Depends on: 1325917
Depends on: 1329700
Depends on: 1329726
Depends on: 1329737
Depends on: 1336155
Depends on: 1337955
Depends on: 1339176
Depends on: 1340587
Now that we have "oxidized" the Gecko builds with Rust code and are on track to ship with Rust support enabled by default, I would support closing this bug and tracking the ongoing stream of dependent issues in a more build-centric component, if that makes sense to others.
I support this as well. Based on IRC discussion I think we have rough concensus. While there are still some unresolved bugs here, most of the newer ones don't need tracking. We can rely on 'rust' or 'cargo' in the title, or come up with a whiteboard key if we need further tracking going forward.

Most of the integration issues are under Core::Build Config, so that's the place for most follow-up.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
No longer blocks: 1231711
Thanks for all the efforts of everyone involved!
Depends on: 1341041
Depends on: 1342815
Depends on: 1369242
Depends on: 1369247
Ralph: this bug is marked as fixed, though there are still blockers for it. I guess you could describe this bug as "stuff we must fix to ship Rust code".

There's a new bug "Make the developer experience for Firefox + Rust great" (bug 1380210), which is more about optional but nice-to-have stuff.

Should some/all of this bug's blockers be moved to the new bug? Because they're clearly not true blockers for shipping Rust code :)
Flags: needinfo?(giles)
That sounds reasonable. The other question is whether there's still value in having a tracking bug for all rust integration issues? I think from the drop-off of blocking bugs that it's not. Please feel free to move the relevant open bugs to the new tracking issue.
Flags: needinfo?(giles)
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.