Closed Bug 1177599 Opened 4 years ago Closed 4 years ago

Make rust compiler support cross-compiling

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox47 fixed)

RESOLVED FIXED
mozilla47
Tracking Status
firefox47 --- fixed

People

(Reporter: ted, Assigned: froydnj)

References

Details

Attachments

(1 file, 1 obsolete file)

Right now we assume that the rustc configure finds will produce binaries for the target architecture, we're not passing a target architecture to the compiler.

We should explicitly pass the target architecture to make cross-compiling work properly. We'll also want to check in configure that the rustc we find can generate code for the target architecture.
> We'll also want to check in configure that the rustc we find can generate 
> code for the target architecture.

The compiler itself can always target any architecture (e.g. it can generate code for anything), but you're probably always linking against the standard library, so this will likely be determined by the number of target directories that the compiler has. You can find this via:

    find `rustc --print sysroot`/lib/rustlib -depth 1 -type d

The triples there are the targets that the compiler has target libraries for, so `rustc --target $triple`  should work for all of those directories.

(just FYI!)
As an update, upstream is now packaging builds of just the standard library to facilitate cross compiling. So one can download e.g. https://static.rust-lang.org/dist/rust-std-nightly-arm-linux-androideabi.tar.gz, unpack it in the sysroot/lib/rustlib, and use it to cross-compile for that target.
I signed up to do the build system integration and the standard library (re-)packing part.  Ralph, are you going to do the OS X update in the dependent bug, or should we just roll that bug into this one?
Assignee: nobody → nfroyd
Flags: needinfo?(giles)
Hi Nathan. Thanks for signing up!

I did the mac-hosted i686-apple-darwin cross as part of bug 1228703 because it was necessary to enable rust in Mac nightly (universal) builds. And I'm doing updated mac- and linux-hosted builds of rustc 1.5 (bug 1211562) so we can land your depinfo patch.

If you want to work on generalizing this per Ted's plan in the description to make the TC cross-mac builds and android builds possible, or just to turn them on one at a time, that would be great.
There are kind of two parts here:
1) Have configure check that rustc has a libstd for the target arch. This is probably just running rustc --target$target and ensuring it works.
2) Actually put toolchains with the right libstd in tooltool for our various platforms. It might be nice (I don't know how big libstd is) to make the Linux toolchain contain libstd for linux32,linux64,mac32,mac64,android-arm, so that we could use the same rust toolchain package to build all of our Linux, Android, and cross-Mac builds. (Seems like it'd be straightforward to write a script to grab a rustc release + a set of matching libstd releases and pack it up.)
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> There are kind of two parts here:
> 1) Have configure check that rustc has a libstd for the target arch. This is
> probably just running rustc --target$target and ensuring it works.

Yup.

> 2) Actually put toolchains with the right libstd in tooltool for our various
> platforms. It might be nice (I don't know how big libstd is) to make the
> Linux toolchain contain libstd for linux32,linux64,mac32,mac64,android-arm,
> so that we could use the same rust toolchain package to build all of our
> Linux, Android, and cross-Mac builds. (Seems like it'd be straightforward to
> write a script to grab a rustc release + a set of matching libstd releases
> and pack it up.)

That's what I was planning to do, script and all.  I can't imagine libstd is more than a couple MB, tops.
(In reply to Nathan Froyd [:froydnj] from comment #6)

> That's what I was planning to do, script and all.

I recommend PRs against https://github.com/mozilla/rust-tooltool

NB we can't currently just repackage the official builds for Mac and Linux. Linux needs --enable-rpath (https://github.com/rust-lang/rust/issues/29941) to avoid a conflict with the gtk3 LD_LIBRARY_PATH stuff and Mac needs --disable-elf-tls. In the case of linux/android targets we should be able to use the upstream libstd builds though.

I've been experimenting with build scripts in https://github.com/rillian/rust-build

> I can't imagine libstd is more than a couple MB, tops.

~17 MB gzipped, looks like.

$ for file in $(curl -s https://static.rust-lang.org/dist/channel-rustc-stable | grep std); do echo -en "$file\t"; curl -s --head https://static.rust-lang.org/dist/$file | grep Content-Length; done
rust-std-1.5.0-aarch64-unknown-linux-gnu.tar.gz	Content-Length: 17164808
rust-std-1.5.0-arm-linux-androideabi.tar.gz	Content-Length: 17003277
rust-std-1.5.0-arm-unknown-linux-gnueabihf.tar.gz	Content-Length: 34480447
rust-std-1.5.0-arm-unknown-linux-gnueabi.tar.gz	Content-Length: 17266775
rust-std-1.5.0-i686-apple-darwin.tar.gz	Content-Length: 45843024
rust-std-1.5.0-i686-pc-windows-gnu.tar.gz	Content-Length: 50318798
rust-std-1.5.0-i686-pc-windows-msvc.tar.gz	Content-Length: 39684800
rust-std-1.5.0-i686-unknown-linux-gnu.tar.gz	Content-Length: 50854601
rust-std-1.5.0-mipsel-unknown-linux-gnu.tar.gz	Content-Length: 17219798
rust-std-1.5.0-mips-unknown-linux-gnu.tar.gz	Content-Length: 17299642
rust-std-1.5.0-x86_64-apple-darwin.tar.gz	Content-Length: 46053324
rust-std-1.5.0-x86_64-pc-windows-gnu.tar.gz	Content-Length: 50054195
rust-std-1.5.0-x86_64-pc-windows-msvc.tar.gz	Content-Length: 43888009
rust-std-1.5.0-x86_64-unknown-linux-gnu.tar.gz	Content-Length: 50388348
rust-std-1.5.0-x86_64-unknown-linux-musl.tar.gz	Content-Length: 13421731
Flags: needinfo?(giles)
17-50 MB, I should have said.
(In reply to Ralph Giles (:rillian) from comment #7)
> (In reply to Nathan Froyd [:froydnj] from comment #6)
> 
> > That's what I was planning to do, script and all.
> 
> I recommend PRs against https://github.com/mozilla/rust-tooltool
> 
> NB we can't currently just repackage the official builds for Mac and Linux.
> Linux needs --enable-rpath (https://github.com/rust-lang/rust/issues/29941)
> to avoid a conflict with the gtk3 LD_LIBRARY_PATH stuff and Mac needs
> --disable-elf-tls. In the case of linux/android targets we should be able to
> use the upstream libstd builds though.
> 
> I've been experimenting with build scripts in
> https://github.com/rillian/rust-build

Neat! FYI, we've stuck some scripts we use to build toolchains in the tree in https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/misc/ (obviously those are written to be run in Taskcluster).
Attached patch always pass --target to rustc (obsolete) — Splinter Review
rustc, unlike our typical C++ compilers, can target multiple platforms
with ease through its use of the --target flag.  To support
cross-compiling, we just need to pass the appropriate --target option.

rustc uses specific names for its accepted --target option, however, and
they are slightly different from the values we get out of autoconf.  So
in addition to checking whether rustc can accept --target for our chosen
platform, we also need to munge autoconf's idea of the target into
something rustc understands.

(This addresses part 1 of comment 5; I think splitting up bits in tooltool
should be something different.)
Attachment #8712783 - Flags: review?(mshal)
Comment on attachment 8712783 [details] [diff] [review]
always pass --target to rustc

>+      # OS X and iOS
>+      i*86-apple-darwin*)
>+          rust_target=i686-apple-darwin
>+          ;;
>+      i*86-apple-ios*)
>+          rust_target=i386-apple-ios
>+          ;;
>+      x86_64-apple-darwin*)
>+          rust_target=x86_64-apple-darwin
>+          ;;

Is there a reason we won't need x86_64-apple-ios as well?

>+      # Windows
>+      i*86-pc-mingw32)
>+          # XXX better detection of CXX needed here, to figure out whether
>+          # we need i686-pc-windows-gnu instead, since mingw32 builds work.

Are you intending to address this as part of this bug or as a followup?

>+  # TODO: separate HOST_RUSTC and RUSTC variables

Same question here.
Attachment #8712783 - Flags: review?(mshal) → review+
(In reply to Michael Shal [:mshal] from comment #11)
> >+      x86_64-apple-darwin*)
> >+          rust_target=x86_64-apple-darwin
> >+          ;;
> 
> Is there a reason we won't need x86_64-apple-ios as well?

Um.  Not sure!  I suppose we could add it, but it seems more likely that we'd need arm{,64}-apple-ios.  Not sure that it matters, since Firefox iOS is not built on Gecko anyway.

> >+      # Windows
> >+      i*86-pc-mingw32)
> >+          # XXX better detection of CXX needed here, to figure out whether
> >+          # we need i686-pc-windows-gnu instead, since mingw32 builds work.
> 
> Are you intending to address this as part of this bug or as a followup?

I figure this can be a followup, or it can be fixed once we start requiring Rust to build the tree and something breaks on mingw32 buildbots or something.

> >+  # TODO: separate HOST_RUSTC and RUSTC variables
> 
> Same question here.

This is also a followup...once somebody re-writes nsinstall.c in Rust... :)
(In reply to Nathan Froyd [:froydnj] from comment #10)
> Created attachment 8712783 [details] [diff] [review]
> always pass --target to rustc

This approach doesn't work with the patches landed in bug 1243037: on Windows, we export RUSTC="/path/to/rustc --target $T", which gets picked up by JS's configure.  The rust checks on the JS side then try to execute:

  $RUSTC --target $T

which fails with a rustc error that you can't pass --target multiple times.

We need the RUSTC export, otherwise JS's configure can't find rustc and fails.

I'm not sure how to handle this.  Should the --target=$T bit be separated out into a RUST_TARGET_FLAGS variable that gets exported by configure and then used in rules.mk?
Flags: needinfo?(mshal)
Probably need a RUST_TARGET_FLAGS to handle things like passing a cross-linker to rustc anyway...
This patch implements something a little different than the RUST_TARGET_FLAGS
variable that we talked about.  It also uses a slightly different method for
testing the suitability of the --target option, one that's more amenable to
cross-compilation.
Attachment #8714954 - Flags: review?(mshal)
Attachment #8712783 - Attachment is obsolete: true
Comment on attachment 8714954 [details] [diff] [review]
always pass --target to rustc

>+    # Check to see whether we need to pass --target to RUSTC.  This can
>+    # happen when building Firefox on Windows: js's configure will receive
>+    # a RUSTC from the toplevel configure that already has --target added to
>+    # it.
>+    rustc_target_arg=
>+    case "$RUSTC" in
>+      *--target=${rust_target}*)
>+        ;;
>+      *)
>+        rustc_target_arg=--target=${rust_target}
>+        ;;
>+    esac

Seems like a reasonable solution :)

nit: this case uses a two-space indent for the ;; and rustc_target_arg lines, while the previous case statement in the file has four spaces (though we seem to be all over the place in configure.in, so meh).
Flags: needinfo?(mshal)
Attachment #8714954 - Flags: review?(mshal) → review+
https://hg.mozilla.org/mozilla-central/rev/627dd0b98523
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 47
Depends on: 1248391
Component: Build Config → General
Product: Firefox → Firefox Build System
Target Milestone: Firefox 47 → mozilla47
You need to log in before you can comment on or make changes to this bug.