1177599 - Make rust compiler support cross-compiling

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

Right now we assume that the rustc configure finds will produce binaries for the target architecture, we're not passing a target architecture to the compiler.

We should explicitly pass the target architecture to make cross-compiling work properly. We'll also want to check in configure that the rustc we find can generate code for the target architecture.

Comment 1

[Alex Crichton [:acrichto]]

> We'll also want to check in configure that the rustc we find can generate 
> code for the target architecture.

The compiler itself can always target any architecture (e.g. it can generate code for anything), but you're probably always linking against the standard library, so this will likely be determined by the number of target directories that the compiler has. You can find this via:

    find `rustc --print sysroot`/lib/rustlib -depth 1 -type d

The triples there are the targets that the compiler has target libraries for, so `rustc --target $triple`  should work for all of those directories.

(just FYI!)

Comment 2

[Ralph Giles (:rillian)]

As an update, upstream is now packaging builds of just the standard library to facilitate cross compiling. So one can download e.g. https://static.rust-lang.org/dist/rust-std-nightly-arm-linux-androideabi.tar.gz, unpack it in the sysroot/lib/rustlib, and use it to cross-compile for that target.

Comment 3

[Nathan Froyd [:froydnj]]

I signed up to do the build system integration and the standard library (re-)packing part.  Ralph, are you going to do the OS X update in the dependent bug, or should we just roll that bug into this one?

Comment 4

[Ralph Giles (:rillian)]

Hi Nathan. Thanks for signing up!

I did the mac-hosted i686-apple-darwin cross as part of bug 1228703 because it was necessary to enable rust in Mac nightly (universal) builds. And I'm doing updated mac- and linux-hosted builds of rustc 1.5 (bug 1211562) so we can land your depinfo patch.

If you want to work on generalizing this per Ted's plan in the description to make the TC cross-mac builds and android builds possible, or just to turn them on one at a time, that would be great.

Comment 5

[(not currently active) Ted Mielczarek]

There are kind of two parts here:
1) Have configure check that rustc has a libstd for the target arch. This is probably just running rustc --target$target and ensuring it works.
2) Actually put toolchains with the right libstd in tooltool for our various platforms. It might be nice (I don't know how big libstd is) to make the Linux toolchain contain libstd for linux32,linux64,mac32,mac64,android-arm, so that we could use the same rust toolchain package to build all of our Linux, Android, and cross-Mac builds. (Seems like it'd be straightforward to write a script to grab a rustc release + a set of matching libstd releases and pack it up.)

Comment 6

[Nathan Froyd [:froydnj]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> There are kind of two parts here:
> 1) Have configure check that rustc has a libstd for the target arch. This is
> probably just running rustc --target$target and ensuring it works.

Yup.

> 2) Actually put toolchains with the right libstd in tooltool for our various
> platforms. It might be nice (I don't know how big libstd is) to make the
> Linux toolchain contain libstd for linux32,linux64,mac32,mac64,android-arm,
> so that we could use the same rust toolchain package to build all of our
> Linux, Android, and cross-Mac builds. (Seems like it'd be straightforward to
> write a script to grab a rustc release + a set of matching libstd releases
> and pack it up.)

That's what I was planning to do, script and all.  I can't imagine libstd is more than a couple MB, tops.

Comment 7

[Ralph Giles (:rillian)]

(In reply to Nathan Froyd [:froydnj] from comment #6)

> That's what I was planning to do, script and all.

I recommend PRs against https://github.com/mozilla/rust-tooltool

NB we can't currently just repackage the official builds for Mac and Linux. Linux needs --enable-rpath (https://github.com/rust-lang/rust/issues/29941) to avoid a conflict with the gtk3 LD_LIBRARY_PATH stuff and Mac needs --disable-elf-tls. In the case of linux/android targets we should be able to use the upstream libstd builds though.

I've been experimenting with build scripts in https://github.com/rillian/rust-build

> I can't imagine libstd is more than a couple MB, tops.

~17 MB gzipped, looks like.

$ for file in $(curl -s https://static.rust-lang.org/dist/channel-rustc-stable | grep std); do echo -en "$file\t"; curl -s --head https://static.rust-lang.org/dist/$file | grep Content-Length; done
rust-std-1.5.0-aarch64-unknown-linux-gnu.tar.gz	Content-Length: 17164808
rust-std-1.5.0-arm-linux-androideabi.tar.gz	Content-Length: 17003277
rust-std-1.5.0-arm-unknown-linux-gnueabihf.tar.gz	Content-Length: 34480447
rust-std-1.5.0-arm-unknown-linux-gnueabi.tar.gz	Content-Length: 17266775
rust-std-1.5.0-i686-apple-darwin.tar.gz	Content-Length: 45843024
rust-std-1.5.0-i686-pc-windows-gnu.tar.gz	Content-Length: 50318798
rust-std-1.5.0-i686-pc-windows-msvc.tar.gz	Content-Length: 39684800
rust-std-1.5.0-i686-unknown-linux-gnu.tar.gz	Content-Length: 50854601
rust-std-1.5.0-mipsel-unknown-linux-gnu.tar.gz	Content-Length: 17219798
rust-std-1.5.0-mips-unknown-linux-gnu.tar.gz	Content-Length: 17299642
rust-std-1.5.0-x86_64-apple-darwin.tar.gz	Content-Length: 46053324
rust-std-1.5.0-x86_64-pc-windows-gnu.tar.gz	Content-Length: 50054195
rust-std-1.5.0-x86_64-pc-windows-msvc.tar.gz	Content-Length: 43888009
rust-std-1.5.0-x86_64-unknown-linux-gnu.tar.gz	Content-Length: 50388348
rust-std-1.5.0-x86_64-unknown-linux-musl.tar.gz	Content-Length: 13421731

Comment 8

[Ralph Giles (:rillian)]

17-50 MB, I should have said.

Comment 9

[(not currently active) Ted Mielczarek]

(In reply to Ralph Giles (:rillian) from comment #7)
> (In reply to Nathan Froyd [:froydnj] from comment #6)
> 
> > That's what I was planning to do, script and all.
> 
> I recommend PRs against https://github.com/mozilla/rust-tooltool
> 
> NB we can't currently just repackage the official builds for Mac and Linux.
> Linux needs --enable-rpath (https://github.com/rust-lang/rust/issues/29941)
> to avoid a conflict with the gtk3 LD_LIBRARY_PATH stuff and Mac needs
> --disable-elf-tls. In the case of linux/android targets we should be able to
> use the upstream libstd builds though.
> 
> I've been experimenting with build scripts in
> https://github.com/rillian/rust-build

Neat! FYI, we've stuck some scripts we use to build toolchains in the tree in https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/misc/ (obviously those are written to be run in Taskcluster).

Comment 10

[Nathan Froyd [:froydnj]]

Attachment: always pass --target to rustc

rustc, unlike our typical C++ compilers, can target multiple platforms
with ease through its use of the --target flag.  To support
cross-compiling, we just need to pass the appropriate --target option.

rustc uses specific names for its accepted --target option, however, and
they are slightly different from the values we get out of autoconf.  So
in addition to checking whether rustc can accept --target for our chosen
platform, we also need to munge autoconf's idea of the target into
something rustc understands.

(This addresses part 1 of comment 5; I think splitting up bits in tooltool
should be something different.)

Comment 11

[Michael Shal [:mshal]]

Comment on attachment 8712783 [details] [diff] [review]
always pass --target to rustc

>+      # OS X and iOS
>+      i*86-apple-darwin*)
>+          rust_target=i686-apple-darwin
>+          ;;
>+      i*86-apple-ios*)
>+          rust_target=i386-apple-ios
>+          ;;
>+      x86_64-apple-darwin*)
>+          rust_target=x86_64-apple-darwin
>+          ;;

Is there a reason we won't need x86_64-apple-ios as well?

>+      # Windows
>+      i*86-pc-mingw32)
>+          # XXX better detection of CXX needed here, to figure out whether
>+          # we need i686-pc-windows-gnu instead, since mingw32 builds work.

Are you intending to address this as part of this bug or as a followup?

>+  # TODO: separate HOST_RUSTC and RUSTC variables

Same question here.

Comment 12

[Nathan Froyd [:froydnj]]

(In reply to Michael Shal [:mshal] from comment #11)
> >+      x86_64-apple-darwin*)
> >+          rust_target=x86_64-apple-darwin
> >+          ;;
> 
> Is there a reason we won't need x86_64-apple-ios as well?

Um.  Not sure!  I suppose we could add it, but it seems more likely that we'd need arm{,64}-apple-ios.  Not sure that it matters, since Firefox iOS is not built on Gecko anyway.

> >+      # Windows
> >+      i*86-pc-mingw32)
> >+          # XXX better detection of CXX needed here, to figure out whether
> >+          # we need i686-pc-windows-gnu instead, since mingw32 builds work.
> 
> Are you intending to address this as part of this bug or as a followup?

I figure this can be a followup, or it can be fixed once we start requiring Rust to build the tree and something breaks on mingw32 buildbots or something.

> >+  # TODO: separate HOST_RUSTC and RUSTC variables
> 
> Same question here.

This is also a followup...once somebody re-writes nsinstall.c in Rust... :)

Comment 13

[Nathan Froyd [:froydnj]]

(In reply to Nathan Froyd [:froydnj] from comment #10)
> Created attachment 8712783 [details] [diff] [review]
> always pass --target to rustc

This approach doesn't work with the patches landed in bug 1243037: on Windows, we export RUSTC="/path/to/rustc --target $T", which gets picked up by JS's configure.  The rust checks on the JS side then try to execute:

  $RUSTC --target $T

which fails with a rustc error that you can't pass --target multiple times.

We need the RUSTC export, otherwise JS's configure can't find rustc and fails.

I'm not sure how to handle this.  Should the --target=$T bit be separated out into a RUST_TARGET_FLAGS variable that gets exported by configure and then used in rules.mk?

Comment 14

[Nathan Froyd [:froydnj]]

Probably need a RUST_TARGET_FLAGS to handle things like passing a cross-linker to rustc anyway...

Comment 15

[Nathan Froyd [:froydnj]]

Attachment: always pass --target to rustc

This patch implements something a little different than the RUST_TARGET_FLAGS
variable that we talked about.  It also uses a slightly different method for
testing the suitability of the --target option, one that's more amenable to
cross-compilation.

Comment 16

[Michael Shal [:mshal]]

Comment on attachment 8714954 [details] [diff] [review]
always pass --target to rustc

>+    # Check to see whether we need to pass --target to RUSTC.  This can
>+    # happen when building Firefox on Windows: js's configure will receive
>+    # a RUSTC from the toplevel configure that already has --target added to
>+    # it.
>+    rustc_target_arg=
>+    case "$RUSTC" in
>+      *--target=${rust_target}*)
>+        ;;
>+      *)
>+        rustc_target_arg=--target=${rust_target}
>+        ;;
>+    esac

Seems like a reasonable solution :)

nit: this case uses a two-space indent for the ;; and rustc_target_arg lines, while the previous case statement in the file has four spaces (though we seem to be all over the place in configure.in, so meh).

Comment 17

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/627dd0b98523

Comment 18

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/627dd0b98523