1184732 - enable rust for windows nightly/aurora builds

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Ralph Giles (:rillian)]

To get some testing we should enable rust for integration and nightly build for windows desktop.

Comment 1

[Ralph Giles (:rillian)]

This may be blocked on some of the windows xp support work happening in the rust tree. I.e. it may require a newer toolchain.

Comment 2

[Alex Crichton [:acrichto]]

To confirm, this basically involves getting a compiler that can target i686-pc-windows-msvc as a target triple, right? Once we've got that the standard library already supports XP so that should do the trick.

Comment 3

[Ralph Giles (:rillian)]

Uploaded repackaged rust-1.1.0 official builds to tooltool.

[
{
"size": 65629796,
"visibility": "public",
"digest": "a4c197cb5b8756e492307b601cc3fb43b713f212049935e21656003ec1f89c5d73aaf020291ec00f722a9a885a081541808357305b2212ab3f61e351a44b9484",
"algorithm": "sha512",
"filename": "rustc-1.1.0-x86_64-pc-windows-gnu.tar.xz"
},
{
"size": 67194876,
"visibility": "public",
"digest": "3a845ea9cee776a82cd01b1e792ed0415f590df21cdb5c2604d60c0bdcdc8071d64dcd210d23f43c4b31745263224154d8eff816cdaa3fc6c4ddfa8c2b3ec6c0",
"algorithm": "sha512",
"filename": "rustc-1.1.0-i686-unknown-linux-gnu.tar.xz"
},
{
"size": 66373156,
"visibility": "public",
"digest": "7b0c0cd47fec183cfd1acdcec89beff50bcfe5c241c3587a6ee62f4e3a286f4fdf1d507527ad136346fb99fc02ef85dae7ca91b6486b9f70777f901caf3cef67",
"algorithm": "sha512",
"filename": "rustc-1.1.0-i686-pc-windows-gnu.tar.xz"
}
]

Comment 4

[Ralph Giles (:rillian)]

(In reply to Alex Crichton [:acrichto] from comment #2)
> To confirm, this basically involves getting a compiler that can target
> i686-pc-windows-msvc as a target triple, right?

I believe so. We'll also need x86_64-pc-windows-msvc, but it looks like that's a separate target and can happen independently of i686.

Comment 5

[Alex Crichton [:acrichto]]

Ok! We currently have nightlies available for x86_64-pc-windows-msvc and official release should start trickling down now (as the current releases just don't have support for MSVC).

There are still a number of LLVM bugs we're working through to get i686 MSVC nightlies, and you can track that in Rust here: https://github.com/rust-lang/rust/issues/26602

Comment 6

[Ralph Giles (:rillian)]

Thanks, Alex. I'll experiment with the x86_64-pc-windows-msvc nightlies, and watch for i686 ones.

re comment #3, our windows xp build machines don't support tar.xz. The windows mulet build doesn't invoke tooltool, just like the linux64 mulet build.

Comment 7

[Ralph Giles (:rillian)]

Today's 1.3.0-nightly:

[
{
"size": 78870112,
"visibility": "public",
"digest": "60766b469c35e2e52b205149f16f0a27bf734250939190b5380e4650e7db74f04497c0da7810a115e5a3ce18a0ad314a773cb453d53d8382ea5193db9bc2f519",
"algorithm": "sha512",
"filename": "rustc-nightly-x86_64-pc-windows-msvc.tar.gz"
}
]

Comment 8

[Alex Crichton [:acrichto]]

As an update, we now have i686-pc-windows-msvc nightlies available:

* https://static.rust-lang.org/dist/rustc-nightly-i686-pc-windows-msvc.tar.gz
* https://static.rust-lang.org/dist/rust-nightly-i686-pc-windows-msvc.msi
* https://static.rust-lang.org/dist/rust-nightly-i686-pc-windows-msvc.exe

This should hopefully do the trick for targeting XP!

Comment 9

[Ralph Giles (:rillian)]

Attachment: Part 1 - Add windows rustc to tooltool

Repackaged upstream builds of 1.5.0 stable release targetting msvc.

Comment 10

[Ralph Giles (:rillian)]

Attachment: Part 2 - Enable rust for windows nightly builds.

Comment 11

[Ralph Giles (:rillian)]

Attachment: Part 3 - Share mozconfig.rust

Remove redundancy.

Comment 12

[Ralph Giles (:rillian)]

This patchset fails because the spidermonkey configure script can't find rustc. Maybe the RUSTC enviroment variable isn't being propagated. Any idea what's up here?

From e.g. https://treeherder.mozilla.org/logviewer.html#?job_id=14642082&repo=try

 21:16:52 INFO - checking for rustc... c:/builds/moz2_slave/try-w32-d-00000000000000000000/build/src/rustc/bin/rustc 
  [...]
 21:17:36     INFO -  js\src> checking for rustc... no
 21:17:36     INFO -  js\src> configure: error: Rust compiler not found.

Comment 13

[(not currently active) Ted Mielczarek]

Yeah, there's a list of things that get exported for the JS subconfigure in the top-level configure.in:
https://dxr.mozilla.org/mozilla-central/source/configure.in#9256

You can either add RUSTC to that list, or you can change the in-tree mozconfig to `export RUSTC=...`.

Comment 14

[Ralph Giles (:rillian)]

Attachment: Part 4 - Export RUSTC from configure

Try Ted's idea of exporting RUSTC. This allows configure to complete and we fail somewhere around linking ICU?

https://treeherder.mozilla.org/#/jobs?repo=try&revision=75a409f0c717

Comment 15

[(not currently active) Ted Mielczarek]

The log parser didn't pick it up (sadfaces) but if you look at the log you'll see:
```
14:04:06     INFO -  c:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/rustc/bin/rustc -O --crate-type staticlib --emit dep-info=.deps/libMP4Metadata.lib.pp,link=libMP4Metadata.lib c:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/media/libstagefright/binding/MP4Metadata.rs
14:04:06     INFO -  PATH=../lib:../stubdata:../tools/ctestfw:$PATH  ../bin/genrb --usePoolBundle -kR -i ./out/build/icudt56l -s c:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/intl/icu/source/data/locales -d ./out/build/icudt56l jgo_CM.txt
14:04:06     INFO -  c:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/media/libstagefright/binding/MP4Metadata.rs:1:1: 1:1 error: can't find crate for `std` [E0463]
14:04:06     INFO -  c:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/media/libstagefright/binding/MP4Metadata.rs:1 //! Module for parsing ISO Base Media Format aka video/mp4 streams.
14:04:06     INFO -                                                                                                              ^
14:04:06     INFO -  error: aborting due to previous error
```

I don't know why that didn't fail the build and it hung, that's pretty unfortunate (and we should try to figure out why).

Comment 16

[Ralph Giles (:rillian)]

Ah, thanks for tracking down the real error, Ted. Not being able to find libstd should be straightforward to track down.

I'll be away for the next week. If someone else wants to tackle getting this working in the meantime, please feel free.

Comment 17

[Nathan Froyd [:froydnj]]

I'm going to poke at the Windows build failiures.

Comment 18

[Ralph Giles (:rillian)]

\o/ Let me know if you get stuck.

Comment 19

[Nathan Froyd [:froydnj]]

The rustc packages in tooltool don't have the .rlib files the compiler expects to find in $SYSROOT/bin/rustlib/$TARGET/lib.  So we'll have to generate new tooltool packages, either by rolling everything together, or packaging the sysroot files separately.

There's also some problem where the necessary files don't appear on the private tooltool server, which makes various spidermonkey builds fail.  I don't understand all the details here...

Comment 20

[Nathan Froyd [:froydnj]]

Ah, we also have the problem that the 32-bit MSVC ABI isn't supported until Rust 1.7.  But that's just what I can tell from the Rust download page.  It's possible that 1.6 (or even 1.5) have support, it's just not enabled in the official builds...?  (Though to be honest, I'm sure the underlying LLVM MSVC ABI support has gotten more stable over time anyway, so we might want to wait for 1.7.)

Comment 21

[Nathan Froyd [:froydnj]]

Hm, there are standard library packages with the MSVC ABI, though!  Maybe those are worth trying out.

Comment 22

[Alex Crichton [:acrichto]]

Ah note that the 32-bit MSVC standard library and compiler have been available for some time now (e.g. it's on all of the stable, beta, and nightly channels). The target, however, does not currently support unwinding (e.g. panic). Panics will translate to a process abort. This support is being implemented in https://github.com/rust-lang/rust/pull/30448, but that will not likely land until 1.8.

Comment 23

[Nathan Froyd [:froydnj]]

Ok, so new build uploaded:

{
"size": 26377309,
"digest": "334dd6f4fbb3b9d1fd67bc05e8b0cdbea9d0b527f32e0b3e792492a4359597a1ada14be201d3ece689ad69d0641127dbb2aa5b2ab22ebef1b0f5f89e82d32c4f",
"algorithm": "sha512",
"filename": "rustc.tar.bz2",
"unpack": true
}

This build combined the rustc package and the bin/ directory of the rust-std package, so it's quite a bit larger.  I'm not sure that's the best strategy in the long run; I think we may want to have separate tooltool packages for the compiler and the standard library on whatever platform we're targeting.

Just trying win32 for now:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7a91b110c6c1

The rustc invocations now work (yay!).  But the linking step fails, because xul.dll is linked with /SAFESEH and the Rust objects don't have the required information in them.  Alex, I assume this information is coming along for the ride in the 1.8 work you linked?

Comment 24

[Alex Crichton [:acrichto]]

That's a good question! This is the first time I've at least heard of SAFESEH, but given that error and the changes here I suspect that it does indeed mean that the Rust object files doesn't have this listed. 

Along those lines I'm not actually sure if the PR I linked will help the linker succeed. Looking at LLVM's history there's commits dating pretty far back which are entitled "Ensure that objects produced by LLVM link with /safeseh", so this may have to be something we need to manually opt into one way or another.

I'll try to do some tests locally to confirm one way or another, but it may indeed be the case that the PR will make everything "Just Work"

Comment 25

[Ralph Giles (:rillian)]

Seems to work on x64. I suggest we enable that while waiting for the SAFESEH. Only the opt build though:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d5f5a649cd0

Comment 26

[Nathan Froyd [:froydnj]]

(In reply to Ralph Giles (:rillian) from comment #25)
> Seems to work on x64. I suggest we enable that while waiting for the
> SAFESEH. Only the opt build though:
> 
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d5f5a649cd0

Yep, that's what bug 1243037 is for.  The patches in that bug enable it for debug builds as well.

Comment 27

[Ralph Giles (:rillian)]

Just to confirm, the unwind fixes do resolve the SAFESEH issue. Win32 build works fine with upstream rustc 1.8 nightly builds.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ead086e31aab

Comment 28

[Ralph Giles (:rillian)]

This was resolved through other bugs. See bug 1258796 and bug 1243037.