Closed Bug 1417268 Opened 2 years ago Closed 2 years ago

win32 opt compile failures with rust 1.22.0-beta.3

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: rillian, Unassigned)

References

Details

Optimized builds of Firefox for 32-bit Windows fail with Rust 1.22.0 beta.

22:10:26     INFO -       Running `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name gkrust_gtest 'Z:\build\build\src\toolkit\library\gtest\rust\lib.rs' --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C lto --cfg 'feature="bindgen"' --cfg 'feature="gkrust-shared"' --cfg 'feature="no-static-ideograph-encoder-tables"' --cfg 'feature="quantum_render"' --cfg 'feature="servo"' --cfg 'feature="simd-accel"' -C metadata=a3361067bcf39900 -C extra-filename=-a3361067bcf39900 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps' --target i686-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'gkrust_shared=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libgkrust_shared-8c5167ebec10d73e.rlib' --extern 'nsstring_gtest=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libnsstring_gtest-f97831b3ce63fe44.rlib' --extern 'mp4parse_gtest=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libmp4parse_gtest-9a029b3ee5e18bdd.rlib' -C opt-level=2 -C debuginfo=2`
22:10:26     INFO -  error: Could not compile `gkrust-gtest`.
22:10:26     INFO -  Caused by:
22:10:26     INFO -    process didn't exit successfully: `z:/build/build/src/sccache2/sccache.exe z:/build/build/src/rustc/bin/rustc.exe --crate-name gkrust_gtest Z:\build\build\src\toolkit\library\gtest\rust\lib.rs --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C lto --cfg feature="bindgen" --cfg feature="gkrust-shared" --cfg feature="no-static-ideograph-encoder-tables" --cfg feature="quantum_render" --cfg feature="servo" --cfg feature="simd-accel" -C metadata=a3361067bcf39900 -C extra-filename=-a3361067bcf39900 --out-dir z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps --target i686-pc-windows-msvc -L dependency=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps -L dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps --extern gkrust_shared=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libgkrust_shared-8c5167ebec10d73e.rlib --extern nsstring_gtest=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libnsstring_gtest-f97831b3ce63fe44.rlib --extern mp4parse_gtest=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libmp4parse_gtest-9a029b3ee5e18bdd.rlib -C opt-level=2 -C debuginfo=2` (exit code: 3221225477)
22:10:26     INFO -  z:/build/build/src/config/rules.mk:952: recipe for target 'force-cargo-library-build' failed
22:10:26     INFO -  mozmake.EXE[4]: *** [force-cargo-library-build] Error 101
(In reply to Ralph Giles (:rillian) | needinfo me from comment #1)
> See
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=0c0f28adb31bb6c188198566190ca318955e9b98 for details.

This seems really bad:

22:10:26     INFO -  warning: this expression will panic at run-time
22:10:26     INFO -       --> z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\build\style-41824e9947f11a7e\out/gecko_properties.rs:22798:13
22:10:26     INFO -        |
22:10:26     INFO -  22798 |             self.gecko.mWindowTransformOrigin[2].set(v.depth);
22:10:26     INFO -        |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ index out of bounds: the len is 2 but the index is 2
22:10:26     INFO -  warning: this expression will panic at run-time
22:10:26     INFO -       --> z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\build\style-41824e9947f11a7e\out/gecko_properties.rs:22807:13
22:10:26     INFO -        |
22:10:26     INFO -  22807 |             self.gecko.mWindowTransformOrigin[2].copy_from(&other.gecko.mWindowTransformOrigin[2]);
22:10:26     INFO -        |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ index out of bounds: the len is 2 but the index is 2
22:10:26     INFO -  warning: this expression will panic at run-time
22:10:26     INFO -       --> z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\build\style-41824e9947f11a7e\out/gecko_properties.rs:22807:61
22:10:26     INFO -        |
22:10:26     INFO -  22807 |             self.gecko.mWindowTransformOrigin[2].copy_from(&other.gecko.mWindowTransformOrigin[2]);
22:10:26     INFO -        |                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ index out of bounds: the len is 2 but the index is 2
22:10:26     INFO -  warning: this expression will panic at run-time
22:10:26     INFO -       --> z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\build\style-41824e9947f11a7e\out/gecko_properties.rs:22825:49
22:10:26     INFO -        |
22:10:26     INFO -  22825 |                 Length::from_gecko_style_coord(&self.gecko.mWindowTransformOrigin[2])
22:10:26     INFO -        |                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ index out of bounds: the len is 2 but the index is 2

Emilio, Xidorn, are either of you able to look at this?
Flags: needinfo?(xidorn+moz)
Flags: needinfo?(emilio)
This warning makes nonsense, and it shouldn't cause build failure. See bug 1417061 comment 2.

If Rust is regressing that on Windows... Rust team should really fix rust-lang/rust#45850 to allow us suppressing this warning.
Flags: needinfo?(xidorn+moz)
But generally, I don't think this warning is the reason that it fails to build, because it is code shared in all platforms, and if other platforms don't bust on this, Windows shouldn't either. It would be surprising if Rust has different error strategy for different platforms for this specific (unsuppressible) warning.
22:10:26     INFO -     Compiling gkrust-gtest v0.1.0 (file:///Z:/build/build/src/toolkit/library/gtest/rust)
22:10:26     INFO -       Running `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name gkrust_gtest 'Z:\build\build\src\toolkit\library\gtest\rust\lib.rs' --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C lto --cfg 'feature="bindgen"' --cfg 'feature="gkrust-shared"' --cfg 'feature="no-static-ideograph-encoder-tables"' --cfg 'feature="quantum_render"' --cfg 'feature="servo"' --cfg 'feature="simd-accel"' -C metadata=a3361067bcf39900 -C extra-filename=-a3361067bcf39900 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps' --target i686-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'gkrust_shared=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libgkrust_shared-8c5167ebec10d73e.rlib' --extern 'nsstring_gtest=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libnsstring_gtest-f97831b3ce63fe44.rlib' --extern 'mp4parse_gtest=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libmp4parse_gtest-9a029b3ee5e18bdd.rlib' -C opt-level=2 -C debuginfo=2`
22:10:26     INFO -  error: Could not compile `gkrust-gtest`.
22:10:26     INFO -  Caused by:
22:10:26     INFO -    process didn't exit successfully: `z:/build/build/src/sccache2/sccache.exe z:/build/build/src/rustc/bin/rustc.exe --crate-name gkrust_gtest Z:\build\build\src\toolkit\library\gtest\rust\lib.rs --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C lto --cfg feature="bindgen" --cfg feature="gkrust-shared" --cfg feature="no-static-ideograph-encoder-tables" --cfg feature="quantum_render" --cfg feature="servo" --cfg feature="simd-accel" -C metadata=a3361067bcf39900 -C extra-filename=-a3361067bcf39900 --out-dir z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps --target i686-pc-windows-msvc -L dependency=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps -L dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps --extern gkrust_shared=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libgkrust_shared-8c5167ebec10d73e.rlib --extern nsstring_gtest=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libnsstring_gtest-f97831b3ce63fe44.rlib --extern mp4parse_gtest=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libmp4parse_gtest-9a029b3ee5e18bdd.rlib -C opt-level=2 -C debuginfo=2` (exit code: 3221225477)
22:10:26     INFO -  z:/build/build/src/config/rules.mk:952: recipe for target 'force-cargo-library-build' failed
22:10:26     INFO -  mozmake.EXE[4]: *** [force-cargo-library-build] Error 101

This suggests rustc itself crashing in some way that prints out nothing.
... or sccache, for that matter. Ralph, can you do a windows-only try with sccache disabled?
Flags: needinfo?(giles)
Yeah, that warning is unfortunately expected :(
Flags: needinfo?(emilio)
It's not sccache. A try build with it disables fails with the same error code from rustc. https://treeherder.mozilla.org/#/jobs?repo=try&revision=3a231dea0a8990eeac5850ec68f4d003da8ad942
Flags: needinfo?(giles)
So we're back to rustc crashing in a way that prints nothing out. Ralph did you file it upstream already?
I spent a while trying to produce a better bug report, but mostly ended up fighting with windows. I've now filed rust-lang/rust#46045.
As someone already noted on the GitHub issue, on Windows exit codes for crashes are the exception code, and:
> (3221225477 = 0xc0000005 = STATUS_ACCESS_VIOLATION)
I've been unable to reproduce this locally. That and ACCESS_VIOLATION suggests it might be something specific to the filesystem on the automation images.
To summarize recent progress: the lto step on gkrust reaches 1778 MB rss before failing. We're using 32-bit Rust to build 32-bit Firefox because of bug 1401647. The access violation is probably running out of address space. That is consistent with the debug and 64-bit builds being fine, and with my not being able to reproduce locally using a 64-bit rustc.

Rust 1.23.0-nightly *does* work however, so this may be a symptom of extra memory use by llvm in the 1.22 release.
Depends on: 1401647
FYI ACCESS_VIOLATION on Windows is basically the same as SIGSEGV elsewhere.
Per IRC discussion, the plan is to block on this for the 1.22 update, which in turn block stylo on android via bug 1399337. If this isn't resolved by release, we plan to work around the issue by bumping the win32 builds to 1.23.0-beta so we can still bump the minimum rust version to 1.22 on schedule. Rust 1.23 will be in stable release near the end of the Firefox 59 nightly period.
I don't see why stylo on android needs to be blocked on the minimum supported rust version being 1.22. All it needs is to build with 1.22 (and it only needs that for code size, not for anything else aiui), it doesn't matter that the minimum supported version is bumped.
I agree. What I was trying to say is that the proposed workaround is compatible with each of those issues independently.
My point is that stylo on android doesn't need to block on this, while this blocks 1.22 for at least windows. What we can reasonaly do, and that would be less "controversial" than going with a beta for windows, is to keep all platforms on 1.21, but upgrade android to 1.22. Or upgrade all platforms to 1.22 except windows. Also note that if the problem we have is indeed adress space exhaustion on win32, we also have no guarantee that it won't happen again with newer versions of rust, it could even happen with newer 1.23 betas.
Why do we need 32bit Rust for building win32 at all? Shouldn't we already have been cross-compiling win32 build on win64 machines and thus we should be able to use 64bit Rust with 32bit std?
We're working around this issue buy building for win32 with Rust 1.23 beta during Firefox 59 nightly, starting with bug 1421100.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WONTFIX
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.