Closed Bug 1533213 Opened 5 years ago Closed 4 years ago

libxul does not load with address-sanitizer enabled for rust.

Categories

(Firefox Build System :: Toolchains, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: truber, Unassigned)

References

Details

Enabling address-sanitizer in RUSTFLAGS works with standalone programs. When I try to add it to my mozconfig (linux64-fuzzing-asan-opt), I get the following error on launch:

XPCOMGlueLoad error for file /home/user/builds/firefox/libxul.so:
/home/user/builds/firefox/libxul.so: cannot allocate memory in static TLS block
Couldn't load XPCOM.

Complete mozconfig:

export CC="~/.mozbuild/clang/bin/clang"
export CXX="~/.mozbuild/clang/bin/clang++"
export RUSTFLAGS="-Z sanitizer=address"

mk_add_options AUTOCLOBBER=1

export MOZ_WEBRTC_TESTS=1
ac_add_options --enable-tests
ac_add_options --enable-address-sanitizer
ac_add_options --disable-jemalloc
ac_add_options --disable-crashreporter

export MOZ_LLVM_HACKS=1
export MOZ_DEBUG_SYMBOLS=1
ac_add_options --enable-debug-symbols
ac_add_options --disable-install-strip

ac_add_options --enable-fuzzing

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj/ff-asan-release
ac_add_options --enable-optimize="-O2 -gline-tables-only"
ac_add_options --enable-application=browser
ac_add_options --disable-debug

Clang: is 7.0.1 from taskcluster linux64-clang-7.
Rustc: 1.34.0-nightly from rustup.

What happens if your mozconfig specifies ASan for rust but not C++?

I wouldn't be surprised if this is the result of some collision between two separate ASan runtimes.

It more sounds like enabling asan on rust adds enough TLS data that we're going past some threshold.

I took a look at a build that :posidron provided me, and it seems the problem is caused by libstdc++ used on the system where you're compiling (at least in his case), which triggers static TLS to be required, and there's too much TLS data for that to work.

My advice would be to try to build on automation rather than locally.

Blocks: 1563033
Blocks: 1563019

Here's a try build:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=259431011&revision=e3211a8442576b8f52823c9395d7102647bc2078

Same issue as my local build:

XPCOMGlueLoad error for file /home/user/builds/firefox/libxul.so:
/home/user/builds/firefox/libxul.so: cannot allocate memory in static TLS block
Couldn't load XPCOM.

(In reply to :dmajor from comment #1)

I wouldn't be surprised if this is the result of some collision between two separate ASan runtimes.

This analysis is correct, the problem is that there are multiple ASan runtimes involved:

$ nm firefox | grep AsanCheckFailed
00000000000fcef0 t _ZN6__asanL15AsanCheckFailedEPKciS1_yy
000000000034e090 b _ZZN6__asanL15AsanCheckFailedEPKciS1_yyE9num_calls
$ nm libxul.so | grep AsanCheckFailed
000000001a8b46e0 t _ZN6__asanL15AsanCheckFailedEPKciS1_yy
0000000021c8ed80 b _ZZN6__asanL15AsanCheckFailedEPKciS1_yyE9num_calls

This is caused by the Rust compiler adding the ASan runtime unconditionally to static archives and DSOs, instead of only adding it when actually linking executables. This needs to be fixed in the Rust compiler and I've opened https://github.com/rust-lang/rust/issues/64629 to do so.

Depends on: 1583471
Depends on: 1587774
Type: enhancement → defect

decoder says this might be fixed in latest nightly rust. Setting a reminder for myself to try it out.

Flags: needinfo?(dmajor)

This works now, with nightly-2020-04-04 (and maybe earlier). \o/

Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(dmajor)
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.