1816588 - `--enable-thread-sanitizer` (TSAN) build fails with `error: no matching package named `compiler_builtins` found`

Status: NEW

(Firefox Build System :: General, defect, P3)

Description

[Daniel Holbert [:dholbert]]

STR:

1. Install the Rust nightly toolchain, and make it your default (this is required to build with TSAN):

rustup toolchain default nightly
rustup default nightly

2. Start a Firefox clobber-build, with this in your mozconfig (or this being your entire mozconfig):

ac_add_options --enable-thread-sanitizer

EXPECTED RESULTS:
Successful build.

ACTUAL RESULTS:
My build errors out pretty early, with:

error: no matching package named `compiler_builtins` found

If I run `./mach build -j1 -vvv, I get this slightly clearer output:

0:02.58 toolkit/library/rust/force-cargo-library-build
0:02.58 /home/dholbert/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo rustc --release --frozen --manifest-path /scratch/work/builds/mozilla-central/mozilla/toolkit/library/rust/Cargo.toml -vv --color=always -j1 -Zbuild-std=std,panic_abort --lib --target=x86_64-unknown-linux-gnu --features 'gkrust-shared/thread_sanitizer gkrust-shared/cubeb_pulse_rust gkrust-shared/cubeb-remoting gkrust-shared/moz_memory gkrust-shared/moz_places gkrust-shared/gecko_profiler gkrust-shared/gecko_profiler_parse_elf gkrust-shared/webrtc gkrust-shared/glean_with_gecko gkrust-shared/glean_disable_upload gkrust-shared/glean_million_queue gkrust-shared/with_dbus gkrust-shared/oxidized_breakpad gkrust-shared/webmidi_midir_impl' --
0:04.69 error: no matching package named compiler_builtins found
0:04.69 location searched: registry crates-io
0:04.69 required by package std v0.0.0 (/home/dholbert/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std)

Note: I tried building with a larger mozconfig based on https://firefox-source-docs.mozilla.org/tools/sanitizer/tsan.html , but I was still blocked by this same error; and it seems that the --enable-thread-sanitizer build option is necessary & sufficient to trigger the issue.

I'm using Ubuntu 22.04, for what it's worth.

Note that we do have TSAN builds on TreeHerder, so presumably these builds can be successful, but I must be missing some bit of configuration, or maybe there's been a recent regression somewhere that's hitting me that hasn't bit TreeHerder yet.

Comment 1

[Daniel Holbert [:dholbert]]

decoder, I wonder if you've seen this before or know what it's about, in your work with TSAN builds?

Comment 2

[Daniel Holbert [:dholbert]]

(or if you know who to point me at, to find a way forward. My goal here is to try to reproduce bug 1770595 locally in rr, which requires a locally-built tsan build, ideally.)

Comment 3

[Mike Hommey [:glandium]]

Try emptying .cargo/config.in

Comment 4

[Daniel Holbert [:dholbert]]

(In reply to Mike Hommey [:glandium] from comment #3)

Try emptying .cargo/config.in

emptying how?

I tried four things:
(1) If I delete that file, my build errors out with
File listed in OBJDIR_PP_FILES does not exist: /scratch/work/builds/mozilla-central/mozilla/.cargo/config.in

(2) If I clear the contents of that file (but leave it there as an empty file), my build errors out with
mozbuild.preprocessor.Preprocessor.Error: ('/scratch/work/builds/mozilla-central/mozilla/.cargo/config.in', None, 'no useful preprocessor directives found', None)

OK, it needs some "useful" preprocessor directives.

(3) If I remove all of the package-specific stuff up to the #define REPLACE_NAME vendored-sources (a "useful" preprocessor directive), my build errors out with

 0:16.97 error: failed to download `adler v1.0.2`
 0:16.97 Caused by:
 0:16.97   attempting to make an HTTP request, but --frozen was specified

(4) If I then add back just the first two lines...

[source.crates-io]
replace-with = "vendored-sources"

...but leave out everything else, then I'm back to the compiler_builtins error from comment 0.

Comment 5

[Mike Hommey [:glandium]]

Try wrapping the entire thing in #if 0/#endif

Comment 6

[Daniel Holbert [:dholbert]]

That gives me the same error output that I noted under (3) above: "error: failed to download adler v1.0.", with "attempting to make an HTTP request, but --frozen was specified"

(Maybe I need to un-specify --frozen? But I don't know where to do that.)

Comment 7

[Mike Hommey [:glandium]]

Yeah, try removing --frozen from https://searchfox.org/mozilla-central/source/build/RunCbindgen.py#48 and https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#27

Comment 8

[Daniel Holbert [:dholbert]]

Attachment: manual hackaround (using dynamically downloaded rust packages, I think)

That seems to do it, yeah. Here's my full diff.

Presumably our TreeHerder TSAN builds don't do this, though; I wonder why this works there but not for me? Are they building with an older/different rust nightly build or something?

Comment 9

[Mike Hommey [:glandium]]

We're using a patched cargo because upstream cargo can't handle -Zbuild-std while using vendored crates.

Comment 10

[Daniel Holbert [:dholbert]]

OK. Clearing needinfo for choller since I think you've answered my questions and I can build now -- thanks!

Ideally we should still figure out what can be done to avoid this footgun though -- hence, leaving this bug open. (I don't know enough about cargo, the build system, etc. to know if this compiler_builtins failure is something where we need to re-vendor some package vs. do something else.)

Feel free to dupe or close as-appropriate, though, as long as this is tracked as a known issue somewhere/somehow.

Comment 11

[Mike Hommey [:glandium]]

That's basically unfixable on our end. https://github.com/rust-lang/cargo/pull/8834 and https://github.com/rust-lang/rust/pull/78790 were an attempt that was backed out from upstream (and our cargo/rustc are patched with that)

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:ahochheiden, could you have a look please?

For more information, please visit auto_nag documentation.