Closed Bug 1760743 Opened 3 years ago Closed 3 years ago

audioipc fails to build with distributed sccache and rust version 1.5.9, with "error: extern location for audio_thread_priority does not exist: ...libaudio_thread_priority-1d4485254d6d65b9.rlib

Categories

(Firefox Build System :: Toolchains, defect, P2)

Product:
Firefox Build System
Component:
Toolchains
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dholbert, Assigned: gerard-majax)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

build error (the full output of ./mach build -j1 after having previously encountered the build error)
6.82 KB, text/plain
Details

tl;dr: it's currently impossible to compile Firefox using distributed sccache with latest stable rustc, version 1.59, due to a build error with audioipc2 -- though everything works if you downgrade to e.g. 1.58.1 (or earlier).

STR:

  1. Set up a distributed sccache environment (note: you can do this all on a single linux machine, with sccache "client" and scheduler and distributed-build-server all on that one machine. I confirmed that this setup is sufficient to trigger the bug.)

  2. Run rustup update to get rust v1.59 (current latest), and then rustup default stable or rustup 1.59 (which are currently equivalent)

  3. clobber your objdir and your sccache cache directory to be sure you don't benefit from any prebuilt binaries

  4. ./mach build

EXPECTED RESULTS:
Successful build.

ACTUAL RESULTS:
Your build eventually fails, with output like the following:

toolkit/library/rust/force-cargo-library-build
Compiling audioipc2 v0.5.0 (https://github.com/kinetiknz/audioipc-2?rev=8e3c4c1b1edbd19cfafc833c3254f12731af31c4#8e3c4c1b)
error: extern location for audio_thread_priority does not exist: /scratch/work/builds/mozilla-central/obj/x86_64-unknown-linux-gnu/debug/deps/libaudio_thread_priority-1d4485254d6d65b9.rlib
--> /scratch/work/builds/mozilla-central/mozilla/third_party/rust/audioipc2/src/messages.rs:9:5
|
9 | use audio_thread_priority::RtPriorityThreadInfo;
| ^^^^^^^^^^^^^^^^^^^^^
error[E0412]: cannot find type RtPriorityThreadInfo in this scope
--> /scratch/work/builds/mozilla-central/mozilla/third_party/rust/audioipc2/src/messages.rs:231:54
|
204 | pub enum ServerMessage {
| - help: you might be missing a type parameter: <RtPriorityThreadInfo>
...
231 | PromoteThreadToRealTime([u8; std::mem::size_of::<RtPriorityThreadInfo>()]),
| ^^^^^^^^^^^^^^^^^^^^ not found in this scope
error[E0412]: cannot find type RtPriorityThreadInfo in this scope
--> /scratch/work/builds/mozilla-central/mozilla/third_party/rust/audioipc2/src/messages.rs:231:54
|
231 | PromoteThreadToRealTime([u8; std::mem::size_of::<RtPriorityThreadInfo>()]),
| ^^^^^^^^^^^^^^^^^^^^ not found in this scope
For more information about this error, try rustc --explain E0412.
sccache: Compiler killed by signal 1
error: could not compile audioipc2 due to 3 previous errors

If I adjust the STR to use rustup default 1.58.1, then the issue goes away. So I think this is something specific to rust 1.59; not sure if it's a regression or an intentional increase in strictness of some sort.

Note: I'm building today's mozilla-central, rev https://hg.mozilla.org/mozilla-central/rev/c15abadbc3c7 , with --enable-debug --disable-optimize, though I tried older commits (e.g. the push before/after bug 1726275 landed) and other build configurations and I reproduced this in all of the commits/build-configs that I tested. It's only downgrading rust (to pre-1.5.9 e.g. to version 1.58.1) that fixes the issue.

kinetik, I'm hoping you can take a look here, since it looks like audioipc2 is your domain. I'm happy to test out any suggested tweaks or help you get a distributed sccache build set up yourself to test locally, if that's useful.

Flags: needinfo?(kinetik)

(sorry, I was off-by-one in my component picking -- I think this belongs in A/V: cubeb based on bug 1726275.)

(Also, it looks like this rust library is no stranger to version-specific rustc issues, based on bug 1726275 comment 13. :) )

Component: Audio/Video: GMP → Audio/Video: cubeb
Summary: audioipc fails to build with distributed sccache and rust version 1.5.9, with " error: extern location for audio_thread_priority does not exist: ...libaudio_thread_priority-1d4485254d6d65b9.rlib → audioipc fails to build with distributed sccache and rust version 1.5.9, with "error: extern location for audio_thread_priority does not exist: ...libaudio_thread_priority-1d4485254d6d65b9.rlib

Two quick questions:

  • can you build without sccache successfully?
  • is your sccache configuration still using the correct rustc versions after updating your default to 1.59?

My local build is fine with rustc 1.59. I'll set up sccache locally when I have a minute and see if I can repro.

Flags: needinfo?(kinetik)

(In reply to Matthew Gregan [:kinetik] from comment #3)

Two quick questions:

  • can you build without sccache successfully?

Yes; I can even build with "non-distributed" sccache (the simplest configuration).

It's only "distributed" sccache (with a scheduler and potentially-remote build servers) where I'm running into this problem.

  • is your sccache configuration still using the correct rustc versions after updating your default to 1.59?

I think/assume so, yeah. I think sccache (and sccache-dist) bundles/distributes the rustc toolchain transparently.

Note that I didn't just update to 1.59 today; I think I've had that version since around when it was released a few weeks back. I just happened to try distributed sccache again today (after not using it for a few months) and I ran into this issue, and I determined that downgrading my rustc is the thing that appears to fix it.

My local build is fine with rustc 1.59. I'll set up sccache locally when I have a minute and see if I can repro.

Thanks! You'll need to set it up with a scheduler and a build server (which can all be the same machine) in order to actually run into this, AFAICT. I'll let you know if I find another simpler way to repro, though.

(Looking at the 1.59 release notes at https://blog.rust-lang.org/2022/02/24/Rust-1.59.0.html , I'm wondering if this might be related to the Incremental compilation off by default change noted there; none of the other highlighted changes seems like it might cause this. I'm trying a clean rebuild with my normally-failing setup and with the env variable noted there to reenable incremental compilation, in case that helps -- though I won't have a chance to look at the results until morning in my timezone.)

Blocks: 1362220
Severity: -- → S4
Priority: -- → P2

(In reply to Daniel Holbert [:dholbert] from comment #5)

(Looking at the 1.59 release notes at https://blog.rust-lang.org/2022/02/24/Rust-1.59.0.html , I'm wondering if this might be related to the Incremental compilation off by default change noted there

Nope; poking at this hasn't turned up a functional workaround yet. I tried the workaround noted in the release notes (exporting RUSTC_FORCE_INCREMENTAL=1 in my .bash_aliases on the machine that I'm using as client and also build-server) but that doesn't seem to have helped. Though, I'm not sure how the build env gets set up on the builder (via bubblewrap), so it's possible that my export (in .bash_aliases) doesn't get picked up in (or transferred through to) that environment.

I can reproduce this locally (with 1.59 and nightly; 1.58.1 works), but can't make much sense of it. When audioipc2 is compiled, rustc acts as if the input for audio_thread_priority doesn't exist:

error: extern location for audio_thread_priority does not exist: /home/mjg/src/mozilla/central/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/deps/libaudio_thread_priority-4330911a01b8fd44.rlib

...but that file is accessible and valid (works fine in non-sccache build).

The rustc error originates here. Using strace on the sccache-dist server's rustc process, I see:

statx(AT_FDCWD, "/home/mjg/src/mozilla/central/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/deps/libaudio_thread_priority-4330911a01b8fd44.rlib", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7f73c1bed680) = -1 ENOENT (No such file or directory)

The next statx is:

statx(AT_FDCWD, "/home/mjg/src/mozilla/central/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/deps/libcubeb-7b52a1a258191860.rmeta", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=102936, ...}) = 0

...which succeeds, and seems no different in terms of access/etc. to libaudio_thread_priority-4330911a01b8fd44.rlib.

If I had to guess, this might be a sandboxing issue in the sccache server. I have no explanation for why audioipc2 triggers this, though.

Turns out rustup lets me test individual Nightly versions pretty easily, so I bisected this to a one-day range of rustc nightlies.

(One clarifying note to avoid confusion on datestamps: it seems that the datestamp from rustc --version is typically one day older than the date that I provide to rustup; I assume this is just a timezone thing or due to the build process.)

LAST GOOD rustc nightly:

$ rustup default nightly-2021-12-02
$ rustc --version
rustc 1.59.0-nightly (48a5999fc 2021-12-01)

FIRST BAD rustc nightly:

$ rustup default nightly-2021-12-03
$ rustc --version
rustc 1.59.0-nightly (acbe4443c 2021-12-02)

So, for rustc (just bisecting / looking at regression range to help get closer to identifying/understanding the issue), we have:
last-known-good commit: https://github.com/rust-lang/rust/commit/48a5999fc
first-known-bad commit: https://github.com/rust-lang/rust/commit/acbe4443c

And if I'm doing it right, I think this shows the commits in the regression range:
https://github.com/rust-lang/rust/compare/48a5999fc...acbe4443c

There are a 3 commits in that range about extern crate:

I suspect one of those might be involved, since the error message here is about an extern crate?

(In reply to Mike Hommey [:glandium] from comment #10)

You can try https://github.com/rust-lang/cargo-bisect-rustc to bisect further.

Thanks. Using that tool to populate my available toolchains (and then selected the current toolchain as the default, and built Firefox, clobbering my objdir and sccache-dir each time).

From testing those builds, it actually seems that the artifact for commit acbe4443c is in fact "good" (supposedly that was the first-bad one per comment 8 - comment 9, based on rust-nightly testing & a nightly that advertised itself as using that commit). bors's immediately-following merge is in fact the first-bad commit when I test using cargo-bisect-rustc:
https://github.com/rust-lang/rust/commit/ff23ad3179014ba258f2b540fb39dd0f26852b7a

Hmm, I'm not sure I trust my cargo-bisect-rustc results - I suspect the tool might've gotten the wrong builds, or perhaps I made a mistake in using it.

For extra assurance, I compiled rustc locally with a handful of bors revisions, and got the following results (going through consecutive bors merges):
first-known-bad: https://github.com/rust-lang/rust/commit/e5038e20999eef35260b070189883edc2a8a34b2
broken/won't-build: https://github.com/rust-lang/rust/commit/dd03c0ff3e203aad854004b6c47adcf8c001129d
last-known-good: https://github.com/rust-lang/rust/commit/18bb8c61a975fff6424cda831ace5b0404277145

Notably, the first-known-bad commit there is a merge of the commits about extern stuff from comment 9.

Next, I tried updating to my first-known-bad commit (https://github.com/rust-lang/rust/commit/e5038e20999eef35260b070189883edc2a8a34b2 ) and doing git revert 2ca933301183b1e913802bc5710c0cb5218ada28 to undo the changes from the latest extern-related commit, and I rebuilt and tested that revision.

That got me back to a "good" state.

So unless I've made a mistake here, this seems to be a regression from https://github.com/rust-lang/rust/commit/2ca933301183b1e913802bc5710c0cb5218ada28 (which is ostensibly about improving suggestions, but does seem to make a lot of code changes as part of that).

Further proof that 2ca933301183b1e913802bc5710c0cb5218ada28 is the guilty commit:

If I build current "beta" rustc locally (using a checkout updated to "beta" branch, which reports itself as v1.60.0-dev after being built), then I hit this issue, but if I git revert these two commits...
https://github.com/rust-lang/rust/commit/2ca933301183b1e913802bc5710c0cb5218ada28
https://github.com/rust-lang/rust/commit/62f4ce993e8aa788c8269d75aea2db84cafb1f9f

...then I can build successfully.

(If I follow this same process but only revert 2ca933301183b1e913802bc5710c0cb5218ada28, then I get an unrelated build issue. I have to revert both 2ca933301183b1e913802bc5710c0cb5218ada28 and 62f4ce993e8aa788c8269d75aea2db84cafb1f9f in order to build successfully. Note also that 62f4ce993e8aa788c8269d75aea2db84cafb1f9f doesn't quite cleanly revert - it hits a merge conflict in a testcase's .stderr expectation file, but I think that can be resolved trivially for local investigation purposes.)

aha, so my theory is that this was always an issue when building with distributed sccache, but it used to be a failure that would gracefully fail-over to a local compile (i.e. it'd be a "failed distributed compilation" as reported by sccache -s).

Whereas now it causes an error that doesn't fail over to a local compile (possibly even a "less severe" error, which unfortunately isn't severe enough to prompt sccache to retry locally?)

I say this because: when I build with "good" rustc version 1.58.1, I nonetheless see this build output from sccache-dist process on the remote build server (which I've run with SCCACHE_NO_DAEMON=1 RUST_LOG=sccache=trace in order to see its error output):

[2022-03-26T23:06:59Z TRACE sccache_dist::build] compile_output: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "{"message":"extern location for audio_thread_priority does not exist: /scratch/work/builds/mozilla-central/obj-debug-opt/x86_64-unknown-linux-gnu/debug/deps/libaudio_thread_priority-1bf7305fe59ad3bd.rlib","code":null,"level":"error","spans":[{"file_name":"/scratch/work/builds/mozilla-central/mozilla/third_party/rust/audioipc-client/src/context.rs","byte_start":310,"byte_end":331,"line_start":10,"line_end":10,"column_start":5,"column_end":26,"is_primary":true,"text":[{"text":"use audio_thread_priority::get_current_thread_info;","highlight_start":5,"highlight_end":26}],"label":null,"suggested_replacement":null,"suggestion_applicability":null,"expansion":null}],"children":[],"rendered":"\u001b[0m\u001b[1m\u001b[38;5;9merror\u001b[0m\u001b[0m\u001b[1m: extern location for audio_thread_priority does not exist: /scratch/work/builds/mozilla-central/obj-debug-opt/x86_64-unknown-linux-gnu/debug/deps/libaudio_thread_priority-1bf7305fe59ad3bd.rlib\u001b[0m\n\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m--> \u001b[0m\u001b[0m/scratch/work/builds/mozilla-central/mozilla/third_party/rust/audioipc-client/src/context.rs:10:5\u001b[0m\n\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m|\u001b[0m\n\u001b[0m\u001b[1m\u001b[38;5;12m10\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m| \u001b[0m\u001b[0muse audio_thread_priority::get_current_thread_info;\u001b[0m\n\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m| \u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;9m^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n\n"}\n{"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":"\u001b[0m\u001b[1m\u001b[38;5;9merror\u001b[0m\u001b[0m\u001b[1m: aborting due to previous error\u001b[0m\n\n"}\n" }

Notice the text towards the beginning: extern location for audio_thread_priority does not exist which is precisely the error that's causing trouble here. But in this case with rustc v1.58.1, my overall Firefox build completes successfully (and this error doesn't appear in my local ./mach build output at all).

I'm hitting that as well now

First there was an issue with RlibDepReader failing to parse newer output (probably here for a few time), filed upstream as https://github.com/mozilla/sccache/issues/1156 and with PR https://github.com/mozilla/sccache/pull/1157

Second it looks like the logic to perform rlib trimming in https://github.com/mozilla/sccache/blob/5edd91a739df0f51f260766d7f68f6f59a89d56e/src/compiler/rust.rs#L1891-L1899 might only care about the crate we are building and not its dependencies ; however we have three cases at least in m-c that have both staticlib and rlib defined: audio_thread_priority, cubeb-coreaudio and cubeb_pulse:

Verifying if a .rlib input path that is being shipped as a dependency can be effectively trimmed seems to fix the issue.

Assignee: nobody → lissyx+mozillians
URL: https://github.com/mozilla/sccache/pu...
Component: Audio/Video: cubeb → Toolchains
Product: Core → Firefox Build System
Version: unspecified → Trunk
Status: NEW → ASSIGNED

Mitchell, I assume you will want to use this bug for performing sccache version bump when it is ready?

Flags: needinfo?(mhentges)

There's going to be other bug impact as part of the v0.3.0 version bump, so I'm going to have a dedicated bug instead.

Flags: needinfo?(mhentges)

The underlying issue is fixed, if people find this bug it means in-tree sccache has not been updated yet.

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Depends on: 1769335

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #20)

There's going to be other bug impact as part of the v0.3.0 version bump, so I'm going to have a dedicated bug instead.

Update: It doesn't look like we ever ended up with a bug filed for this, so I filed bug 1769335 with a patch.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: