1754597 - mach build blocked for ~4.5s due to cargo "Blocking waiting for file lock on package cache"

Status: RESOLVED INVALID

(Firefox Build System :: General, defect)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

When I run an incremental ./mach build (with geckoview configured), I see the following output early on:

 0:03.96 netwerk/test/http3server/force-cargo-program-build
 0:04.03     Blocking waiting for file lock on package cache
 0:05.77     Blocking waiting for file lock on package cache
 0:08.43     Blocking waiting for file lock on package cache
 0:08.54     Finished dev [optimized + debuginfo] target(s) in 4.57s

It appears the build is blocked on cargo which is trying to access a lock file.

I found here that it can be caused by rust-analyzer but I'm able to reproduce the issue whether or not VSCode, with the rust-analyzer extension, is open.

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Hmm, interesting - I wonder if it is rust-analyzer, if you're not running VSCode.
Can you check to see if the rust-analyzer executable is running when you see this error?
If it's not, do you see other processes active that are rust-related?

We won't be able to change cargo's behaviour of only allowing a single process to lock on the package cache at a time, so I think the fix here will be to determine what else you have running locally that's occupying the lock, and turn it down/disable it.

Comment 2

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

During a build, I searched for rust and cargo processes with:

$ while true; do sleep 2; ps aux | grep -e rust -e cargo; echo ""; done

And, removing the irrelevant lines, I see the following around the time the lock file is read:

/Users/mcomella/.rustup/toolchains/stable-x86_64-apple-darwin/bin/cargo rustc --release --frozen --manifest-path /Users/mcomella/dev/mozilla-central/toolkit/library/rust/Cargo.toml --color=always --lib --target=thumbv7neon-linux-androideabi --features moz_memory gecko_profiler gecko_profiler_parse_elf new_xulstore webrtc glean_with_gecko glean_disable_upload --
/Users/mcomella/.rustup/toolchains/stable-x86_64-apple-darwin/bin/cargo rustc --release --frozen --manifest-path /Users/mcomella/dev/mozilla-central/netwerk/test/http3server/Cargo.toml --color=always --bin http3server --target=thumbv7neon-linux-androideabi --
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C toolkit/library/rust target

/Users/mcomella/.rustup/toolchains/stable-x86_64-apple-darwin/bin/cargo rustc --release --frozen --manifest-path /Users/mcomella/dev/mozilla-central/netwerk/test/http3server/Cargo.toml --color=always --bin http3server --target=thumbv7neon-linux-androideabi --
/Users/mcomella/.rustup/toolchains/stable-x86_64-apple-darwin/bin/cargo rustc --release --frozen --manifest-path /Users/mcomella/dev/mozilla-central/toolkit/library/rust/Cargo.toml --color=always --lib --target=thumbv7neon-linux-androideabi --features moz_memory gecko_profiler gecko_profiler_parse_elf new_xulstore webrtc glean_with_gecko glean_disable_upload --
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C toolkit/library/rust target

/Users/mcomella/.rustup/toolchains/stable-x86_64-apple-darwin/bin/cargo rustc --release --frozen --manifest-path /Users/mcomella/dev/mozilla-central/netwerk/test/http3server/Cargo.toml --color=always --bin http3server --target=thumbv7neon-linux-androideabi --
/Users/mcomella/.rustup/toolchains/stable-x86_64-apple-darwin/bin/cargo rustc --release --frozen --manifest-path /Users/mcomella/dev/mozilla-central/toolkit/library/rust/Cargo.toml --color=always --lib --target=thumbv7neon-linux-androideabi --features moz_memory gecko_profiler gecko_profiler_parse_elf new_xulstore webrtc glean_with_gecko glean_disable_upload --
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C toolkit/library/rust target

/Users/mcomella/.rustup/toolchains/stable-x86_64-apple-darwin/bin/cargo rustc --release --frozen --manifest-path /Users/mcomella/dev/mozilla-central/security/manager/ssl/ipcclientcerts/Cargo.toml --color=always --lib --target=thumbv7neon-linux-androideabi --

To my untrained eye, it looks like we're compiling multiple rust modules in parallel. In that case, it's possible the package lock warning is repeated because each time a new module starts up, it needs to block briefly on the previous one (i.e. the current implementation is efficient) rather than my concern that we're blocking on either 1) a single process locking the file for a long time or 2) a livelock-like situation across multiple processes (e.g. lock-unlock-lock-unlock style race condition).

Comment 3

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

fwiw, maybe I should have done a case-insensitive grep: I noticed Cargo.toml wouldn't have been caught here.

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

That could be the case, and that's the most ideal, optimistic theory :)
One other option is that the package cache file lock might include the PID of the process holding it: if you can query it before the lock is released, you might be able to confirm the root cause.

I think the cache file lock is at ~/.cargo/.package-cache, but I could be mistaken.

Comment 5

[Mike Hommey [:glandium]]

It's not actually blocked. Yes, we run multiple cargos, so all but one of them is blocked. The one that is not blocked does do work (and yes, it sucks that it takes so long, but that's how it is... it's checking there's no change in the ton of rust code we have (which includes checking sha256s for vendored code))

Eventually we'll only invoke cargo once, which will help a little, but overall, this is working as it's supposed to.