Closed Bug 1461553 Opened 2 years ago Closed 2 years ago

`mach vendor rust` on clean tree generates changes on Mac

Categories

(Firefox Build System :: General, defect)

3 Branch
defect
Not set

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: myk, Unassigned)

Details

(Keywords: in-triage)

Attachments

(1 file)

`mach vendor rust` on a clean tree (tip of mozilla-central, fresh clone using git-cinnabar) with cargo 0.26.0 (41480f5cc 2018-02-26) generates these changes on my Mac.  It doesn't generate them on Linux with the same cargo version, however.

It looks like cargo is generating Cargo.lock files, which then causes cargo-vendor to update the .cargo-checksum.json files.  I suspect this is an upstream bug in cargo, but I'm filing it here for now.
Assignee: nobody → myk
Keywords: in-triage
The attached patch isn't a patch but a diff of the output so reassigning to default owner.
Assignee: myk → nobody
This turned out to be an issue with my Mac's cargo registry cache (~/.cargo/registry), which included Cargo.lock files in some crates that don't have them in the newer registry cache on my Linux box.

I'm unsure how they got there, as I haven't found a rust-lang/cargo issue or commit that seems related (but perhaps Alex Crichton knows, so cc:ing him).  In any case, invalidating the cache and reloading crates on the Mac resolves the problem.  So I'm resolving this bug as invalid.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INVALID
FWIW, I had a similar issue with a Cargo.lock and a map file in the winapi crate which I got from doing unrelated rust things with RUSTFLAGS to set to pass -Wl,-Map,map to the linker.
Oh dear this sounds bad for Cargo! I'm not aware of anything in Cargo that adds lock files by accident to the global registry, but it could be a bug in Cargo. This may also be a bug in other tools which are using Cargo's registry the wrong way, but it'd be hard to pinpoint them :(
FWIW, I dealt with a similar issue recently where I was getting weird results from `cargo vendor` and it turned out that I also had strange files in my ~/.cargo/registry. I removed the entire registry and let cargo repopulate it and things worked properly. It's entirely possible that I did something weird to cause this, like running custom builds of cargo from git or doing weird fiddling with the registry. Our vendored crates contain hashes of their contents--is that calculated from the files in the registry, or does that come from crates.io? It would be nice if we had a way to detect this.
Cargo currently (mistakenly) assumes that the registry is immutable once a crate is checked out on disk (aka unpacked from the tarball). The cargo-vendor tool links to Cargo as a library and uses the same cache, and then it calculates the checksums for each file as it reads it off the filesystem and places it into the vendor directory. In that sense cargo-vendor is inheriting Cargo's (broken) assumption that the files aren't changing after they're unpacked.

Agreed it'd be good to detect this! One change would be to have cargo-vendor work directly from the tarballs instead of the checked-out state, but such a change isn't entirely trivial to implement today. Otherwise Cargo could also do something like periodically execute a sanity check to make sure checked out crates look alright.
It happened to me again today, and I wondered if the Rust Enhanced Sublime Text package <https://github.com/rust-lang/rust-enhanced> could be involved, but I wasn't able to reproduce the problem after using that package to build changes to several Rust crates.
This happened to :arai today with a spurious third_party/rust/lalrpop/src/parser/lrgrammar.rs.

Alex, should cargo maybe stage things out of the registry before compiling them? Because it seems a bunch of crates end up creating files in the source directory...
Flags: needinfo?(acrichton)
Perhaps! That's something we'd have to implement and evaluate as it runs the risk of slowing down builds. I'd personally prefer to dig in and discover what crates are modifying the registry. I suspect it's actually very few but perhaps heavily depended on ones that have consequences for the ecosystem.
Flags: needinfo?(acrichton)
As a next step, I filed Cargo issue https://github.com/rust-lang/cargo/issues/5956 on this.
That seems like it's going to take a lot of work to fix, so I filed a cargo-vendor issue with a proposal for working around this: https://github.com/alexcrichton/cargo-vendor/issues/130
Keywords: in-triage
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.