1461553 - `mach vendor rust` on clean tree generates changes on Mac

Status: RESOLVED INVALID

(Firefox Build System :: General, defect)

Description

[Myk Melez [:myk] [@mykmelez]]

Attachment: mach vendor rust results on Mac

`mach vendor rust` on a clean tree (tip of mozilla-central, fresh clone using git-cinnabar) with cargo 0.26.0 (41480f5cc 2018-02-26) generates these changes on my Mac.  It doesn't generate them on Linux with the same cargo version, however.

It looks like cargo is generating Cargo.lock files, which then causes cargo-vendor to update the .cargo-checksum.json files.  I suspect this is an upstream bug in cargo, but I'm filing it here for now.

Comment 1

[Kim Moir [:kmoir] ET]

The attached patch isn't a patch but a diff of the output so reassigning to default owner.

Comment 2

[Myk Melez [:myk] [@mykmelez]]

This turned out to be an issue with my Mac's cargo registry cache (~/.cargo/registry), which included Cargo.lock files in some crates that don't have them in the newer registry cache on my Linux box.

I'm unsure how they got there, as I haven't found a rust-lang/cargo issue or commit that seems related (but perhaps Alex Crichton knows, so cc:ing him).  In any case, invalidating the cache and reloading crates on the Mac resolves the problem.  So I'm resolving this bug as invalid.

Comment 3

[Mike Hommey [:glandium]]

FWIW, I had a similar issue with a Cargo.lock and a map file in the winapi crate which I got from doing unrelated rust things with RUSTFLAGS to set to pass -Wl,-Map,map to the linker.

Comment 4

[Alex Crichton [:acrichto]]

Oh dear this sounds bad for Cargo! I'm not aware of anything in Cargo that adds lock files by accident to the global registry, but it could be a bug in Cargo. This may also be a bug in other tools which are using Cargo's registry the wrong way, but it'd be hard to pinpoint them :(

Comment 5

[(not currently active) Ted Mielczarek]

FWIW, I dealt with a similar issue recently where I was getting weird results from `cargo vendor` and it turned out that I also had strange files in my ~/.cargo/registry. I removed the entire registry and let cargo repopulate it and things worked properly. It's entirely possible that I did something weird to cause this, like running custom builds of cargo from git or doing weird fiddling with the registry. Our vendored crates contain hashes of their contents--is that calculated from the files in the registry, or does that come from crates.io? It would be nice if we had a way to detect this.

Comment 6

[Alex Crichton [:acrichto]]

Cargo currently (mistakenly) assumes that the registry is immutable once a crate is checked out on disk (aka unpacked from the tarball). The cargo-vendor tool links to Cargo as a library and uses the same cache, and then it calculates the checksums for each file as it reads it off the filesystem and places it into the vendor directory. In that sense cargo-vendor is inheriting Cargo's (broken) assumption that the files aren't changing after they're unpacked.

Agreed it'd be good to detect this! One change would be to have cargo-vendor work directly from the tarballs instead of the checked-out state, but such a change isn't entirely trivial to implement today. Otherwise Cargo could also do something like periodically execute a sanity check to make sure checked out crates look alright.

Comment 7

[Myk Melez [:myk] [@mykmelez]]

It happened to me again today, and I wondered if the Rust Enhanced Sublime Text package <https://github.com/rust-lang/rust-enhanced> could be involved, but I wasn't able to reproduce the problem after using that package to build changes to several Rust crates.

Comment 8

[Mike Hommey [:glandium]]

This happened to :arai today with a spurious third_party/rust/lalrpop/src/parser/lrgrammar.rs.

Alex, should cargo maybe stage things out of the registry before compiling them? Because it seems a bunch of crates end up creating files in the source directory...

Comment 9

[Alex Crichton [:acrichto]]

Perhaps! That's something we'd have to implement and evaluate as it runs the risk of slowing down builds. I'd personally prefer to dig in and discover what crates are modifying the registry. I suspect it's actually very few but perhaps heavily depended on ones that have consequences for the ecosystem.

Comment 10

[Myk Melez [:myk] [@mykmelez]]

As a next step, I filed Cargo issue https://github.com/rust-lang/cargo/issues/5956 on this.

Comment 11

[(not currently active) Ted Mielczarek]

That seems like it's going to take a lot of work to fix, so I filed a cargo-vendor issue with a proposal for working around this: https://github.com/alexcrichton/cargo-vendor/issues/130