1323901 - Rust detection on Windows is broken with latest rustup.rs

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Xidorn Quan [:xidorn] UTC+10]

mach fails to configure with my local m-c (checked out one or two days ago) with the latest version of rustup (which I just update this morning).

THe issue is that configure located "cargo.EXE" and "rustc.EXE". Those files exist on Windows because the filesystem on Windows is case-insensitive. However, rustup's proxy cannot handle that correctly (probably because it only recognize the lowercased ".exe" somehow), so things are broken.

I think we probably need to fix this on both rustup and m-c, in case rustup doesn't get a quick enough update.

A temporary workaround would be lowercasing everything in the PATHEXT environment variable.

Comment 1

[Xidorn Quan [:xidorn] UTC+10]

Fix to rustup: https://github.com/rust-lang-nursery/rustup.rs/pull/861

Comment 2

[(not currently active) Ted Mielczarek]

Oh, this is the bug that RyanVM was hitting yesterday.

Comment 3

[:Gijs (he/him)]

Is there anything we can do about this on the m-c side? If so, can we get that into the tree so people's builds don't break en masse on affected versions of win10 or whatever it is that's causing this?

Comment 4

[:Gijs (he/him)]

strawman proposal: make us use "rustc.exe" and "cargo.exe" on Windows in whatever the build vars are for those tools, just like we already harcode "hg.exe" in some places.

Comment 5

[Marco Bonardo [:mak]]

Just for others coming here, I added this to my .profile, based on comment 0:
export PATHEXT=$( echo "$PATHEXT" | tr -s  '[:upper:]'  '[:lower:]' )

Comment 6

[Xidorn Quan [:xidorn] UTC+10]

Attachment: Bug 1323901 - Hardcode rustc.exe and cargo.exe for detection on Windows.

Review commit: https://reviewboard.mozilla.org/r/100462/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/100462/

Comment 7

[Mike Hommey [:glandium]]

Comment on attachment 8821086 [details]
Bug 1323901 - Hardcode rustc.exe and cargo.exe for detection on Windows.

https://reviewboard.mozilla.org/r/100462/#review101134

Let's kill 2 birds with one stone instead. (Patch incoming)

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1323901 - Force `which` to use lowercase extensions on Windows.

Review commit: https://reviewboard.mozilla.org/r/100616/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/100616/

Comment 9

[Mike Hommey [:glandium]]

Comment on attachment 8821316 [details]
Bug 1323901 - Force `which` to use lowercase extensions on Windows.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/100616/diff/1-2/

Comment 10

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8821316 [details]
Bug 1323901 - Force `which` to use lowercase extensions on Windows.

https://reviewboard.mozilla.org/r/100616/#review101404

::: build/moz.configure/util.configure:157
(Diff revision 2)
> +        for ext in exts:
> +            if ext == ".exe":
> +                break
> +        else:
> +            exts = ['.com', '.exe', '.bat']

This might be clearer as `if '.exe' not in exts:` etc.

Comment 11

[Jorg K (CEST = GMT+2)]

(In reply to Marco Bonardo [::mak] from comment #5)
> Just for others coming here, I added this to my .profile, based on comment 0:
> export PATHEXT=$( echo "$PATHEXT" | tr -s  '[:upper:]'  '[:lower:]' )
Thanks Marco, that got me going.

My rust experience wasn't great so far. I've installed it from the stand-alone installer, which suggested C:\Users\jorgk\.cargo\bin as install location. C:\Users\jorgk\.cargo\bin; got added to the Windows environment variables.

However, the build still didn't work. So ran |mach bootstrap|. The result was that c:/mozilla-source/.cargo/bin got added to my bash path (echo $PATH). That obviously didn't work, since that's not were it was installed. Since I couldn't work out where that was done, I moved the .cargo/bin to where it was expected. And then I ran into this bug here.

So can someone tell me where the rust installation path is set to c:/mozilla-source/.cargo/bin. (**)

Also, what is C:\Users\jorgk\.rustup\toolchains\stable-x86_64-pc-windows-msvc\bin\rustc.exe ?

Oh, just for added fun: C:\Users\jorgk\.cargo\bin\ reappeared during the build despite being moved previously.

Given that C:\Users\jorgk\.cargo\bin\ reappeared, I should be able to remove the copy in c:/mozilla-source/.cargo/bin, but then I need to know where c:/mozilla-source/.cargo/bin is set up as path, see (**) above.

Comment 12

[Jorg K (CEST = GMT+2)]

Sorry about the off-topic question ;-)

(In reply to Jorg K (GMT+1) from comment #11)
> So can someone tell me where the rust installation path is set to
> c:/mozilla-source/.cargo/bin. (**)
Found it: C:\mozilla-build\msys\etc\profile.d\profile-rustup.sh

#!/bash/sh
if test -n "$MOZILLABUILD"; then
    WIN_HOME=$(cd "$HOME" && pwd)
    PATH="$WIN_HOME/.cargo/bin:$PATH"
    export PATH
fi
And since I have HOME=C:\mozilla-source for my Mozilla development, I get c:/mozilla-source/.cargo/bin.

Comment 13

[Mike Hommey [:glandium]]

Comment on attachment 8821316 [details]
Bug 1323901 - Force `which` to use lowercase extensions on Windows.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/100616/diff/2-3/

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/21e27f6e2770
Force `which` to use lowercase extensions on Windows. r=cmanchester+432261

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/536b05a23fa0
Bustage fix from glandium

Comment 16

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/21e27f6e2770
https://hg.mozilla.org/mozilla-central/rev/536b05a23fa0

Comment 17

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Hm, so I'm building with 232424f9511d and I'm still getting the cargo.EXE issue.  I don't have an updated rustup proxy -- do I need to?

https://hg.mozilla.org/mozilla-central/file/tip/python/mozbuild/mozbuild/vendor_rust.py#l27 is calling 'which.which' directly, which returns "cargo.EXE".  Note that I did not have cargo-vendor installed, so it tried to build it first (and failed since I didn't have cmake in my path, but that's a different thing).

Comment 18

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

I was running from an unbuilt tree, so I didn't have substs['CARGO'].  However, can we somehow capture stdout/stderr from these commands and print them? All I get now is:

 0:11.39 c:\Users\vladimir\.cargo\bin\cargo.exe update --manifest-path c:/proj/moz/central/toolkit/library/gtest/rust/Cargo.toml -p gkrust-gtest
Error running mach:

    ['vendor', 'rust', '--ignore-modified']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 101: [u'c:\\Users\\vladimir\\.cargo\\bin\\cargo.exe', u'update', u'--manifest-path', u'c:/proj/moz/central/toolkit/library/gtest/rust/Cargo.toml', u'-p', u'gkrust-gtest']


When what I really needed to see was:

$ cargo update --manifest-path c:/proj/moz/central/toolkit/library/gtest/rust/Cargo.toml -p gkrust-gtest
    Updating registry `https://github.com/rust-lang/crates.io-index`
error: failed to select a version for `dwrote` (required by `webrender`):
all possible versions conflict with previously selected versions of `dwrote`
  version 0.1.3 in use by dwrote v0.1.3
  possible versions to select: 0.1.4

Comment 19

[(not currently active) Ted Mielczarek]

File a new bug on that? Seems reasonable, I think I just didn't want `mach vendor rust` to produce a ton of output in the normal case.