Closed Bug 1365918 Opened 7 years ago Closed 6 years ago

Intermittent mozmake.EXE[5]: *** [force-cargo-library-build] Error 101 after "could not exec the linker `link.exe`"

Categories

(Taskcluster :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure)

Filed by: kgupta [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=100058362&repo=graphics

https://queue.taskcluster.net/v1/task/L1TfDzdZSHaRaHCXVzWl9g/runs/0/artifacts/public/logs/live_backing.log

Root cause appears to be this:
  error: could not exec the linker `link.exe`: The system cannot find the file specified. (os error 2)
which happens at some point during the cargo build

Not sure if this is infra or a race in the build
The full error output:

10:20:04     INFO -  error: could not exec the linker `link.exe`: The system cannot find the file specified. (os error 2)
10:20:04     INFO -    |
10:20:04     INFO -    = note: "link.exe" "/NOLOGO" "/NXCOMPAT" "/LIBPATH:Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "z:/task_1495101977/build/src/obj-firefox/toolkit/library\\release\\build\\rayon-core-318c6585264007d9\\build_script_build-318c6585264007d9.0.o" "/OUT:z:/task_1495101977/build/src/obj-firefox/toolkit/library\\release\\build\\rayon-core-318c6585264007d9\\build_script_build-318c6585264007d9.exe" "/OPT:REF,ICF" "/DEBUG" "/LIBPATH:z:/task_1495101977/build/src/obj-firefox/toolkit/library\\release\\deps" "/LIBPATH:Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-0a78323911070f99.rlib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librand-c279a51d66700350.rlib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcollections-d7bf31a4ca1ea637.rlib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd_unicode-d367c3ba0db49600.rlib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-2d4bf02140c11dcb.rlib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-add7a84d7e82d084.rlib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liblibc-84688accbc86d6b7.rlib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-fe2e68b21f0bdd7a.rlib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc_system-7fc0381594c93f56.rlib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-ea9d77e7c23fe65c.rlib" "Z:\\task_1495101977\\build\\src\\rustc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-91b619d34dd1f5aa.rlib" "advapi32.lib" "ws2_32.lib" "userenv.lib" "shell32.lib" "msvcrt.lib"
10:20:04     INFO -  note: the msvc targets depend on the msvc linker but `link.exe` was not found
10:20:04     INFO -  note: please ensure that VS 2013 or VS 2015 was installed with the Visual C++ option
10:20:04     INFO -  error: aborting due to previous error

This appears to be happening trying to link the Rayon build script. The cargo invocation looks like:
10:20:02     INFO -  env   CARGO_TARGET_DIR=z:/task_1495101977/build/src/obj-firefox/toolkit/library RUSTC=z:/task_1495101977/build/src/rustc/bin/rustc.exe MOZ_DIST=z:/task_1495101977/build/src/obj-firefox/dist LIBCLANG_PATH= CLANG_PATH= PKG_CONFIG_ALLOW_CROSS=1 RUST_BACKTRACE=1 MOZ_TOPOBJDIR=z:/task_1495101977/build/src/obj-firefox  z:/task_1495101977/build/src/rustc/bin/cargo.exe build  --release --frozen --manifest-path z:/task_1495101977/build/src/toolkit/library/gtest/rust/Cargo.toml --verbose --lib --target=x86_64-pc-windows-msvc --features "quantum_render"

Up at the start of the build where it echoes options from the mozconfig I see:
10:16:20     INFO -      export PATH=z:\task_1495101977\build\src\vs2015u3\VC\bin\amd64;z:\task_1495101977\build\src\vs2015u3\VC\bin;z:\task_1495101977\build\src\vs2015u3\SDK\bin\x64;z:\task_1495101977\build\src\vs2015u3\VC\redist\x64\Microsoft.VC140.CRT;z:\task_1495101977\build\src\vs2015u3\SDK\Redist\ucrt\DLLs\x64;z:\task_1495101977\build\src\vs2015u3\DIA SDK\bin\amd64;z:\task_1495101977\build\src\clang\bin;z:\task_1495101977\build\src\vs2015u3\VC\bin\amd64;z:\task_1495101977\build\src\vs2015u3\VC\bin;z:\task_1495101977\build\src\vs2015u3\SDK\bin\x64;z:\task_1495101977\build\src\vs2015u3\VC\redist\x64\Microsoft.VC140.CRT;z:\task_1495101977\build\src\vs2015u3\SDK\Redist\ucrt\DLLs\x64;z:\task_1495101977\build\src\vs2015u3\DIA SDK\bin\amd64;c:\Program Files\Mercurial;c:\mozilla-build\7zip;c:\mozilla-build\info-zip;c:\mozilla-build\kdiff3;c:\mozilla-build\moztools-x64\bin;c:\mozilla-build\mozmake;C:\mozilla-build\msys\bin;C:\mozilla-build\msys\local\bin;c:\mozilla-build\nsis-3.01;c:\mozilla-build\nsis-3.0b3;c:\mozilla-build\nsis-2.46u;c:\mozilla-build\python;c:\mozilla-build\python\Scripts;c:\mozilla-build\upx391w;c:\mozilla-build\wget;c:\mozilla-build\yasm;c:\Windows\system32;c:\Windows;c:\Windows\System32\Wbem;c:\Windows\System32\WindowsPowerShell\v1.0\;c:\Program Files\Amazon\cfn-bootstrap\;c:\Program Files (x86)\GNU\GnuPG\pub;c:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;c:\mozilla-build\python\lib\site-packages\pywin32_system32;c:\mozilla-build\python\lib\site-packages\pywin32_system32;c:\mozilla-build\python\lib\site-packages\pywin32_system32;c:\mozilla-build\python\lib\site-packages\pywin32_system32
<...>
10:16:20     INFO -      export PATH=z:\task_1495101977\build\src\clang\bin;z:\task_1495101977\build\src\vs2015u3\VC\bin\amd64;z:\task_1495101977\build\src\vs2015u3\VC\bin;z:\task_1495101977\build\src\vs2015u3\SDK\bin\x64;z:\task_1495101977\build\src\vs2015u3\VC\redist\x64\Microsoft.VC140.CRT;z:\task_1495101977\build\src\vs2015u3\SDK\Redist\ucrt\DLLs\x64;z:\task_1495101977\build\src\vs2015u3\DIA SDK\bin\amd64;z:\task_1495101977\build\src\clang\bin;z:\task_1495101977\build\src\vs2015u3\VC\bin\amd64;z:\task_1495101977\build\src\vs2015u3\VC\bin;z:\task_1495101977\build\src\vs2015u3\SDK\bin\x64;z:\task_1495101977\build\src\vs2015u3\VC\redist\x64\Microsoft.VC140.CRT;z:\task_1495101977\build\src\vs2015u3\SDK\Redist\ucrt\DLLs\x64;z:\task_1495101977\build\src\vs2015u3\DIA SDK\bin\amd64;c:\Program Files\Mercurial;c:\mozilla-build\7zip;c:\mozilla-build\info-zip;c:\mozilla-build\kdiff3;c:\mozilla-build\moztools-x64\bin;c:\mozilla-build\mozmake;C:\mozilla-build\msys\bin;C:\mozilla-build\msys\local\bin;c:\mozilla-build\nsis-3.01;c:\mozilla-build\nsis-3.0b3;c:\mozilla-build\nsis-2.46u;c:\mozilla-build\python;c:\mozilla-build\python\Scripts;c:\mozilla-build\upx391w;c:\mozilla-build\wget;c:\mozilla-build\yasm;c:\Windows\system32;c:\Windows;c:\Windows\System32\Wbem;c:\Windows\System32\WindowsPowerShell\v1.0\;c:\Program Files\Amazon\cfn-bootstrap\;c:\Program Files (x86)\GNU\GnuPG\pub;c:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;c:\mozilla-build\python\lib\site-packages\pywin32_system32;c:\mozilla-build\python\lib\site-packages\pywin32_system32;c:\mozilla-build\python\lib\site-packages\pywin32_system32;c:\mozilla-build\python\lib\site-packages\pywin32_system32

So it definitely has MSVC in the PATH at some point here, although our build system seems to execute link.exe with its full path so it's hard to tell precisely.
We've had problems in the past where link.exe was in PATH but the dlls it needed weren't, although I forget if Windows will return "cannot find the file specified" in that situation or not (I think on Linux it's a runtime error?)
Component: Build Config → General
Depends on: 1364651
Product: Core → Taskcluster
Which version of the build tool do we have installed? Might it be the 2017 one? If yes, this could be related to https://github.com/rust-lang/rust/issues/38584.
(In reply to Henrik Skupin (:whimboo) from comment #5)
> Which version of the build tool do we have installed? Might it be the 2017
> one? If yes, this could be related to
> https://github.com/rust-lang/rust/issues/38584.

We don't have any build tools actually installed, we download VC2015 from tooltool:
https://dxr.mozilla.org/mozilla-central/rev/7b8937970f9ca85db88cb2496f2112175fd847c8/browser/config/tooltool-manifests/win32/releng.manifest#26
...oh wait, I lied, on buildbot builders we do have VC2013 installed on the system, but on taskcluster builders we don't have any MSVC toolchain installed systemwide.

Looking at OrangeFactor this only seems to happen on taskcluster builds, which sort of makes sense. If the failure is that rustc isn't finding the link.exe from our tooltool package, then on buildbot it's probably finding the one from VC2013. On taskcluster it won't find anything and it'll fail.
Interestingly, I found at least one of these that was misstarred:
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=100577813&lineNumber=19997

has in its log:
23:16:11     INFO -  error: couldn't create a temp dir: The system cannot find the path specified. (os error 3)
23:16:11     INFO -  error: Could not compile `gkrust`.
23:16:11     INFO -  Caused by:
23:16:11     INFO -    process didn't exit successfully: `z:/task_1495234214/build/src/rustc/bin/rustc.exe --crate-name gkrust Z:\task_1495234214\build\src\toolkit\library\rust\lib.rs --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C lto -C debuginfo=2 --cfg feature="quantum_render" --cfg feature="gkrust-shared" -C metadata=f6896f3421875ea1 -C extra-filename=-f6896f3421875ea1 --out-dir z:/task_1495234214/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps --target i686-pc-windows-msvc -L dependency=z:/task_1495234214/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps -L dependency=z:/task_1495234214/build/src/obj-firefox/toolkit/library\release\deps --extern gkrust_shared=z:/task_1495234214/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libgkrust_shared-2584a39f0e8359bd.rlib` (exit code: 101)
I've been assuming that this was not a matter of building on a worker which did not and never did have a linker, but instead suddenly removing link.exe out from under the build, thus the dependency on bug 1364651, which I'm calling the main bug for a whole huge raft of symptoms of suddenly deleting the toolchain and various directories including tmp out from under an in-progress build.
Oh, OK, that makes a lot more sense and would explain the symptoms here! If something deleted the toolchain directory after configure ran but before rustc tried to link anything then it makes sense that it would fail like this. As I said in comment 7, this particular failure mode would not show up on buildbot builds, since they do have a system VC2013 install. I think the "cl.exe went missing" failure would look the same on buildbot or taskcluster, since configure saves the full path to cl.exe.
No instances since bug 1364651 was fixed.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → DUPLICATE
Rob, last time this was "fixed" by bug 1364651 and a "here be dragons" comment.  Any idea what's going on this time?
Status: RESOLVED → REOPENED
Flags: needinfo?(rthijssen)
Resolution: DUPLICATE → ---
i can't see that the issue in bug 1428126 is the same as the one in bug 1364651.

in 1364651 we established that the mach call had been changed to use the python wrapper rather than bash. the log from the failure in 1428126 shows that bash was used so while the symptoms might be similar, i don't think it's the same issue.

the error in 1428126 suggests that some file referenced in the link.exe call (or link.exe itself) is missing or that environment configuration that would point to link.exe is missing. link.exe is part of the visual studio tooltool artifact. another thing that's new here is all the rust references. that kinda points at build environment config. specifically environment variables with "LIB" in their name (would be my guess).

14:17:42     INFO - error: could not exec the linker `link.exe`: The system cannot find the file specified. (os error 2)
14:17:42     INFO -   |
14:17:42     INFO -   = note: "link.exe" "/NOLOGO" "/NXCOMPAT" "/LARGEADDRESSAWARE" "/SAFESEH" "/LIBPATH:Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib" "z:/build/build/src/obj-firefox/toolkit/library\\release\\build\\bindgen-e7d239fab7499eb9\\build_script_build-e7d239fab7499eb9.build_script_build0.rcgu.o" "/OUT:z:/build/build/src/obj-firefox/toolkit/library\\release\\build\\bindgen-e7d239fab7499eb9\\build_script_build-e7d239fab7499eb9.exe" "z:/build/build/src/obj-firefox/toolkit/library\\release\\build\\bindgen-e7d239fab7499eb9\\build_script_build-e7d239fab7499eb9.crate.allocator.rcgu.o" "/OPT:REF,ICF" "/DEBUG" "/NATVIS:Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\etc\\libcore.natvis" "/LIBPATH:z:/build/build/src/obj-firefox/toolkit/library\\release\\deps" "/LIBPATH:Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib" "Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib\\libstd-c7c9df4d0b6e0a70.rlib" "Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib\\liballoc_system-a3c972fc99d2ce78.rlib" "Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib\\libunwind-c73684c6db884e03.rlib" "Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib\\libpanic_unwind-65ea901abd146f7d.rlib" "Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib\\liblibc-2f9b6e16a5e4764d.rlib" "Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib\\liballoc-2c52a979c242522c.rlib" "Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib\\libstd_unicode-46379ace75c02c60.rlib" "Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib\\libcore-673999a69af00166.rlib" "Z:\\task_1515071348\\build\\src\\rustc\\lib\\rustlib\\i686-pc-windows-msvc\\lib\\libcompiler_builtins-d5d3c1d182c873cf.rlib" "advapi32.lib" "ws2_32.lib" "userenv.lib" "shell32.lib" "msvcrt.lib"
14:17:42     INFO - note: the msvc targets depend on the msvc linker but `link.exe` was not found
14:17:42     INFO - note: please ensure that VS 2013 or VS 2015 was installed with the Visual C++ option
Flags: needinfo?(rthijssen)
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
Status: REOPENED → RESOLVED
Closed: 7 years ago6 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.