1470384 - Intermittent mozmake.EXE[2]: *** [force-cargo-test-run] Error 101 after error: linker `link.exe` not found

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de

https://treeherder.mozilla.org/logviewer.html#?job_id=184124637&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/bg0Sc09EQwqHfs0d7ZUEow/runs/0/artifacts/public/logs/live_backing.log

11:26:28     INFO -    Compiling unicode-xid v0.1.0
11:26:28     INFO -      Running `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name unicode_xid 'third_party\rust\unicode-xid\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="default"' -C metadata=9ea2f1482d0fd3cb -C extra-filename=-9ea2f1482d0fd3cb --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --cap-lints allow`
11:26:28     INFO - error: linker `link.exe` not found
11:26:28     INFO -   |
11:26:28     INFO -   = note: The system cannot find the file specified. (os error 2)
11:26:28     INFO - note: the msvc targets depend on the msvc linker but `link.exe` was not found
11:26:28     INFO - note: please ensure that VS 2013 or VS 2015 was installed with the Visual C++ option
11:26:28     INFO - error: aborting due to previous error
11:26:28     INFO - error: Could not compile `rayon-core`.
11:26:28     INFO - Caused by:
11:26:28     INFO -   process didn't exit successfully: `z:/build/build/src/sccache2/sccache.exe z:/build/build/src/rustc/bin/rustc.exe --crate-name build_script_build third_party\rust\rayon-core\build.rs --crate-type bin --emit=dep-info,link -C opt-level=2 -C codegen-units=1 -C metadata=1578873bfa3ae287 -C extra-filename=-1578873bfa3ae287 --out-dir z:/build/build/src/obj-firefox/toolkit/library\release\build\rayon-core-1578873bfa3ae287 -L dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps --cap-lints allow` (exit code: 101)
11:26:28     INFO - warning: build failed, waiting for other jobs to finish...
11:26:29     INFO - ....error: linker `link.exe` not found
11:26:29     INFO -   |
11:26:29     INFO -   = note: The system cannot find the file specified. (os error 2)
11:26:29     INFO - note: the msvc targets depend on the msvc linker but `link.exe` was not found
11:26:29     INFO - note: please ensure that VS 2013 or VS 2015 was installed with the Visual C++ option
11:26:29     INFO - error: aborting due to previous error
11:26:29     INFO - error: Could not compile `serde`.
11:26:29     INFO - Caused by:
11:26:29     INFO -   process didn't exit successfully: `z:/build/build/src/sccache2/sccache.exe z:/build/build/src/rustc/bin/rustc.exe --crate-name build_script_build third_party\rust\serde\build.rs --crate-type bin --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg feature="default" --cfg feature="rc" --cfg feature="serde_derive" --cfg feature="std" -C metadata=2076f3a9f9db8f35 -C extra-filename=-2076f3a9f9db8f35 --out-dir z:/build/build/src/obj-firefox/toolkit/library\release\build\serde-2076f3a9f9db8f35 -L dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps --cap-lints allow` (exit code: 101)
11:26:29     INFO - warning: build failed, waiting for other jobs to finish...
11:26:31     INFO - ...........TEST-PASS | autobinscope.py | ../../dist/bin/xul.dll succeeded
11:26:31     INFO - z:/build/build/src/obj-firefox/_virtualenvs/init/Scripts/python.exe z:/build/build/src/build/win32/autobinscope.py ../../dist/bin/nss3.dll ../../dist/crashreporter-symbols/
11:26:31     INFO - ..TEST-PASS | autobinscope.py | ../../dist/bin/nss3.dll succeeded
11:26:31     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/build/win32'
11:26:32     INFO - ...........error: build failed
11:26:32     INFO - z:/build/build/src/config/rules.mk:974: recipe for target 'force-cargo-test-run' failed
11:26:32     INFO - mozmake.EXE[2]: *** [force-cargo-test-run] Error 101
11:26:32     INFO - mozmake.EXE[2]: Target 'check' not remade because of errors.
11:26:32     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/toolkit/library/rust'
11:26:32     INFO - z:/build/build/src/config/recurse.mk:101: recipe for target 'toolkit/library/rust/check' failed
11:26:32     INFO - mozmake.EXE[1]: *** [toolkit/library/rust/check] Error 2
11:26:35     INFO - ...............TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\dom/bindings/Errors.msg | ok
11:26:35     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/ctypes/ctypes.msg | ok
11:26:35     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/js.msg | ok
11:26:35     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/jsshell.msg | ok
11:26:35     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/xpconnect/src/jsshell.msg | ok
11:26:35     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/js/src'
11:26:35     INFO - .mozmake.EXE[2]: Entering directory 'z:/build/build/src/obj-firefox/js/src/build'
11:26:35     INFO - mozmake.EXE[2]: Nothing to be done for 'check'.
11:26:35     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/js/src/build'
11:26:42     INFO - ............................................................
11:26:42     INFO - ----------------------------------------------------------------------
11:26:42     INFO - Ran 120 tests in 17.409s
11:26:42     INFO - OK
11:26:42     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/ipc/ipdl/test/ipdl'
11:26:42     INFO - mozmake.EXE[1]: Target 'recurse_check' not remade because of errors.
11:26:42     INFO - mozmake.EXE[1]: Leaving directory 'z:/build/build/src/obj-firefox'
11:26:42     INFO - z:/build/build/src/config/recurse.mk:32: recipe for target 'check' failed
11:26:42     INFO - mozmake.EXE: *** [check] Error 2

Comment 1

[Intermittent Failures Robot]

4 failures in 521 pushes (0.008 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2
* mozilla-central: 2

Platform breakdown:
* windows2012-32: 3
* windows2012-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-06-18&endday=2018-06-24&tree=trunk

Comment 2

[Intermittent Failures Robot]

1 failures in 718 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* windows2012-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-06-25&endday=2018-07-01&tree=trunk

Comment 3

[Intermittent Failures Robot]

7 failures in 623 pushes (0.011 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 3
* mozilla-central: 2
* try: 1
* autoland: 1

Platform breakdown:
* windows2012-64: 6
* windows2012-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-07-09&endday=2018-07-15&tree=trunk

Comment 4

[Intermittent Failures Robot]

21 failures in 653 pushes (0.032 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 12
* mozilla-inbound: 5
* mozilla-central: 4

Platform breakdown:
* windows2012-64: 16
* windows2012-32: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-07-16&endday=2018-07-22&tree=trunk

Comment 6

[Intermittent Failures Robot]

33 failures in 760 pushes (0.043 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 17
* mozilla-inbound: 12
* mozilla-central: 4

Platform breakdown:
* windows2012-64: 30
* windows2012-32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-07-23&endday=2018-07-29&tree=trunk

Comment 7

[Intermittent Failures Robot]

31 failures in 2561 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 14
* autoland: 10
* mozilla-central: 7

Platform breakdown:
* windows2012-64: 20
* windows2012-32: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-08-05&endday=2018-08-11&tree=all

Comment 8

[Intermittent Failures Robot]

30 failures in 2579 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 9
* mozilla-inbound: 13
* mozilla-central: 8

Platform breakdown:
* windows2012-32: 11
* windows2012-64: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-08-06&endday=2018-08-12&tree=all

Comment 9

[Intermittent Failures Robot]

28 failures in 2672 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 6
* autoland: 8
* mozilla-inbound: 14

Platform breakdown:
* windows2012-64: 22
* windows2012-32: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-08-13&endday=2018-08-19&tree=all

Comment 10

[Dorel Luca [:dluca]]

This bug has failed 37 times in the last 7 days. All failures occur on windows 2012 platform on opt build type.

Log:
https://treeherder.mozilla.org/logviewer.html#?job_id=194973208&repo=mozilla-inbound&lineNumber=50573

........error: build failed
23:59:27     INFO - z:/build/build/src/config/rules.mk:1040: recipe for target 'force-cargo-test-run' failed
23:59:27     INFO - mozmake.EXE[2]: *** [force-cargo-test-run] Error 101
23:59:27     INFO - mozmake.EXE[2]: Target 'check' not remade because of errors.
23:59:27     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/toolkit/library/rust'
23:59:27     INFO - z:/build/build/src/config/recurse.mk:101: recipe for target 'toolkit/library/rust/check' failed
23:59:27     INFO - mozmake.EXE[1]: *** [toolkit/library/rust/check] Error 2
23:59:27     INFO - .TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\dom/bindings/Errors.msg | ok
23:59:27     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/ctypes/ctypes.msg | ok
23:59:27     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/js.msg | ok
23:59:27     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/jsshell.msg | ok
23:59:27     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/xpconnect/src/jsshell.msg | ok
23:59:27     INFO - .mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/js/src'
23:59:27     INFO - mozmake.EXE[2]: Entering directory 'z:/build/build/src/obj-firefox/js/src/build'
23:59:27     INFO - mozmake.EXE[2]: Nothing to be done for 'check'.
23:59:27     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/js/src/build'
23:59:38     INFO - ..........................................................................
23:59:38     INFO - ----------------------------------------------------------------------
23:59:38     INFO - Ran 120 tests in 20.392s
23:59:38     INFO - OK
23:59:38     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/ipc/ipdl/test/ipdl'
23:59:38     INFO - mozmake.EXE[1]: Target 'recurse_check' not remade because of errors.
23:59:38     INFO - mozmake.EXE[1]: Leaving directory 'z:/build/build/src/obj-firefox'
23:59:38     INFO - z:/build/build/src/config/recurse.mk:32: recipe for target 'check' failed
23:59:38     INFO - mozmake.EXE: *** [check] Error 2
23:59:38     INFO - 631 compiler warnings present.
23:59:38    ERROR - Return code: 2
23:59:38     INFO - TinderboxPrint: check<br/>45/0/0
23:59:38  WARNING - setting return code to 2
23:59:38    ERROR - 'mach build check' did not run successfully. Please check log for errors.
23:59:38     INFO - [mozharness: 2018-08-20 23:59:38.541000Z] Finished check-test step (success)
23:59:38     INFO - [mozharness: 2018-08-20 23:59:38.542000Z] Skipping valgrind-test step.
23:59:38     INFO - [mozharness: 2018-08-20 23:59:38.542000Z] Skipping multi-l10n step.
23:59:38     INFO - [mozharness: 2018-08-20 23:59:38.542000Z] Skipping package-source step.
23:59:38     INFO - [mozharness: 2018-08-20 23:59:38.542000Z] Skipping update step.
23:59:38     INFO - [mozharness: 2018-08-20 23:59:38.542000Z] Skipping ensure-upload-path step.
23:59:38     INFO - Running post-run listener: _summarize
23:59:38    ERROR - # TBPL FAILURE #
23:59:38     INFO - [mozharness: 2018-08-20 23:59:38.542000Z] FxDesktopBuild summary:
23:59:38    ERROR - # TBPL FAILURE #
23:59:38     INFO - Running post-run listener: copy_logs_to_upload_dir

froydnj: Can you please take a look at this bug?

Comment 11

[Nathan Froyd [:froydnj]]

I don't have time to dig into this; this is a peculiar error.  Why do we *intermittently* fail to find the linker?  Could this be some weird sccache thing, Ted?

Comment 12

[(not currently active) Ted Mielczarek]

Seems unlikely, since sccache won't cache compiles that are actually links, so it just punts and lets the compiler work. We've seen failures like this in the past, the only things I can guess are that maybe:
a) The environment got screwed up so PATH is not set properly and rustc can't locate link.exe
b) Something cleaned up the toolchain while we were still using it.

Comment 13

[(not currently active) Ted Mielczarek]

FWIW in Rust 1.28 rustc just uses the `cc` crate to locate link.exe:
https://github.com/rust-lang/rust/blob/d2048b6db375299b681d4f4728b8e7cad9f74d5f/src/librustc_codegen_llvm/back/link.rs#L57

Some other things I noted while looking into this a bit:
1) We're using lld-link in our Windows builds but rustc is still using link.exe. We could fix that by setting the linker path in .cargo/config or via an environment variable, and passing `-Z linker_flavor=lld-link`, I think.
2) There's an unstable rustc option to print link lines, maybe we should enable that to get more useful output in the logs? It's `-Z print_link_args`.

Comment 14

[Intermittent Failures Robot]

34 failures in 2568 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 4
* try: 1
* mozilla-inbound: 17
* autoland: 12

Platform breakdown:
* windows2012-64: 27
* windows2012-32: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-08-20&endday=2018-08-26&tree=all

Comment 15

[Intermittent Failures Robot]

21 failures in 537 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 2
* autoland: 11
* mozilla-inbound: 7
* try: 1

Platform breakdown:
* windows2012-64: 12
* windows2012-32: 8
* windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-08-29&endday=2018-08-29&tree=all

Comment 16

[Tiberius Oros[:tiberius_oros]]

Update:
There have been 52 failures in the last week.
All failures are on the opt build type.

Failures per platform:
- windows2012-64: 37
- windows2012-32: 14
- windows10-64: 1

Recent relevat log file: https://treeherder.mozilla.org/logviewer.html#?job_id=197203615&repo=mozilla-inbound&lineNumber=48070

Comment 17

[Intermittent Failures Robot]

50 failures in 2545 pushes (0.02 failures/push) were associated with this bug in the last 7 days.

This is the #30 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* try: 1
* mozilla-central: 5
* mozilla-inbound: 21
* autoland: 23

Platform breakdown:
* windows2012-32: 14
* windows2012-64: 35
* windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-08-27&endday=2018-09-02&tree=all

Comment 18

[Intermittent Failures Robot]

35 failures in 2667 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #45 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 8
* mozilla-inbound: 9
* autoland: 18

Platform breakdown:
* windows2012-32: 7
* windows2012-64: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-09-03&endday=2018-09-09&tree=all

Comment 19

[Tiberius Oros[:tiberius_oros]]

There have been 34 failures in the last week.
All of the failures occur on the opt build type.

Failures per platform:

- windows2012-64: 28
- windows2012-32: 6

Summary: Intermittent mozmake.EXE[2]: *** [force-cargo-test-run] Error 101 after error: linker `link.exe` not found
Recent relevant log file:
https://treeherder.mozilla.org/logviewer.html#?job_id=198505997&repo=autoland&lineNumber=47331

:ted, do you think you could have another look at this?

Comment 20

[Intermittent Failures Robot]

42 failures in 2684 pushes (0.016 failures/push) were associated with this bug in the last 7 days.

This is the #26 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 19
* mozilla-inbound: 20
* mozilla-central: 3

Platform breakdown:
* windows2012-32: 6
* windows2012-64: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-09-10&endday=2018-09-16&tree=all

Comment 21

[Andreea Pavel [:apavel]]

There are 44 total failures in the last 7 days, all on  windows2012-64 and  windows2012-32.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=199954395&repo=mozilla-central&lineNumber=46999

15:28:22     INFO -    ::: third_party\rust\simd\src\v256.rs:163:1
15:28:22     INFO -     |
15:28:22     INFO - 163 | / bool_impls! {
15:28:22     INFO - 164 | |     bool64ix4: bool64i, i64x4, i64, 4, bool64ix4_all, bool64ix4_any, x0, x1 | x2, x3
15:28:22     INFO - 165 | |         [/// Convert `self` to a boolean vector for interacting with floating point vectors.
15:28:22     INFO - 166 | |          to_f -> bool64fx4];
15:28:22     INFO - ...   |
15:28:22     INFO - 185 | |         x16, x17, x18, x19, x20, x21, x22, x23, x24, x25, x26, x27, x28, x29, x30, x31 [];
15:28:22     INFO - 186 | | }
15:28:22     INFO -     | |_- in this macro invocation
15:28:22     INFO -    Compiling proc-macro2 v0.3.5
15:28:22     INFO -      Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name proc_macro2 'third_party\rust\proc-macro2-0.3.5\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=0b94b1f59eac2a8a -C extra-filename=-0b94b1f59eac2a8a --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'unicode_xid=z:/build/build/src/obj-firefox/toolkit/library\release\deps\libunicode_xid-5bd8ab608523f65a.rlib' --cap-lints warn`
15:28:22     INFO -    Compiling proc-macro2 v0.4.9
15:28:22     INFO -      Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name proc_macro2 'third_party\rust\proc-macro2\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=3c754d43858a505a -C extra-filename=-3c754d43858a505a --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'unicode_xid=z:/build/build/src/obj-firefox/toolkit/library\release\deps\libunicode_xid-5bd8ab608523f65a.rlib' --cap-lints warn`
15:28:22     INFO - warning: type annotations needed
15:28:22     INFO -    --> third_party\rust\arrayvec\src\lib.rs:312:32
15:28:22     INFO -     |
15:28:22     INFO - 312 |                 ptr::copy(p, p.offset(1), len - index);
15:28:22     INFO -     |                                ^^^^^^
15:28:22     INFO -     |
15:28:22     INFO -     = note: #[warn(tyvar_behind_raw_pointer)] on by default
15:28:22     INFO -     = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in the 2018 edition!
15:28:22     INFO -     = note: for more information, see issue #46906 <https://github.com/rust-lang/rust/issues/46906>
15:28:22     INFO -    Compiling dtoa-short v0.3.1
15:28:22     INFO -      Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name dtoa_short 'third_party\rust\dtoa-short\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 -C metadata=1678ea006aaf3eee -C extra-filename=-1678ea006aaf3eee --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'dtoa=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps\libdtoa-4012af297245995f.rlib' --cap-lints warn -C opt-level=2 -C debuginfo=2`
15:28:22     INFO -    Compiling num_cpus v1.7.0
15:28:22     INFO -      Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name num_cpus 'third_party\rust\num_cpus\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 -C metadata=0b08bb3d6400f966 -C extra-filename=-0b08bb3d6400f966 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'libc=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps\liblibc-9110c9de12692d34.rlib' --cap-lints warn -C opt-level=2 -C debuginfo=2`
15:28:22     INFO -    Compiling hashglobe v0.1.0 (file:///Z:/build/build/src/servo/components/hashglobe)
15:28:22     INFO -      Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name hashglobe 'servo\components\hashglobe\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 -C metadata=bd5532f0d5486ad6 -C extra-filename=-bd5532f0d5486ad6 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'libc=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps\liblibc-9110c9de12692d34.rlib' -C opt-level=2 -C debuginfo=2`
15:28:23     INFO - .   Compiling phf_shared v0.7.21
15:28:23     INFO -      Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name phf_shared 'third_party\rust\phf_shared\src/lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 -C metadata=0d11ec122cbf612f -C extra-filename=-0d11ec122cbf612f --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'siphasher=z:/build/build/src/obj-firefox/toolkit/library\release\deps\libsiphasher-2f1deef41a0ba09e.rlib' --cap-lints warn`
15:28:23     INFO -    Compiling winapi v0.3.4
15:28:23     INFO -      Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name winapi 'third_party\rust\winapi\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="consoleapi"' --cfg 'feature="errhandlingapi"' --cfg 'feature="fileapi"' --cfg 'feature="handleapi"' --cfg 'feature="libloaderapi"' --cfg 'feature="minwinbase"' --cfg 'feature="minwindef"' --cfg 'feature="ntdef"' --cfg 'feature="ntsecapi"' --cfg 'feature="ntstatus"' --cfg 'feature="processenv"' --cfg 'feature="profileapi"' --cfg 'feature="std"' --cfg 'feature="sysinfoapi"' --cfg 'feature="timezoneapi"' --cfg 'feature="winbase"' --cfg 'feature="wincon"' --cfg 'feature="winerror"' --cfg 'feature="winnt"' -C metadata=f397112c04927b88 -C extra-filename=-f397112c04927b88 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --cap-lints warn --cfg 'feature="basetsd"' --cfg 'feature="vadefs"' --cfg 'feature="winreg"' --cfg 'feature="subauth"' --cfg 'feature="sspi"' --cfg 'feature="excpt"' --cfg 'feature="ktmtypes"' --cfg 'feature="vcruntime"' --cfg 'feature="windef"' --cfg 'feature="cfg"' --cfg 'feature="guiddef"' --cfg 'feature="cfgmgr32"' --cfg 'feature="processthreadsapi"' --cfg 'feature="wincred"' --cfg 'feature="lsalookup"' --cfg 'feature="wingdi"' -l dylib=credui -l dylib=opengl32 -l dylib=gdi32 -l dylib=msimg32 -l dylib=secur32 -l dylib=setupapi -l dylib=kernel32 -l dylib=user32 -l dylib=winspool -l dylib=advapi32`
15:28:23     INFO - .     Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name winapi 'third_party\rust\winapi\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="consoleapi"' --cfg 'feature="errhandlingapi"' --cfg 'feature="fileapi"' --cfg 'feature="handleapi"' --cfg 'feature="libloaderapi"' --cfg 'feature="minwinbase"' --cfg 'feature="minwindef"' --cfg 'feature="ntdef"' --cfg 'feature="ntsecapi"' --cfg 'feature="ntstatus"' --cfg 'feature="processenv"' --cfg 'feature="profileapi"' --cfg 'feature="std"' --cfg 'feature="sysinfoapi"' --cfg 'feature="timezoneapi"' --cfg 'feature="winbase"' --cfg 'feature="wincon"' --cfg 'feature="winerror"' --cfg 'feature="winnt"' -C metadata=33bc19386d8c375f -C extra-filename=-33bc19386d8c375f --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --cap-lints warn -C opt-level=2 -C debuginfo=2 --cfg 'feature="cfg"' --cfg 'feature="sspi"' --cfg 'feature="basetsd"' --cfg 'feature="vcruntime"' --cfg 'feature="windef"' --cfg 'feature="vadefs"' --cfg 'feature="winreg"' --cfg 'feature="processthreadsapi"' --cfg 'feature="ktmtypes"' --cfg 'feature="guiddef"' --cfg 'feature="wingdi"' --cfg 'feature="lsalookup"' --cfg 'feature="excpt"' --cfg 'feature="subauth"' --cfg 'feature="wincred"' --cfg 'feature="cfgmgr32"' -l dylib=winspool -l dylib=kernel32 -l dylib=gdi32 -l dylib=user32 -l dylib=msimg32 -l dylib=secur32 -l dylib=opengl32 -l dylib=setupapi -l dylib=credui -l dylib=advapi32`
15:28:23     INFO -        Fresh aho-corasick v0.6.8
15:28:23     INFO -        Fresh nom v3.2.1
15:28:23     INFO -        Fresh thread_local v0.3.5
15:28:23     INFO -    Compiling humantime v1.1.1
15:28:23     INFO -      Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name phf_shared 'third_party\rust\phf_shared\src/lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 -C metadata=573ae359e1e9ea83 -C extra-filename=-573ae359e1e9ea83 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'siphasher=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps\libsiphasher-6b0165bf4753af50.rlib' --cap-lints warn -C opt-level=2 -C debuginfo=2`
15:28:23     INFO -      Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name humantime 'third_party\rust\humantime\src/lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 -C metadata=8e09e2f92dffc27c -C extra-filename=-8e09e2f92dffc27c --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'quick_error=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps\libquick_error-658b92a78952d663.rlib' --cap-lints warn -C opt-level=2 -C debuginfo=2`
15:28:23     INFO - error: linker `link.exe` not found
15:28:23     INFO -   |
15:28:23     INFO -   = note: The system cannot find the file specified. (os error 2)
15:28:23     INFO - note: the msvc targets depend on the msvc linker but `link.exe` was not found
15:28:23     INFO - note: please ensure that VS 2013, VS 2015 or VS 2017 was installed with the Visual C++ option
15:28:23     INFO - error: aborting due to previous error
15:28:23     INFO - .   Compiling itertools v0.7.6
15:28:23     INFO - error: Could not compile `servo_arc`.
15:28:23     INFO - Caused by:
15:28:23     INFO -   process didn't exit successfully: `z:/build/build/src/rustc/bin/rustc.exe --crate-name servo_arc 'servo\components\servo_arc\lib.rs' --emit=dep-info,link -C opt-level=3 --test -C metadata=b1aa4c68b95822d9 -C extra-filename=-b1aa4c68b95822d9 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'nodrop=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps\libnodrop-9515c80b4938dd02.rlib' --extern 'stable_deref_trait=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps\libstable_deref_trait-09c2ad651e74bab0.rlib' -C opt-level=2 -C debuginfo=2` (exit code: 1)
15:28:23     INFO - warning: build failed, waiting for other jobs to finish...
15:28:23     INFO -      Running `z:/build/build/src/rustc/bin/rustc.exe --crate-name itertools 'third_party\rust\itertools\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=4adfe4e3d570777f -C extra-filename=-4adfe4e3d570777f --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'either=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps\libeither-15cb6a5034c72d30.rlib' --cap-lints warn -C opt-level=2 -C debuginfo=2`
15:28:23     INFO - TEST-PASS | autobinscope.py | ../../dist/bin/xul.dll succeeded
15:28:23     INFO - z:/build/build/src/obj-firefox/_virtualenvs/init/Scripts/python.exe z:/build/build/src/build/win32/autobinscope.py ../../dist/bin/nss3.dll ../../dist/crashreporter-symbols/
15:28:23     INFO - .warning: this feature has been stable since 1.27.0. Attribute no longer needed
15:28:23     INFO -  --> third_party\rust\simd\src\lib.rs:4:12
15:28:23     INFO -   |
15:28:23     INFO - 4 | #![feature(cfg_target_feature, repr_simd, platform_intrinsics, const_fn)]
15:28:23     INFO -   |            ^^^^^^^^^^^^^^^^^^
15:28:23     INFO -   |
15:28:23     INFO -   = note: #[warn(stable_features)] on by default
15:28:24     INFO - .TEST-PASS | autobinscope.py | ../../dist/bin/nss3.dll succeeded
15:28:24     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/build/win32'
15:28:28     INFO - .............................error: build failed
15:28:28     INFO - z:/build/build/src/config/rules.mk:1064: recipe for target 'force-cargo-test-run' failed
15:28:28     INFO - mozmake.EXE[2]: *** [force-cargo-test-run] Error 101
15:28:28     INFO - mozmake.EXE[2]: Target 'check' not remade because of errors.
15:28:28     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/toolkit/library/rust'
15:28:28     INFO - z:/build/build/src/config/recurse.mk:101: recipe for target 'toolkit/library/rust/check' failed
15:28:28     INFO - mozmake.EXE[1]: *** [toolkit/library/rust/check] Error 2
15:28:30     INFO - ...........TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\dom/bindings/Errors.msg | ok
15:28:30     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/ctypes/ctypes.msg | ok
15:28:30     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/js.msg | ok
15:28:30     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/jsshell.msg | ok
15:28:30     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/xpconnect/src/jsshell.msg | ok
15:28:30     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/js/src'
15:28:30     INFO - mozmake.EXE[2]: Entering directory 'z:/build/build/src/obj-firefox/js/src/build'
15:28:30     INFO - mozmake.EXE[2]: Nothing to be done for 'check'.
15:28:30     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/js/src/build'
15:28:39     INFO - ...................................................................
15:28:39     INFO - ----------------------------------------------------------------------
15:28:39     INFO - Ran 120 tests in 19.585s
15:28:39     INFO - OK
15:28:39     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/ipc/ipdl/test/ipdl'
15:28:39     INFO - mozmake.EXE[1]: Target 'recurse_check' not remade because of errors.
15:28:39     INFO - mozmake.EXE[1]: Leaving directory 'z:/build/build/src/obj-firefox'
15:28:39     INFO - z:/build/build/src/config/recurse.mk:32: recipe for target 'check' failed
15:28:39     INFO - mozmake.EXE: *** [check] Error 2
15:28:39     INFO - 435 compiler warnings present.
15:28:39    ERROR - Return code: 2
15:28:39     INFO - TinderboxPrint: check<br/>45/0/0
15:28:39  WARNING - setting return code to 2
15:28:39    ERROR - 'mach build check' did not run successfully. Please check log for errors.
15:28:39     INFO - [mozharness: 2018-09-18 15:28:39.518000Z] Finished check-test step (success)
15:28:39     INFO - [mozharness: 2018-09-18 15:28:39.519000Z] Skipping valgrind-test step.
15:28:39     INFO - [mozharness: 2018-09-18 15:28:39.519000Z] Skipping multi-l10n step.
15:28:39     INFO - [mozharness: 2018-09-18 15:28:39.519000Z] Skipping package-source step.
15:28:39     INFO - [mozharness: 2018-09-18 15:28:39.519000Z] Skipping update step.
15:28:39     INFO - [mozharness: 2018-09-18 15:28:39.519000Z] Skipping ensure-upload-path step.
15:28:39     INFO - Running post-run listener: _summarize
15:28:39    ERROR - # TBPL FAILURE #
15:28:39     INFO - [mozharness: 2018-09-18 15:28:39.519000Z] FxDesktopBuild summary:
15:28:39    ERROR - # TBPL FAILURE #
15:28:39     INFO - Running post-run listener: copy_logs_to_upload_dir
15:28:39     INFO - Copying logs to upload dir...
15:28:39     INFO - mkdir: z:\build\build\upload\logs
15:28:39     INFO - Copying logs to upload dir...
15:28:39     INFO - Using _rmtree_windows ...
15:28:39     INFO - Using _rmtree_windows ...
15:28:39     INFO - Using _rmtree_windows ...
15:28:39     INFO - Using _rmtree_windows ...
15:28:39     INFO - Using _rmtree_windows ...
15:28:39     INFO - Using _rmtree_windows ...
15:28:39     INFO - Using _rmtree_windows ...
15:28:39  WARNING - returning nonzero exit status 2
[taskcluster 2018-09-18T15:28:39.583Z]    Exit Code: 2
[taskcluster 2018-09-18T15:28:39.583Z]    User Time: 0s
[taskcluster 2018-09-18T15:28:39.583Z]  Kernel Time: 15.625ms
[taskcluster 2018-09-18T15:28:39.583Z]    Wall Time: 42m45.4050024s
[taskcluster 2018-09-18T15:28:39.583Z]       Result: FAILED
[taskcluster 2018-09-18T15:28:39.583Z] === Task Finished ===



Ted,are there any updates?

Comment 22

[Intermittent Failures Robot]

42 failures in 2902 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #36 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 25
* mozilla-inbound: 10
* try: 1
* mozilla-central: 6

Platform breakdown:
* windows2012-32: 7
* windows2012-64: 35

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-09-17&endday=2018-09-23&tree=all

Comment 23

[Dorel Luca [:dluca]]

This bug has failed 49 times in the last 7 days. All failures seem to occur on windows2012-64 opt.

Recent log:
https://treeherder.mozilla.org/logviewer.html#?job_id=202015491&repo=mozilla-inbound&lineNumber=47576

INFO - Caused by:
19:49:31     INFO -   process didn't exit successfully: `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name build_script_build 'servo\ports\geckolib\tests\build.rs' --crate-type bin --emit=dep-info,link -C opt-level=2 -C codegen-units=1 -C metadata=43baea8111549fcb -C extra-filename=-43baea8111549fcb --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\release\build\stylo_tests-43baea8111549fcb' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'regex=z:/build/build/src/obj-firefox/toolkit/library\release\deps\libregex-9bf31ce226af7ac4.rlib'` (exit code: 1)
19:49:31     INFO - warning: build failed, waiting for other jobs to finish...
19:49:32     INFO - ........error: build failed
19:49:32     INFO - z:/build/build/src/config/rules.mk:1065: recipe for target 'force-cargo-test-run' failed
19:49:32     INFO - mozmake.EXE[2]: *** [force-cargo-test-run] Error 101
19:49:32     INFO - mozmake.EXE[2]: Target 'check' not remade because of errors.
19:49:32     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/toolkit/library/rust'
19:49:32     INFO - z:/build/build/src/config/recurse.mk:101: recipe for target 'toolkit/library/rust/check' failed
19:49:32     INFO - mozmake.EXE[1]: *** [toolkit/library/rust/check] Error 2
19:49:36     INFO - ...........................TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\dom/bindings/Errors.msg | ok
19:49:36     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/ctypes/ctypes.msg | ok
19:49:36     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/js.msg | ok
19:49:36     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/src/jsshell.msg | ok
19:49:36     INFO - TEST-PASS | check_js_msg_encoding.py | z:\build\build\src\js/xpconnect/src/jsshell.msg | ok
19:49:36     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/js/src'
19:49:36     INFO - mozmake.EXE[2]: Entering directory 'z:/build/build/src/obj-firefox/js/src/build'
19:49:36     INFO - mozmake.EXE[2]: Nothing to be done for 'check'.
19:49:36     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/js/src/build'
19:49:45     INFO - ...............................................................
19:49:45     INFO - ----------------------------------------------------------------------
19:49:45     INFO - Ran 120 tests in 19.739s
19:49:45     INFO - OK
19:49:45     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/ipc/ipdl/test/ipdl'
19:49:45     INFO - mozmake.EXE[1]: Target 'recurse_check' not remade because of errors.
19:49:45     INFO - mozmake.EXE[1]: Leaving directory 'z:/build/build/src/obj-firefox'
19:49:45     INFO - z:/build/build/src/config/recurse.mk:32: recipe for target 'check' failed
19:49:45     INFO - mozmake.EXE: *** [check] Error 2
19:49:45     INFO - 390 compiler warnings present.
19:49:45    ERROR - Return code: 2
19:49:45     INFO - TinderboxPrint: check<br/>45/0/0
19:49:45  WARNING - setting return code to 2
19:49:45    ERROR - 'mach build check' did not run successfully. Please check log for errors.
19:49:45     INFO - [mozharness: 2018-09-27 19:49:45.210000Z] Finished check-test step (success)
19:49:45     INFO - [mozharness: 2018-09-27 19:49:45.210000Z] Skipping valgrind-test step.
19:49:45     INFO - [mozharness: 2018-09-27 19:49:45.211000Z] Skipping multi-l10n step.
19:49:45     INFO - [mozharness: 2018-09-27 19:49:45.211000Z] Skipping package-source step.
19:49:45     INFO - Running post-run listener: _summarize
19:49:45    ERROR - # TBPL FAILURE #
19:49:45     INFO - [mozharness: 2018-09-27 19:49:45.211000Z] FxDesktopBuild summary:
19:49:45    ERROR - # TBPL FAILURE #

froydnj: Can you please take a look at this bug?

Comment 24

[Nathan Froyd [:froydnj]]

Better errors:

19:49:31     INFO -    Compiling nsstring v0.1.0 (file:///Z:/build/build/src/servo/support/gecko/nsstring)
19:49:31     INFO -      Running `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name nsstring 'servo\support\gecko\nsstring\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 -C metadata=7c606d1b5930ed66 -C extra-filename=-7c606d1b5930ed66 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'bitflags=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps\libbitflags-51bd89643f2699b8.rlib' --extern 'encoding_rs=z:/build/build/src/obj-firefox/toolkit/library\x86_64-pc-windows-msvc\release\deps\libencoding_rs-73c6531757d1754f.rlib' -C opt-level=2 -C debuginfo=2`
19:49:31     INFO - ..error: linker `link.exe` not found
19:49:31     INFO -   |
19:49:31     INFO -   = note: The system cannot find the file specified. (os error 2)
19:49:31     INFO - note: the msvc targets depend on the msvc linker but `link.exe` was not found
19:49:31     INFO - note: please ensure that VS 2013, VS 2015 or VS 2017 was installed with the Visual C++ option
19:49:31     INFO - error: aborting due to previous error
19:49:31     INFO - error: Could not compile `stylo_tests`.
19:49:31     INFO - Caused by:
19:49:31     INFO -   process didn't exit successfully: `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name build_script_build 'servo\ports\geckolib\tests\build.rs' --crate-type bin --emit=dep-info,link -C opt-level=2 -C codegen-units=1 -C metadata=43baea8111549fcb -C extra-filename=-43baea8111549fcb --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\release\build\stylo_tests-43baea8111549fcb' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'regex=z:/build/build/src/obj-firefox/toolkit/library\release\deps\libregex-9bf31ce226af7ac4.rlib'` (exit code: 1)

I have no idea what's preventing rustc from finding the linker.  Or wait, is this the case where we decided that somebody might have decided to start removing all the tooltool packages before the job has actually completed?

I do see some things that are removing directory trees higher up in the log, but I don't see what they're removing...I guess we could add logging for that?

Comment 25

[Intermittent Failures Robot]

33 failures in 2773 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 21
* mozilla-inbound: 10
* mozilla-central: 2

Platform breakdown:
* windows2012-32: 7
* windows2012-64: 26

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-09-24&endday=2018-09-30&tree=all

Comment 26

[(not currently active) Ted Mielczarek]

Since trying to get more info out of rustc itself seems hard, maybe we should write a small batch script that invokes the actual linker and prints more information if the command fails? We already have a batch script that we use for invoking the 64-bit linker on 32-bit builds:
https://dxr.mozilla.org/mozilla-central/rev/17c314f6930d2b8d6e456aa9e9d41407a45c3008/build/win64/cargo-linker.bat.in

It's also possible that this has the same root cause as bug 1396845. It definitely feels similar! I have a candidate patch in bug 1495798 that will hopefully help mitigate that, it's possible it will help with this as well. A quick glance over the failures from comment 25 does show that they all seem to be in rusttest builds, so it's pretty plausible.

Comment 27

[(not currently active) Ted Mielczarek]

Unfortunately even with the patch from that bug I still managed to hit this error on try:
https://treeherder.mozilla.org/logviewer.html#?job_id=203191245&repo=try&lineNumber=48225

I'll see if I can put together a simple patch to use a batch script and print out some more verbose error logging.

Comment 28

[(not currently active) Ted Mielczarek]

Thinking about it for another minute, it makes sense that that patch would not fix this issue--sccache won't cache Rust compiles that invoke the linker, so none of these compiles can be happening in the server process! The error is happening when rustc is invoked from the client sccache process, which is invoked directly from cargo, which is invoked by make.

Comment 29

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ae29502f2f7f212846aed72b81a05d5c9e44e5da

Comment 30

[(not currently active) Ted Mielczarek]

See bug 1396845 comment 92. My patch should fix this.

Comment 31

[Intermittent Failures Robot]

27 failures in 2708 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 11
* mozilla-inbound: 11
* mozilla-central: 5

Platform breakdown:
* windows2012-32: 7
* windows2012-64: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1470384&startday=2018-10-01&endday=2018-10-07&tree=all

Comment 32

[(not currently active) Ted Mielczarek]

My patch in bug 1396845 landed on autoland on 2018-10-05, and merged to central 2018-10-06. The graph from the previous comment shows this failure disappearing as of that date.