1520159 - rustc reliably dies partway through a clean build on Mac

Status: RESOLVED DUPLICATE of bug 1500436

(Firefox Build System :: Toolchains, defect)

Description

[Boris Zbarsky [:bzbarsky]]

Attachment: log-build.txt

For at least the last several weeks, whenever I do a clean build (ccache, bit no sccache) on Mac OSX 10.12, rustc dies partway through the build. If I just restart the build it usually completes, though sometimes I'll need multiple restarts.

This is particularly frustrating because it means I can't build overnight.

The full build log is attached, but the key part seems to be:

17:48.59 thread 'main' panicked at 'failed printing to stdout: Resource temporarily unavailable (os error 35)', libstd/io/stdio.rs:700:9
17:48.59 stack backtrace:
...
17:48.59    6:        0x100e0e4a0 - std::panicking::begin_panic_fmt::h2bdefd173f570a0b
17:48.59    7:        0x100e03b8b - std::io::stdio::_print::hd811110021038dff
17:48.59    8:        0x1007bb02c - cargo::core::compiler::job_queue::JobQueue::drain_the_queue::h5d61210172b85c62
17:48.59    9:        0x100742f42 - crossbeam_utils::thread::scope::h299c5d8d8ddddf08
17:48.59   10:        0x1007b93b5 - cargo::core::compiler::job_queue::JobQueue::execute::h5aaf65413952cf5b

Comment 1

[Boris Zbarsky [:bzbarsky]]

Oh, and I expect "os error 35" is EAGAIN.

Also filed as a rustc issue at https://github.com/rust-lang/rust/issues/57632

Comment 2

[Kurt Miller]

Attachment: patch-python_mozbuild_mozbuild_action_node_py

Comment 3

[Kurt Miller]

This failure has been seen on OpenBSD/amd64 as well. I tracked the root cause back to nodejs 10 setting stdout/stderr as non-blocking and leaving it that way upon exit. Later in the build cargo runs and when it outputs a series of printf's to stdout, it receives EAGAIN but doesn't retry. It fails as seen in this bug report.

This problem has been fixed in node under two bug reports and three pull requests:
https://github.com/nodejs/node/issues/14752
https://github.com/nodejs/node/issues/28479
https://github.com/nodejs/node/pull/24260
https://github.com/nodejs/node/pull/28490
https://github.com/nodejs/node/pull/28535

Building with node 12.7 or later should restore stdout/stderr back to blocking and correct this build failure. Meanwhile, I have attached a work-around diff that can be used to reset stdout/stderr back to blocking mode for older node versions.

Comment 4

[Landry Breuil (:gaston)]

Comment on attachment 9104378 [details] [diff] [review]
patch-python_mozbuild_mozbuild_action_node_py

Mike, something to consider for mozbuild ? At least for build compat with older nodejs..

Comment 5

[Mike Hommey [:glandium]]

As of bug 1500436, node doesn't have direct access to stdout/stderr from make, so it can't screw it up. Also, how is that related to comment 0?

Comment 6

[Kurt Miller]

Hi Mike. The fix from bug 1500436 is a better way to deal with this then what I attached here. It isolates node's non-blocking modification of stdout and stderr to to its side of the pipe. Sorry I missed this was fixed already here. The issue was noticed on our build of firefox 68.2.0esr which doesn't have the node popen/pipe fix.

Regarding your question about comment 0: Write to a file descriptor that is set to non-blocking is the only way to get EAGAIN/EWOULDBLOCK back as an error. The old code that doesn't use popen and pipes to invoke node causes stdout and stderr to be set to non-blocking from that point in the build onward. Most applications are not prepared to deal with a non-blocking stdout/stderr. When cargo writes quickly to stdout it fills the kernel buffer and gets back EAGAIN/EWOULDBLOCK which it treats as an error and exits with 'failed printing to stdout: Resource temporarily unavailable'.

I believe this bug should be closed as it has been fixed already by bug 1500436.