1457583 - Build fails: error: failed to acquire jobserver token: Bad file descriptor (os error 9)

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, defect)

Description

[Jim Blandy :jimb]

I'm building Mozilla Central rev b62ad926cf2a on Fedora 26 on x86_64, and the build died with the following message:

 0:00.64 Clobber not needed.
 0:00.64 Adding make options from /home/jimb/moz/ebg/.mozconfig
    MOZ_OBJDIR=/home/jimb/moz/ebg/obj-bug
    OBJDIR=/home/jimb/moz/ebg/obj-bug
    FOUND_MOZCONFIG=/home/jimb/moz/ebg/.mozconfig
    export FOUND_MOZCONFIG
 0:00.66 /usr/bin/gmake -f client.mk -s
 0:01.94 Elapsed: 0.01s; From dist/private: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:01.94 Elapsed: 0.01s; From dist/public: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:01.97 Elapsed: 0.01s; From dist/xpi-stage: Kept 9 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:02.06 Elapsed: 0.19s; From dist/idl: Kept 873 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:02.55 Elapsed: 0.56s; From _tests: Kept 994 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:02.93 Elapsed: 1.00s; From dist/bin: Kept 2081 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:03.43 Elapsed: 1.44s; From dist/include: Kept 5360 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:04.33 Elapsed: 0.14s; From ../../dist/idl: Kept 873 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:05.15 js/src
 0:35.65 force-cargo-library-build
 0:35.65 security/sandbox/linux/reporter
 0:35.65 security/sandbox/linux/broker
 0:35.65 security/sandbox/common
 0:35.68 security/sandbox/linux/gtest
 0:37.98    Compiling unicode-xid v0.1.0
 0:39.47 thread '<unnamed>' panicked at 'failed to acquire jobserver token: Bad file descriptor (os error 9)', librustc_trans/back/write.rs:1835:29
 0:39.47 stack backtrace:
 0:39.47 error: failed to acquire jobserver token: Bad file descriptor (os error 9)
 0:39.48    0:     0x7fb31d35a76f - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h2fb330a06456ac7e
 0:39.48                                at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
 0:39.48    1:     0x7fb31d3270f0 - std::sys_common::backtrace::print::hf6ccec7cfd12ed67
 0:39.48 error: aborting due to previous error
 0:39.48                                at libstd/sys_common/backtrace.rs:71
 0:39.48                                at libstd/sys_common/backtrace.rs:59
 0:39.48    2:     0x7fb31d32c573 - std::panicking::default_hook::{{closure}}::h464de9af91e4ca8e
 0:39.48                                at libstd/panicking.rs:205
 0:39.48    3:     0x7fb31d32c2fd - std::panicking::default_hook::hcde1a6167c711c5f
 0:39.48                                at libstd/panicking.rs:221
 0:39.48    4:     0x7fb319757145 - rustc::util::common::panic_hook::h4530c027f4a88a1b
 0:39.48    5:     0x7fb31d32cc0c - std::panicking::rust_panic_with_hook::h8dcdd9a7e80a2917
 0:39.48                                at libstd/panicking.rs:461
 0:39.48    6:     0x7fb31d32c810 - std::panicking::begin_panic_fmt::haaf6e293fe070518
 0:39.48                                at libstd/panicking.rs:344
 0:39.49 error: Could not compile `unicode-xid`.
 0:39.49 To learn more, run the command again with --verbose.
 0:39.50 gmake[4]: *** [/home/jimb/moz/ebg/config/rules.mk:949: force-cargo-library-build] Error 101
 0:39.51 gmake[3]: *** [/home/jimb/moz/ebg/config/recurse.mk:73: toolkit/library/rust/target] Error 2
 0:39.51 gmake[3]: *** Waiting for unfinished jobs....
 1:23.59 gmake[2]: *** [/home/jimb/moz/ebg/config/recurse.mk:33: compile] Error 2
 1:23.59 gmake[1]: *** [/home/jimb/moz/ebg/config/rules.mk:418: default] Error 2
 1:23.59 gmake: *** [client.mk:172: build] Error 2
 1:23.60 0 compiler warnings present.
 1:23.62 ccache (direct) hit rate: 0.0%; (preprocessed) hit rate: 0.0%; miss rate: 100.0%

Comment 1

[Jim Blandy :jimb]

It seems to be intermittent; I tried 'mach build' again, and it seemed to work okay.

This blog post by Paul Smith (GNU Make maintainer) describes some circumstances under which make closes the file descriptors for subprocesses. Perhaps rustc's jobserver implementation isn't prepared for those circumstances?

http://make.mad-scientist.net/papers/jobserver-implementation/

Comment 2

[Alex Crichton [:acrichto]]

Hm what version of rustc are you using? Pieces around this have been fixed relatively recently, although there may be some remaining!

Comment 3

[Jim Blandy :jimb]

I'm not able to reproduce this, but I'll re-open if it comes up again.

Comment 4

[Kris Maglione [:kmag]]

I'm now seeing this whenever I try to run configure. The failing command works fine when I run it locally.

DEBUG: rustc: Trying rustc
INFO: /usr/bin/rustc
INFO: checking for cargo... 
DEBUG: cargo: Trying cargo
INFO: /usr/bin/cargo
INFO: checking rustc version... 
DEBUG: Executing: `/usr/bin/rustc --version --verbose`
INFO: 1.29.1
INFO: checking cargo version... 
DEBUG: Executing: `/usr/bin/cargo --version --verbose`
INFO: 1.29.0
DEBUG: Executing: `/usr/bin/rustc --print target-list`
DEBUG: Creating `/tmp/conftestRkjYSg.rs` with content:
DEBUG: | pub extern fn hello() { println!("Hello world"); }
DEBUG: Executing: `/usr/bin/rustc --crate-type staticlib --target=x86_64-unknown-linux-gnu -o /tmp/conftest7Vx1zH.rlib /tmp/conftestRkjYSg.rs`
DEBUG: The command returned non-zero exit status 1.
DEBUG: Its error output was:
DEBUG: | thread '<unnamed>' panicked at 'failed to acquire jobserver token: Bad file descriptor (os error 9)', librustc_codegen_llvm/back/write.rs:1824:29
DEBUG: | note: Run with `RUST_BACKTRACE=1` for a backtrace.
DEBUG: | error: failed to acquire jobserver token: Bad file descriptor (os error 9)
DEBUG: | 
DEBUG: | error: aborting due to previous error
DEBUG: | 
ERROR: Cannot compile for x86_64-pc-linux-gnu with /usr/bin/rustc

Seeing as this prevents me from doing my job, marking critical.

Comment 5

[Kris Maglione [:kmag]]

Note: This started after I upgraded to rust 1.29.1. Downgrading to 1.29.0 didn't help. I can't downgrade to 1.28.* because the build system won't let me.

Comment 6

[Mike Hommey [:glandium]]

It happening on configure is different from comment 0. I've had this happen in a very specific case: with a wrapper Makefile invoking configure, and with that Makefile being invoked with -j<n>.

Comment 7

[Kris Maglione [:kmag]]

Apparently this has something to do with configure running rustc with fd 4 opened to $OBJDIR/config.log. Hacking util.configure to run it with a wrapper that closes that fd makes the problem go away.

Comment 8

[Kris Maglione [:kmag]]

Attachment: Bug 1457583: Pass close_fds when running commands for configure. r=glandium

Running rustc with an open fd 4 sometimes causes it to fail with an obscure
error when it tries to connect to a job server on that fd. Closing the fd
solves the problem.

close_fds does what we need, and is the default value in Python 3, but not
in Python 2.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Comment 10

[Mike Hommey [:glandium]]

If I recall correctly, there was some bug in the jobserver crate back then, which has been fixed since.