1570349 - Build often hangs in compile step

Status: NEW

(Firefox Build System :: General, defect)

Description

[Simon Giesecke [:sg] [he/him]]

Quite often, my mach build appears to hang forever at some random point in the compile step, e.g. here:

11:18.27 gfx/ycbcr
11:18.29 gfx/src
11:18.33 gfx/2d
11:18.35 gfx/angle/targets/angle_common
11:23.21    Compiling gkrust-shared v0.1.0 (/home/osboxes/work/mozilla-unified/toolkit/library/rust/shared)
11:24.80    Compiling gkrust v0.1.0 (/home/osboxes/work/mozilla-unified/toolkit/library/rust)
11:30.09     Finished dev [optimized + debuginfo] target(s) in 8m 53s
11:37.86     Finished dev [optimized + debuginfo] target(s) in 9m 01s

When I break this after several minutes (here after ca. 15 minutes), I get the following:

^CProcess Process-1:export export compile misc libs tools
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/osboxes/work/mozilla-unified/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 121, in _collect
    while not _poll(pipe, poll_interval=sleep_interval):
  File "/home/osboxes/work/mozilla-unified/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 87, in _poll
    return pipe.poll(poll_interval)
KeyboardInterrupt
25:46.65 gmake[4]: *** Deleting file 'system_utils.o'
25:46.65 gmake[3]: *** [/home/osboxes/work/mozilla-unified/config/recurse.mk:74: gfx/angle/targets/angle_common/target] Interrupt
25:46.65 gmake[4]: *** [/home/osboxes/work/mozilla-unified/config/makefiles/rust.mk:288: force-cargo-host-library-build] Interrupt
25:46.65 gmake[3]: *** [/home/osboxes/work/mozilla-unified/config/recurse.mk:74: toolkit/crashreporter/rust/host] Interrupt
25:46.65 gmake[2]: *** [/home/osboxes/work/mozilla-unified/config/recurse.mk:34: compile] Interrupt
25:46.66 gmake[1]: *** [/home/osboxes/work/mozilla-unified/config/rules.mk:391: default] Interrupt
25:46.66 gmake: *** [client.mk:125: build] Interrupt
25:46.66 323 compiler warnings present.
25:46.96 ccache (direct) hit rate: 0.0%; (preprocessed) hit rate: 0.0%; miss rate: 100.0%
25:46.96 /usr/bin/notify-send --app-name=Mozilla Build System Mozilla Build System Build failed

Usually, I have to break and call mach build again several times when there were many changes.

I have no idea if the problem is within the resourcemonitor, or this is just the effect of something else hanging. I will add information on that when it happens again.

Comment 1

[Simon Giesecke [:sg] [he/him]]

I hit this again and ran pstree:

[osboxes@sigiesec-mozbln mozilla-unified]$ pstree 26452 -ac
python2.7 ./mach build
  ├─gmake -f client.mk -s
  │   └─gmake -j6 -C /home/osboxes/work/mozilla-unified/obj-x86_64-pc-linux-gnu
  │       └─gmake compile
  │           └─gmake recurse_compile
  │               ├─gmake -C js/src/rust target
  │               │   └─cargo rustc --frozen --manifest-path/home/osboxes/work/mozil
  │               ├─gmake -C toolkit/library/rust target
  │               │   └─cargo rustc --frozen --manifest-path/home/osboxes/work/mozil
  │               ├─(gmake)
  │               ├─(gmake)
  │               ├─(gmake)
  │               ├─(gmake)
  │               ├─gmake -C dom/base target
  │               │   └─(clang++)
  │               └─gmake -C dom/bindings target
  │                   └─(clang++)
  ├─python2.7 ./mach build
  ├─{python2.7}
  └─{python2.7}

Comment 2

[Nathan Froyd [:froydnj]]

Are those processes still running (as shown by top, or whatever)?

Comment 3

[Simon Giesecke [:sg] [he/him]]

Yes, those processes are still running when the build appears stuck, before I would abort by Ctrl-C (as shown by pstree here). The processes in parentheses (gmake, clang++) are zombie processes, so I think that the resourcemonitor should probably have queries their exit status, but did not do that yet. The cargo processes are still running this morning, this is where they hang:

(gdb) thread apply all bt

Thread 1 (Thread 0x7f90233d79c0 (LWP 26667)):
#0  0x00007f902360e60b in flock () at ../sysdeps/unix/syscall-template.S:78
#1  0x000055c19905aa2e in <std::fs::File as fs2::FileExt>::lock_exclusive ()
#2  0x000055c198ea3add in cargo::util::flock::Filesystem::open::{{closure}} ()
#3  0x000055c198ea415a in cargo::util::flock::acquire ()
#4  0x000055c198ea3934 in cargo::util::flock::Filesystem::open ()
#5  0x000055c198c72d99 in cargo::core::compiler::layout::Layout::at ()
#6  0x000055c198c72acb in cargo::core::compiler::layout::Layout::new ()
#7  0x000055c198f2a523 in cargo::core::compiler::context::Context::prepare_units ()
#8  0x000055c198f26855 in cargo::core::compiler::context::Context::compile ()
#9  0x000055c198fe275f in cargo::ops::cargo_compile::compile_ws ()
#10 0x000055c198fe1077 in cargo::ops::cargo_compile::compile ()
#11 0x000055c198c0311c in cargo::commands::rustc::exec ()
#12 0x000055c198be507a in cargo::cli::main ()
#13 0x000055c198c0cb1b in cargo::main ()
#14 0x000055c198c24673 in std::rt::lang_start::{{closure}} ()
#15 0x000055c199489dc3 in std::rt::lang_start_internal::{{closure}} () at src/libstd/rt.rs:49
#16 std::panicking::try::do_call () at src/libstd/panicking.rs:293
#17 0x000055c199495aaa in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:85
#18 0x000055c19948a98d in std::panicking::try () at src/libstd/panicking.rs:272
#19 std::panic::catch_unwind () at src/libstd/panic.rs:394
#20 std::rt::lang_start_internal () at src/libstd/rt.rs:48
#21 0x000055c198c0f1a2 in main ()

(gdb) thread apply all bt

Thread 1 (Thread 0x7fde387b29c0 (LWP 26666)):
#0  0x00007fde389ed588 in __GI___poll (fds=0x7ffc20bc5d28, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055a13e76cee5 in jobserver::imp::Client::acquire ()
#2  0x000055a13e7706a9 in jobserver::Client::acquire_raw ()
#3  0x000055a13e1bf187 in cargo::util::flock::acquire ()
#4  0x000055a13e1be934 in cargo::util::flock::Filesystem::open ()
#5  0x000055a13df8dd99 in cargo::core::compiler::layout::Layout::at ()
#6  0x000055a13df8dacb in cargo::core::compiler::layout::Layout::new ()
#7  0x000055a13e245523 in cargo::core::compiler::context::Context::prepare_units ()
#8  0x000055a13e241855 in cargo::core::compiler::context::Context::compile ()
#9  0x000055a13e2fd75f in cargo::ops::cargo_compile::compile_ws ()
#10 0x000055a13e2fc077 in cargo::ops::cargo_compile::compile ()
#11 0x000055a13df1e11c in cargo::commands::rustc::exec ()
#12 0x000055a13df0007a in cargo::cli::main ()
#13 0x000055a13df27b1b in cargo::main ()
#14 0x000055a13df3f673 in std::rt::lang_start::{{closure}} ()
#15 0x000055a13e7a4dc3 in std::rt::lang_start_internal::{{closure}} () at src/libstd/rt.rs:49
#16 std::panicking::try::do_call () at src/libstd/panicking.rs:293
#17 0x000055a13e7b0aaa in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:85
#18 0x000055a13e7a598d in std::panicking::try () at src/libstd/panicking.rs:272
#19 std::panic::catch_unwind () at src/libstd/panic.rs:394
#20 std::rt::lang_start_internal () at src/libstd/rt.rs:48
#21 0x000055a13df2a1a2 in main ()

So maybe it is a problem of cargo? (I also opened https://github.com/rust-lang/cargo/issues/7200 now)

Comment 4

[Simon Giesecke [:sg] [he/him]]

I have another instance without a cargo process running, and actually no active 'worker' processes, only zombies:

[osboxes@sigiesec-mozbln mozilla-unified]$ pstree 28563 -ac
python2.7 ./mach build
  ├─gmake -f client.mk -s
  │   └─gmake -j6 -C /home/osboxes/work/mozilla-unified/obj-x86_64-pc-linux-gnu
  │       └─gmake compile
  │           └─gmake recurse_compile
  │               ├─(gmake)
  │               ├─(gmake)
  │               ├─(gmake)
  │               ├─(gmake)
  │               ├─(gmake)
  │               └─(gmake)
  ├─(python2.7)
  ├─{python2.7}
  └─{python2.7}