1429847 - When remoting audio streams, bump the priority of the child process thread to avoid underruns

Status: RESOLVED FIXED

(Core :: Audio/Video: cubeb, enhancement, P2)

Description

[Paul Adenot (:padenot)]

This cannot be done on stock Linux, but we can do something on OSX, Windows and Android.

Comment 1

[Paul Adenot (:padenot)]

Attachment: Bug 1429847 - Bump the audio callback thread priority to real-time when it drives a MediaStreamGraph WIP

Review commit: https://reviewboard.mozilla.org/r/212132/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/212132/

Comment 2

[Paul Adenot (:padenot)]

Attachment: test case

This file is a test case for the attached patch (which is a proof of concept, certainly not final, and certainly not tuned).

The four leftmost buttons allow to add and remove OscillatorNodes to a MediaStreamGraph, creating or removing audio callback load (and doing some volume management so it does not blow up our ears).

The other buttons add and remove web workers that are touching some memory randomly and performing some silly computation, to stress the machine.

With the patch compiled in, and its pref set to "false" (i.e. the thread is not bumped), if I create 8 workers, I have glitches (on my machine, mbp 2016 maxed up) at around 400 oscillators/

With pref flipped (i.e. the thread is bumped to real-time), if I create 8 workers, I have very solid playback until about 800 something oscillators.

I think maybe this should go in the audioipc create, not sure.

I have something similar for Windows, but we don't do audio ipc on windows just yet, so having the mac version was more important.

Comment 3

[Paul Adenot (:padenot)]

Some resources on this:
- Some docs: https://developer.apple.com/library/content/documentation/Darwin/Conceptual/KernelProgramming/scheduler/scheduler.html#//apple_ref/doc/uid/TP30000905-CH211-BABHGEFA
- A technical note that talks about real-time: https://developer.apple.com/library/content/technotes/tn2169/_index.html
- What chrome does, that we can't use as is because we batch the processing differently: https://cs.chromium.org/chromium/src/base/threading/platform_thread_mac.mm?l=67

Matthew, Dan, would you prefer this to go in the audioipc crate, or in gecko? Ultimately, I'll factor it out so that we can use it for playback as well.

Comment 4

[Matthew Gregan [:kinetik]]

Thanks for experimenting with this.  It was on my list of things we'd lose and might need to reinstate manually when remoting audio.  I think we'll need similar code inside the audioipc crate ultimately because we're moving towards running the callbacks from a thread pool, so we'll need to tweak all of the threads participating in the pool.

Comment 5

[Paul Adenot (:padenot)]

I'm going to land a variant of this now (in C++), because WebRTC calls have taken rather a serious quality hit. The audio output is constantly under-runing, even with the recent optimizations we have landed.

Comment 6

[Matthew Gregan [:kinetik]]

We could also pref the remoting off on macOS until we've addressed this in the best way, rather than rushing to land code we're going to have to revert at some later date.  Enabling remoting on macOS was an experiment, after all, so it's not a big deal to disable it yet.

Comment 7

[u480271]

Thanks :padenot for putting together this test.

I agree that we should disable on macOS and look for a better fix than putting in some C+ band aids.

:kinetik can you land a patch to disable the pref on macOS

Comment 8

[Matthew Gregan [:kinetik]]

I'll do it in bug 1425788.

Comment 9

[Paul Adenot (:padenot)]

Thanks. I've started working on a rust version, but it's nowhere near done.

Comment 10

[Matthew Gregan [:kinetik]]

(In reply to Paul Adenot (:padenot) from comment #9)
> Thanks. I've started working on a rust version, but it's nowhere near done.

If you're making the changes against https://github.com/djg/audioipc-2, please be aware that the tokio-rebase branch is about to become master, so you'll need to base any work on that since it's a huge refactoring.

Comment 11

[Paul Adenot (:padenot)]

Tentatively taking.

Comment 12

[Paul Adenot (:padenot)]

I have some code here [0], it's almost finished now. What I'm missing is the "revert" part, to have the thread be scheduled normally.

[0] https://github.com/padenot/audio_thread_priority

Comment 13

[Paul Adenot (:padenot)]

This is now done and available at https://crates.io/crates/audio_thread_priority. Next up is a PR to audioipc and then we'll be able to re-pref on. Also this include Windows code in rust as well.

This is a separate crate because we'll be using this for AudioWorklet as well.

Comment 14

[Paul Adenot (:padenot)]

Dan, anything more we need here? The PR to fix this is at https://github.com/djg/audioipc-2/pull/27

Comment 15

[u480271]

It's all good :padenot. :kinetik disabled MacOS for now and I want him to take a look once he's available.

Comment 16

[Paul Adenot (:padenot)]

We now have a way to do this on Linux, it's all implemented in Rust. I'm planning to cbindgen it and to use it in Gecko. It should do wonders, especially on Linux and/or when remoting is on. The Linux bit needs a couple adjustments ideally, but it's largely working.

Comment 17

[Paul Adenot (:padenot)]

Attachment: Bug 1429847 - Add `audio_thread_priority` to mozilla-central. r?kinetik

Comment 18

[Paul Adenot (:padenot)]

Attachment: Bug 1429847 - Vendor rust, to pick up the changes. r?kinetik

Comment 19

[Paul Adenot (:padenot)]

Attachment: Bug 1429847 - Bump the audio threads that are doing real-time work in Gecko. r?kinetik

This is way to much threads, but until we refactor the design a bit as planned,
it works.

Comment 20

[Paul Adenot (:padenot)]

Matthew, this is a first stab at doing this, mainly to fix the performances issues on Linux (but it works on other platforms too). It makes quite a difference, especially when doing heavy webrtc calls or when running heavy web audio api application, but also when just doing playback and the machine is otherwise loaded (i.e. doing a build when listening to music).

I'm not happy about the thread setup: it requires bumping too much threads to real-time priority, and there are too many context switches but I think it's best talked about and re-designed in person in front of measurements and with data about the problem.

Comment 21

[Matthew Gregan [:kinetik]]

Thanks Paul! I filed bug 1551391 to track/discuss the audioipc threading model;feel free to add your thoughts there and/or we can discuss it in person in Whistler too.

Comment 22

[Paul Adenot (:padenot)]

Attachment: Bug 1429847 - Promote the GraphRunner thread. r?pehrsons

Comment 23

[Paul Adenot (:padenot)]

I've changed the OSX bits to make it behave like a normal CoreAudio thread (thanks to Stephane Letz from GRAME for all the hard-to-find info), removed all the linux priority things, and made it so we try only once to promote a thread (I think it's better, if it doesn't work, it doesn't work).

This can probably be landed, and will kick in when remoting is activated.

I've also promoted the GraphRunner thread to real-time, until we remove it.

Comment 24

[Paul Adenot (:padenot)]

Attachment: Bug 1429847 - Allow promoting CpuPool threads for audio remoting separately from the client creation. r?kinetik

Comment 25

[Paul Adenot (:padenot)]

Attachment: Bug 1429847 - Promote audio threads to real-time priority before locking down the sandbox. r?kinetik

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1d9377463591
Add `audio_thread_priority` to mozilla-central. r=kinetik
https://hg.mozilla.org/integration/autoland/rev/15fadc5ad294
Vendor rust, to pick up the changes. r=kinetik
https://hg.mozilla.org/integration/autoland/rev/b3cb4712a543
Promote the GraphRunner thread. r=pehrsons
https://hg.mozilla.org/integration/autoland/rev/6ccf1ede2479
Allow promoting CpuPool threads for audio remoting separately from the client creation. r=kinetik
https://hg.mozilla.org/integration/autoland/rev/f058271e55d9
Promote audio threads to real-time priority before locking down the sandbox. r=kinetik

Comment 27

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/1d9377463591
https://hg.mozilla.org/mozilla-central/rev/15fadc5ad294
https://hg.mozilla.org/mozilla-central/rev/b3cb4712a543
https://hg.mozilla.org/mozilla-central/rev/6ccf1ede2479
https://hg.mozilla.org/mozilla-central/rev/f058271e55d9

Comment 28

[Mike Hommey [:glandium]]

Paul, how did you vendor the crates? If I revert https://hg.mozilla.org/mozilla-central/rev/15fadc5ad294 and run mach vendor rust, I get the same error as bug 1561102, which makes me wonder how you did it without hitting that.

Comment 29

[u636583]

This breaks linking with ac_add_options --disable-dbus (ie. build regression):

 0:03.93 make[4]: Leaving directory '/home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/browser/app'
 0:14.43 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_error_init'
 0:14.52 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_bus_get_private'
 0:14.60 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_connection_set_exit_on_disconnect'
 0:14.69 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_connection_add_filter'
 0:14.78 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_connection_set_watch_functions'
 0:14.88 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_error_free'
 0:14.97 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_error_free'
 0:15.06 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_error_free'
 0:15.16 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_error_free'
 0:15.24 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_message_new_method_call'
 0:15.32 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_message_iter_init_append'
 0:15.42 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function audio_thread_priority::promote_current_thread_to_real_time: error: undefined reference to 'dbus_message_unref'
 0:15.50 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function core::ptr::real_drop_in_place: error: undefined reference to 'dbus_connection_close'
 0:15.60 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function core::ptr::real_drop_in_place: error: undefined reference to 'dbus_connection_unref'
 0:15.68 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function core::ptr::real_drop_in_place: error: undefined reference to 'dbus_message_unref'
 0:15.77 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function core::ptr::real_drop_in_place: error: undefined reference to 'dbus_message_unref'
 0:15.95 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::strings::Signature::new: error: undefined reference to 'dbus_error_init'
 0:16.04 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::strings::Signature::new: error: undefined reference to 'dbus_signature_validate_single'
 0:16.21 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function std::sync::once::Once::call_once::{{closure}}: error: undefined reference to 'dbus_threads_init_default'
 0:16.29 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function core::ops::function::FnOnce::call_once{{vtable-shim}}: error: undefined reference to 'dbus_threads_init_default'
 0:16.38 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::iter_append: error: undefined reference to 'dbus_message_iter_open_container'
 0:16.47 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::iter_append: error: undefined reference to 'dbus_message_iter_close_container'
 0:16.55 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::iter_append: error: undefined reference to 'dbus_message_iter_open_container'
 0:16.64 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::iter_append: error: undefined reference to 'dbus_message_iter_close_container'
 0:16.72 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::iter_append: error: undefined reference to 'dbus_message_iter_open_container'
 0:16.82 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::iter_append: error: undefined reference to 'dbus_message_iter_close_container'
 0:16.91 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::iter_append: error: undefined reference to 'dbus_message_iter_open_container'
 0:17.00 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::iter_append: error: undefined reference to 'dbus_message_iter_close_container'
 0:17.09 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::iter_append: error: undefined reference to 'dbus_message_iter_append_basic'
 0:17.19 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::iter_append: error: undefined reference to 'dbus_message_iter_append_basic'
 0:17.27 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter: error: undefined reference to 'dbus_message_iter_next'
 0:17.35 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter: error: undefined reference to 'dbus_message_iter_next'
 0:17.44 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_message_iter_get_arg_type'
 0:17.52 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_message_iter_get_basic'
 0:17.62 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_message_iter_recurse'
 0:17.72 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_message_iter_recurse'
 0:17.82 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_message_iter_get_basic'
 0:17.91 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_message_iter_recurse'
 0:17.99 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_message_iter_get_signature'
 0:18.09 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_free'
 0:18.19 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_message_iter_get_basic'
 0:18.27 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_message_iter_get_basic'
 0:18.66 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::message::MessageItem::from_iter_single: error: undefined reference to 'dbus_message_iter_recurse'
 0:19.08 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::strings::Path::check_valid: error: undefined reference to 'dbus_error_init'
 0:19.16 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::strings::Path::check_valid: error: undefined reference to 'dbus_validate_path'
 0:19.33 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::prop::Props::get: error: undefined reference to 'dbus_message_new_method_call'
 0:19.43 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::prop::Props::get: error: undefined reference to 'dbus_message_iter_init_append'
 0:19.51 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::prop::Props::get: error: undefined reference to 'dbus_error_init'
 0:19.61 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::prop::Props::get: error: undefined reference to 'dbus_set_error_from_message'
 0:19.77 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::prop::Props::get: error: undefined reference to 'dbus_message_iter_init'
 0:20.04 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::prop::Props::get: error: undefined reference to 'dbus_set_error'
 0:20.15 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::prop::Props::get: error: undefined reference to 'dbus_message_unref'
 0:20.32 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::connection::Connection::send_with_reply_and_block: error: undefined reference to 'dbus_connection_send_with_reply_and_block'
 0:23.31 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::strings::Member::check_valid: error: undefined reference to 'dbus_validate_member'
 0:23.56 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::strings::Interface::check_valid: error: undefined reference to 'dbus_validate_interface'
 0:23.73 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::watch::WatchList::get_watch: error: undefined reference to 'dbus_watch_get_unix_fd'
 0:23.82 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::watch::WatchList::get_watch: error: undefined reference to 'dbus_watch_get_enabled'
 0:23.92 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::watch::WatchList::get_watch: error: undefined reference to 'dbus_watch_get_flags'
 0:24.02 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function std::panicking::try::do_call: error: undefined reference to 'dbus_message_ref'
 0:24.10 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function dbus::connection::default_filter_callback: error: undefined reference to 'dbus_message_get_type'
 0:24.28 /home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libgkrust.a(gkrust-4b4c1bdef7d934df.gkrust.ba3rei45-cgu.0.rcgu.o):gkrust.ba3rei45-cgu.0:function <dbus::strings::BusName as core::convert::From<&str>>::from: error: undefined reference to 'dbus_validate_bus_name'
 0:26.85 clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
 0:26.85 make[4]: *** [/home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/config/rules.mk:669: libxul.so] Error 1
 0:26.86 make[4]: Leaving directory '/home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu/toolkit/library'
 0:26.86 make[3]: *** [/home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/config/recurse.mk:74: toolkit/library/target] Error 2
 0:26.86 make[3]: Leaving directory '/home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu'
 0:26.86 make[2]: *** [/home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/config/recurse.mk:34: compile] Error 2
 0:26.86 make[2]: Leaving directory '/home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu'
 0:26.86 make[1]: *** [/home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/config/rules.mk:400: default] Error 2
 0:26.86 make[1]: Leaving directory '/home/user/build/1packages/4used/firefox-hg/makepkg_pacman/firefox-hg/src/firefox-hg/obj-x86_64-pc-linux-gnu'
 0:26.86 make: *** [client.mk:125: build] Error 2
 0:26.89 52 compiler warnings present.

Comment 30

[Julien Cristau [:jcristau]]

Please file a new bug for comment 29

Comment 31

[u636583]

Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1561207
Thanks.

Comment 32

[Paul Adenot (:padenot)]

(In reply to Mike Hommey [:glandium] from comment #28)

Paul, how did you vendor the crates? If I revert https://hg.mozilla.org/mozilla-central/rev/15fadc5ad294 and run mach vendor rust, I get the same error as bug 1561102, which makes me wonder how you did it without hitting that.

Hrm mach appears to return 0 when this errors out. I don't have the machine where I did this handy (still traveling), but I was probably doing something like mach vendor rust && mach build and it got lost in the output.

Why is BSD-2-clause not approved? We use a bunch of code with the same license already? I'm thinking of making this fail a bit harder and rework the error message so that it's a bit easier to know what to do.

Comment 33

[:gerard-majax]

Attachment: syslog output of rtkit

Paul, I'm getting some syslog spam everytime I restart Nightly:

```

Comment 34

[:gerard-majax]

Should I file a bug ?

Comment 35

[Paul Adenot (:padenot)]

This is expected for now.