1371319 - Startup crash in libpulsecommon-10.0.so@0x54b39 following enabling Rust port of cubeb PulseAudio backend

Status: RESOLVED FIXED

(Core :: Audio/Video: cubeb, defect)

Description

[Andreas Tolfsen ❲:ato❳]

This bug was filed from the Socorro interface and is 
report bp-ac5e48e5-3c13-4553-8b00-b6fbc0170608.
=============================================================

This was originally reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1360060#c25, but creating a separate bug to track the crash.

The hand-generated stack is:

> #01: /home/ato/src/gecko/js/src/ds/MemoryProtectionExceptionHandler.cpp:267
> #02: /home/ato/src/gecko/js/src/wasm/WasmSignalHandlers.cpp:1313
> #03: ??:?
> #04: ??:?
> #05: ??:0
> #06: /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/pulse-ffi/src/ffi_funcs.rs:1235
> #07: /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/src/backend/context.rs:161
> #08: /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/src/backend/context.rs:80
> #09: cubeb_pulse.cgu-1.rs:?
> #10: cubeb_pulse.cgu-1.rs:?
> #11: /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/src/backend/context.rs:131
> #12: /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/src/capi.rs:12
> #13: /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/src/capi.rs:249
> #14: /home/ato/src/gecko/media/libcubeb/src/cubeb.c:211
> #15: /home/ato/src/gecko/dom/media/CubebUtils.cpp:342
> #16: /home/ato/src/gecko/dom/media/CubebUtils.cpp:205
> #17: /home/ato/src/gecko/dom/ipc/ContentChild.cpp:1470
> #18: /home/ato/src/gecko/obj-x86_64-pc-linux-gnu/ipc/ipdl/PContentChild.cpp:5868 (discriminator 1)
> #19: /home/ato/src/gecko/ipc/glue/MessageChannel.cpp:2075
> #20: /home/ato/src/gecko/ipc/glue/MessageChannel.cpp:2001
> #21: /home/ato/src/gecko/ipc/glue/MessageChannel.cpp:1871
> #22: /home/ato/src/gecko/ipc/glue/MessageChannel.cpp:1904
> #23: /home/ato/src/gecko/xpcom/threads/nsThread.cpp:1369 (discriminator 1)
> #24: /home/ato/src/gecko/xpcom/threads/nsThreadUtils.cpp:472 (discriminator 3)
> #25: /home/ato/src/gecko/ipc/glue/MessagePump.cpp:96
> #26: /home/ato/src/gecko/ipc/glue/MessagePump.cpp:302 (discriminator 1)
> #27: /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:239
> #28: /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:232
> #29: /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:211
> #30: /home/ato/src/gecko/widget/nsBaseAppShell.cpp:156 (discriminator 1)
> #31: /home/ato/src/gecko/toolkit/xre/nsEmbedFunctions.cpp:896
> #32: /home/ato/src/gecko/ipc/glue/MessagePump.cpp:269 (discriminator 1)
> #33: /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:239
> #34: /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:232
> #35: /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:211
> #36: /home/ato/src/gecko/toolkit/xre/nsEmbedFunctions.cpp:712
> #37: /home/ato/src/gecko/toolkit/xre/Bootstrap.cpp:65
> #38: ??:0
> #39: ??:0
> #40: ??:0
> #41: ??:0

Comment 1

[Andreas Tolfsen ❲:ato❳]

And the backtrace from attaching to the thread:

> (gdb) bt
> #0  0x00007ffff6e3424d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007ffff6e3419a in sleep () from /lib/x86_64-linux-gnu/libc.so.6
> #2  0x00007fffe8f974ab in ah_crap_handler (signum=11) at /home/ato/src/gecko/toolkit/xre/nsSigHandlers.cpp:103
> #3  0x00007fffe8f975a9 in child_ah_crap_handler (signum=11) at /home/ato/src/gecko/toolkit/xre/nsSigHandlers.cpp:115
> #4  0x00007fffe9b07f5f in js::UnixExceptionHandler (signum=11, info=0x7fffffff99b0, context=0x7fffffff9880)
>     at /home/ato/src/gecko/js/src/ds/MemoryProtectionExceptionHandler.cpp:267
> #5  0x00007fffea0126b9 in WasmFaultHandler<(Signal)0> (signum=11, info=0x7fffffff99b0, context=0x7fffffff9880)
>     at /home/ato/src/gecko/js/src/wasm/WasmSignalHandlers.cpp:1313
> #6  <signal handler called>
> #7  0x00007fffd027fb39 in pa_thread_is_running () from /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-10.0.so
> #8  0x00007fffd04e0e67 in pa_threaded_mainloop_stop () from /usr/lib/x86_64-linux-gnu/libpulse.so.0
> #9  0x00007fffea81a144 in pulse_ffi::ffi_funcs::dynamic_fns::pa_threaded_mainloop_stop (m=0x7fffd9d995c0)
>     at /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/pulse-ffi/src/ffi_funcs.rs:1234
> #10 0x00007fffea81aed6 in cubeb_pulse::backend::context::Context::destroy (self=0x7ffff6958820)
>     at /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/src/backend/context.rs:161
> #11 0x00007fffea81a97d in cubeb_pulse::backend::context::{{impl}}::drop (self=0x7ffff6958820)
>     at /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/src/backend/context.rs:79
> #12 0x00007fffea819211 in drop::hfe3e68367ff567c9 () at /checkout/src/libcore/cmp.rs:742
> #13 0x00007fffea8191d4 in drop::hea5094ce59b43108 () at /checkout/src/libcore/cmp.rs:742
> #14 0x00007fffea81ad4b in cubeb_pulse::backend::context::Context::new (name=0x7fffd9c301f0 "")
>     at /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/src/backend/context.rs:152
> #15 0x00007fffea379925 in cubeb_pulse::capi::capi_init (c=0x7fffed8bda40 <mozilla::(anonymous namespace)::sCubebContext>, 
>     context_name=0x7fffd9c301f0 "") at /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/src/capi.rs:11
> #16 0x00007fffea37a49d in cubeb_pulse::capi::pulse_rust_init (c=0x7fffed8bda40 <mozilla::(anonymous namespace)::sCubebContext>, 
>     context_name=0x7fffd9c301f0 "") at /home/ato/src/gecko/media/libcubeb/cubeb-pulse-rs/src/capi.rs:249
> #17 0x00007fffe75d382c in cubeb_init (context=0x7fffed8bda40 <mozilla::(anonymous namespace)::sCubebContext>, 
>     context_name=0x7fffd9c301f0 "", backend_name=0x0) at /home/ato/src/gecko/media/libcubeb/src/cubeb.c:211
> #18 0x00007fffe5b14ab9 in mozilla::CubebUtils::GetCubebContextUnlocked () at /home/ato/src/gecko/dom/media/CubebUtils.cpp:342
> #19 0x00007fffe5b0dfe3 in mozilla::CubebUtils::GetCubebContext () at /home/ato/src/gecko/dom/media/CubebUtils.cpp:205
> #20 0x00007fffe645cb55 in mozilla::dom::ContentChild::RecvSetProcessSandbox (this=0x7ffff692d020, aBroker=...)
>     at /home/ato/src/gecko/dom/ipc/ContentChild.cpp:1470
> #21 0x00007fffe37b4a04 in mozilla::dom::PContentChild::OnMessageReceived (this=0x7ffff692d020, msg__=...)
>     at /home/ato/src/gecko/obj-x86_64-pc-linux-gnu/ipc/ipdl/PContentChild.cpp:5868
> #22 0x00007fffe3105123 in mozilla::ipc::MessageChannel::DispatchAsyncMessage (this=0x7ffff692d140, aMsg=...)
>     at /home/ato/src/gecko/ipc/glue/MessageChannel.cpp:2075
> #23 0x00007fffe3103cd7 in mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) (this=0x7ffff692d140, 
>     aMsg=<unknown type in /home/ato/src/gecko/obj-x86_64-pc-linux-gnu/dist/bin/libxul.so, CU 0x176ea29, DIE 0x17c7533>)
>     at /home/ato/src/gecko/ipc/glue/MessageChannel.cpp:2001
> #24 0x00007fffe310463c in mozilla::ipc::MessageChannel::RunMessage (this=0x7ffff692d140, aTask=...)
>     at /home/ato/src/gecko/ipc/glue/MessageChannel.cpp:1870
> #25 0x00007fffe3104b38 in mozilla::ipc::MessageChannel::MessageTask::Run (this=0x7ffff69a8880)
>     at /home/ato/src/gecko/ipc/glue/MessageChannel.cpp:1903
> #26 0x00007fffe27dbdfd in nsThread::ProcessNextEvent (this=0x7ffff696f700, aMayWait=false, aResult=0x7fffffffd6fe)
>     at /home/ato/src/gecko/xpcom/threads/nsThread.cpp:1369
> #27 0x00007fffe27e1b4c in NS_ProcessNextEvent (aThread=0x7ffff696f700, aMayWait=false)
>     at /home/ato/src/gecko/xpcom/threads/nsThreadUtils.cpp:472
> #28 0x00007fffe3107e6f in mozilla::ipc::MessagePump::Run (this=0x7ffff69c4dd0, aDelegate=0x7fffffffda98)
>     at /home/ato/src/gecko/ipc/glue/MessagePump.cpp:96
> #29 0x00007fffe3108b53 in mozilla::ipc::MessagePumpForChildProcess::Run (this=0x7ffff69c4dd0, aDelegate=0x7fffffffda98) at /home/ato/src/gecko/ipc/glue/MessagePump.cpp:301
> #30 0x00007fffe3042865 in MessageLoop::RunInternal (this=0x7fffffffda98) at /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:238
> #31 0x00007fffe30427e5 in MessageLoop::RunHandler (this=0x7fffffffda98) at /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:231
> #32 0x00007fffe30427bd in MessageLoop::Run (this=0x7fffffffda98) at /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:211
> #33 0x00007fffe6956d73 in nsBaseAppShell::Run (this=0x7fffdb2c5dd0) at /home/ato/src/gecko/widget/nsBaseAppShell.cpp:156
> #34 0x00007fffe8f90da7 in XRE_RunAppShell () at /home/ato/src/gecko/toolkit/xre/nsEmbedFunctions.cpp:896
> #35 0x00007fffe31089b1 in mozilla::ipc::MessagePumpForChildProcess::Run (this=0x7ffff69c4dd0, aDelegate=0x7fffffffda98) at /home/ato/src/gecko/ipc/glue/MessagePump.cpp:269
> #36 0x00007fffe3042865 in MessageLoop::RunInternal (this=0x7fffffffda98) at /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:238
> #37 0x00007fffe30427e5 in MessageLoop::RunHandler (this=0x7fffffffda98) at /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:231
> #38 0x00007fffe30427bd in MessageLoop::Run (this=0x7fffffffda98) at /home/ato/src/gecko/ipc/chromium/src/base/message_loop.cc:211
> #39 0x00007fffe8f90664 in XRE_InitChildProcess (aArgc=13, aArgv=0x7fffffffdf28, aChildData=0x7fffffffddb8) at /home/ato/src/gecko/toolkit/xre/nsEmbedFunctions.cpp:712
> #40 0x00007fffe8fa0587 in mozilla::BootstrapImpl::XRE_InitChildProcess (this=0x7ffff69bb0a0, argc=15, argv=0x7fffffffdf28, aChildData=0x7fffffffddb8)
>     at /home/ato/src/gecko/toolkit/xre/Bootstrap.cpp:65
> #41 0x000000000040644a in content_process_main (bootstrap=0x7ffff69bb0a0, argc=15, argv=0x7fffffffdf28) at /home/ato/src/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:64
> #42 0x0000000000406526 in main (argc=16, argv=0x7fffffffdf28, envp=0x7fffffffdfb0) at /home/ato/src/gecko/browser/app/nsBrowserApp.cpp:285

Comment 2

[Alex Chronopoulos [:achronop]]

It works here. I get also logs from the new backend when I play a youtube video, like the following:

[MediaPlayback #2]: E/cubeb /mozilla/firefox/media/libcubeb/cubeb-pulse-rs/src/backend/stream.rs:90: Requested buffer attributes maxlength {}, tlength {}, prebuf {}, minreq {}, fragsize {}
[MediaPlayback #2]: E/cubeb /mozilla/firefox/media/libcubeb/cubeb-pulse-rs/src/backend/stream.rs:90: Output buffer attributes maxlength 196, tlength 4194304, prebuf 28800, minreq 19208, fragsize 9600

I am on Fedora 25, would you like to do a general update and execute `./mach bootstrap` just in case.

Comment 3

[Andreas Tolfsen ❲:ato❳]

I notice that I only experience this on a relatively new installation of Firefox where the Firefox window is being X forwarded to my laptop.  If I grab the build off TaskCluster that crashes on my headless machine, I’m not able to reproduce the issue.

Could this indicate a missing library of sorts?  The machine does have /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-10.0.so.

Comment 4

[Andreas Tolfsen ❲:ato❳]

(In reply to Alex Chronopoulos [:achronop] from comment #2)
> I am on Fedora 25, would you like to do a general update and execute `./mach
> bootstrap` just in case.

I did a bootstrap, but it does not install any new packages.  This is on Debian.

Comment 5

[Andreas Tolfsen ❲:ato❳]

(In reply to Andreas Tolfsen ‹:ato› from comment #3)
> I notice that I only experience this on a relatively new installation of
> Firefox

s/Firefox/Linux/

I meant that the machine I see this doesn’t have a lot of normal “desktop” packages installed.

Comment 6

[u480271]

Andreas,

Thanks for the bug report and sorry you're running into this crash.  From the stack I think I see what's happening. It looks like creating the cubeb context failed and the mainloop is being double freed.

Comment 7

[u480271]

Attachment: Bug 1371319 - null out pulse pointers on destruction.

To avoid a double freeing of pa_context and pa_threaded_mainloop on
error, null out pointers once the object is freed.

Added assertions that pointer are null after destroy()'ing.

Review commit: https://reviewboard.mozilla.org/r/147346/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/147346/

Comment 8

[u480271]

I reproduced the crash by forcing pulse context creation to fail.  After apply the attached patch, `pulse-rust` backend fails cleanly and then falls back to using `pulse` backend.

Andreas, are you able to apply this patch to test if it fixes the crash you experience?

Comment 9

[Matthew Gregan [:kinetik]]

Comment on attachment 8875941 [details]
Bug 1371319 - null out pulse pointers on destruction.

https://reviewboard.mozilla.org/r/147346/#review151594

Comment 10

[Matthew Gregan [:kinetik]]

(In reply to Dan Glastonbury :kamidphish from comment #8)
> Andreas, are you able to apply this patch to test if it fixes the crash you
> experience?

Also can you verify whether PulseAudio is working on your machine (e.g. via using paplay to play a .wav and testing with the media.cubeb.backend set to "pulse").  It'd be useful to confirm if it's just the Rust backend that's failing for some reason, or if any of the PA backends would've failed to initialize because PA wasn't running/configured.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3db546295a0a
null out pulse pointers on destruction. r=kinetik

Comment 12

[Andreas Tolfsen ❲:ato❳]

(In reply to Matthew Gregan [:kinetik] from comment #10)

> Also can you verify whether PulseAudio is working on your machine
> (e.g. via using paplay to play a .wav and testing with the
> media.cubeb.backend set to "pulse").  It'd be useful to confirm if
> it's just the Rust backend that's failing for some reason, or if any
> of the PA backends would've failed to initialize because PA wasn't
> running/configured.

media.cubeb.backend is not a recognised preference in about:config,
but if I create a new preference with the value "pulse", the crash
still occurs.  I’m unable to spot any functional difference with this
preference set.

To get the paplay program I had to install pulseaudio-utils.  This
machine is a compilation machine without a desktop environment and
consequently does not have a pulseaudio server:

> % paplay cello.wav 
> Connection failure: Connection refused
> pa_context_connect() failed: Connection refused

Comment 13

[Andreas Tolfsen ❲:ato❳]

(In reply to Dan Glastonbury :kamidphish from comment #8)
> I reproduced the crash by forcing pulse context creation to fail.  After
> apply the attached patch, `pulse-rust` backend fails cleanly and then falls
> back to using `pulse` backend.
> 
> Andreas, are you able to apply this patch to test if it fixes the crash you
> experience?

Yes, applying this patch fixes the browser crash.  Thanks for fixing it so
quickly!

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/3db546295a0a