Open Bug 1660158 Opened 4 years ago Updated 1 month ago

Assertion failure: false (AudioCallbackDriver took too long to shut down and we let shutdown continue - freezing and leaking), at /builds/worker/checkouts/gecko/dom/media/MediaTrackGraph.cpp:1611

Categories

(Core :: Web Audio, defect)

Product:
Core
Component:
Web Audio
Type:
defect

Tracking

()

Tracking Flags:
Tracking Status
firefox81 --- affected

People

(Reporter: jkratzer, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: assertion, testcase, Whiteboard: [bugmon:confirmed])

Attachments

(1 file)

testcase.zip
784 bytes, application/zip
Details
Attached file testcase.zip —

Testcase found while fuzzing mozilla-central rev 157db696462d (built with --enable-debug).

Assertion failure: false (AudioCallbackDriver took too long to shut down and we let shutdown continue - freezing and leaking), at /builds/worker/checkouts/gecko/dom/media/MediaTrackGraph.cpp:1611

    #0 0x7fdc8c81b73b in AnnotateMozCrashReason /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:42:19
    #1 0x7fdc8c81b73b in mozilla::(anonymous namespace)::MediaTrackGraphShutDownRunnable::Run() /builds/worker/checkouts/gecko/dom/media/MediaTrackGraph.cpp:1608:7
    #2 0x7fdc88d0869c in mozilla::AutoTaskDispatcher::TaskGroupRunnable::Run() /builds/worker/workspace/obj-build/dist/include/mozilla/TaskDispatcher.h:228:35
    #3 0x7fdc88d072fc in mozilla::XPCOMThreadWrapper::Runner::Run() /builds/worker/checkouts/gecko/xpcom/threads/AbstractThread.cpp:194:25
    #4 0x7fdc88d06dd4 in mozilla::RunnableTask::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:242:16
    #5 0x7fdc88d04b9d in mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:512:26
    #6 0x7fdc88d03b64 in mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:371:15
    #7 0x7fdc88d03d17 in mozilla::TaskController::ProcessPendingMTTask(bool) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:168:36
    #8 0x7fdc88d0b889 in operator() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:86:37
    #9 0x7fdc88d0b889 in mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_5>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:577:5
    #10 0x7fdc88d1ec68 in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1242:14
    #11 0x7fdc88d1d7fd in NS_ProcessNextEvent /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:513:10
    #12 0x7fdc88d1d7fd in SpinEventLoopUntil<mozilla::ProcessFailureBehavior::ReportToCaller, (lambda at /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:906:22)> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:362:25
    #13 0x7fdc88d1d7fd in nsThread::Shutdown() /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:906:3
    #14 0x7fdc8c81b158 in mozilla::(anonymous namespace)::MediaTrackGraphShutDownRunnable::Run() /builds/worker/checkouts/gecko/dom/media/MediaTrackGraph.cpp:1588:50
    #15 0x7fdc88d0869c in mozilla::AutoTaskDispatcher::TaskGroupRunnable::Run() /builds/worker/workspace/obj-build/dist/include/mozilla/TaskDispatcher.h:228:35
    #16 0x7fdc88d072fc in mozilla::XPCOMThreadWrapper::Runner::Run() /builds/worker/checkouts/gecko/xpcom/threads/AbstractThread.cpp:194:25
    #17 0x7fdc88d06dd4 in mozilla::RunnableTask::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:242:16
    #18 0x7fdc88d04b9d in mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:512:26
    #19 0x7fdc88d03b64 in mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:371:15
    #20 0x7fdc88d03d17 in mozilla::TaskController::ProcessPendingMTTask(bool) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:168:36
    #21 0x7fdc88d0b816 in operator() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:83:37
    #22 0x7fdc88d0b816 in mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_4>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:577:5
    #23 0x7fdc88d1ec68 in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1242:14
    #24 0x7fdc88d2463a in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:513:10
    #25 0x7fdc896491df in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:87:21
    #26 0x7fdc895ba033 in MessageLoop::RunInternal() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:334:10
    #27 0x7fdc895b9f4d in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:327:3
    #28 0x7fdc895b9f4d in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:309:3
    #29 0x7fdc8d4cd758 in nsBaseAppShell::Run() /builds/worker/checkouts/gecko/widget/nsBaseAppShell.cpp:137:27
    #30 0x7fdc8ecf51c3 in XRE_RunAppShell() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:913:20
    #31 0x7fdc89649fa7 in mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:237:9
    #32 0x7fdc895ba033 in MessageLoop::RunInternal() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:334:10
    #33 0x7fdc895b9f4d in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:327:3
    #34 0x7fdc895b9f4d in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:309:3
    #35 0x7fdc8ecf4d62 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:744:34
    #36 0x55c82570009f in content_process_main /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
    #37 0x55c82570009f in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:303:18
    #38 0x7fdca44f7b96 in __libc_start_main /build/glibc-2ORdQG/glibc-2.27/csu/../csu/libc-start.c:310
Flags: in-testsuite?
Keywords: bugmon
Whiteboard: [bugmon:confirm] → [bugmon:confirmed]
Bugmon Analysis: Unable to reproduce bug using the following builds: > mozilla-central 20200819212829-b0888d07df69 > None Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.
Component: Audio/Video → Audio/Video: MediaStreamGraph

Extreme test case but what happens is that for every channel for every output on the audio worklet we allocate a buffer that holds an audio block. So on the build above (non-opt, but still repros with opt), under rr, with the testcase (55588 outputs, channelcount varying between 1 and 32 along a sine, roughly 900k allocations) PrepareBufferArrays takes 200 seconds.

Karl, can any mitigation be done here? Allocating on the audio thread doesn't seem great but this being AudioWorklet I'm not sure of the constraints we have.

Flags: needinfo?(karlt)
Component: Audio/Video: MediaStreamGraph → Web Audio

I guess there are mitigations that can be put in place, but the same kind of problem exists at multiple levels, so it's not easy to manage, and I'm not convinced it would be worthwhile to do so.

We limit the number of outputs to 65535. I guess that's larger than necessary, but Chrome doesn't have a limit so I hesitate to set one. A limit on the output count moves the problem up one level because there's no limit on the number of AudioWorkletNodes that can be created.

We could check for a pending interrupt more frequently, but there's no limit on the number of native nodes either. At least native nodes have a limit of 32 outputs.

I guess we could check an atomic for graph shutdown between the processing of each track.

I wonder about changing this assertion to a warning. Given it is expecting "freezing", maybe an early exit is better than a hang, but I assume we have another hang watchdog, which may already be in the fuzzer's ignore-list.

Flags: needinfo?(karlt)
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: