1771411 - crash near null in [@ mozilla::BlockingResourceBase::CheckAcquire]

Status: RESOLVED FIXED

(Core :: Audio/Video: MediaStreamGraph, defect)

Description

[Tyson Smith [:tsmith]]

Found while fuzzing m-c 20220526-f392452506ec (--enable-debug --enable-fuzzing)

This really seems to have spiked around 2022-05-24. Reducing a test case is difficult because they become less reliable as they are reduced. I will attach a Pernosco session shortly.

#0 0x7f5843f870ee in mozilla::BlockingResourceBase::CheckAcquire() /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp:213:13
#1 0x7f5843f87955 in mozilla::OffTheBooksMutex::Lock() /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp:310:3
#2 0x7f58479e57f4 in Lock /builds/worker/workspace/obj-build/dist/include/mozilla/Monitor.h:31:45
#3 0x7f58479e57f4 in BaseMonitorAutoLock /builds/worker/workspace/obj-build/dist/include/mozilla/Monitor.h:125:15
#4 0x7f58479e57f4 in mozilla::MediaTrackGraphImpl::PendingResumeOperation::Abort() /builds/worker/checkouts/gecko/dom/media/MediaTrackGraph.cpp:3805:21
#5 0x7f5847a070aa in mozilla::(anonymous namespace)::MediaTrackGraphShutDownRunnable::Run() /builds/worker/checkouts/gecko/dom/media/MediaTrackGraph.cpp:1733:10
#6 0x7f5843f91eb2 in mozilla::SchedulerGroup::Runnable::Run() /builds/worker/checkouts/gecko/xpcom/threads/SchedulerGroup.cpp:140:20
#7 0x7f5843fc15ae in mozilla::RunnableTask::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:475:16
#8 0x7f5843f9bf63 in mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:788:26
#9 0x7f5843f9ab13 in mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:620:15
#10 0x7f5843f9ad83 in mozilla::TaskController::ProcessPendingMTTask(bool) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:398:36
#11 0x7f5843fc4da9 in operator() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:127:37
#12 0x7f5843fc4da9 in mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_1>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:531:5
#13 0x7f5843fb080f in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1180:16
#14 0x7f5843faf441 in NS_ProcessNextEvent /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:465:10
#15 0x7f5843faf441 in SpinEventLoopUntil<mozilla::ProcessFailureBehavior::ReportToCaller, (lambda at /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:876:22)> /builds/worker/workspace/obj-build/dist/include/mozilla/SpinEventLoopUntil.h:176:25
#16 0x7f5843faf441 in nsThread::Shutdown() /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:875:3
#17 0x7f584789e27e in mozilla::MediaTrackGraphShutdownThreadRunnable::Run() /builds/worker/checkouts/gecko/dom/media/GraphDriver.cpp:86:14
#18 0x7f5843f91eb2 in mozilla::SchedulerGroup::Runnable::Run() /builds/worker/checkouts/gecko/xpcom/threads/SchedulerGroup.cpp:140:20
#19 0x7f5843fc15ae in mozilla::RunnableTask::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:475:16
#20 0x7f5843f9bf63 in mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:788:26
#21 0x7f5843f9ab13 in mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:620:15
#22 0x7f5843f9ad83 in mozilla::TaskController::ProcessPendingMTTask(bool) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:398:36
#23 0x7f5843fc4d36 in operator() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:124:37
#24 0x7f5843fc4d36 in mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_0>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:531:5
#25 0x7f5843fb080f in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1180:16
#26 0x7f5843fb6e0d in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:465:10
#27 0x7f5844b78386 in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:85:21
#28 0x7f5844aa08e7 in MessageLoop::RunInternal() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:380:10
#29 0x7f5844aa07f2 in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:373:3
#30 0x7f5844aa07f2 in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:355:3
#31 0x7f5848ccc478 in nsBaseAppShell::Run() /builds/worker/checkouts/gecko/widget/nsBaseAppShell.cpp:137:27
#32 0x7f584ae0d16b in XRE_RunAppShell() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:875:20
#33 0x7f5844b7927a in mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:235:9
#34 0x7f5844aa08e7 in MessageLoop::RunInternal() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:380:10
#35 0x7f5844aa07f2 in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:373:3
#36 0x7f5844aa07f2 in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:355:3
#37 0x7f584ae0c78c in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:734:34
#38 0x55876b9dfe90 in content_process_main /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:57:28
#39 0x55876b9dfe90 in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:338:18
#40 0x7f585a50e082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#41 0x55876b9b5c3c in _start (/home/worker/builds/m-c-20220526213638-fuzzing-debug/firefox-bin+0x15c3c) (BuildId: 884774cf4cb829ebd029303db18c9611a6e524a2)

Comment 1

[Tyson Smith [:tsmith]]

A Pernosco session is available here: https://pernos.co/debug/lKzOU0CiuBvxZkdxSYqEgQ/index.html

Comment 2

[Paul Adenot (:padenot)]

This is the case of a track (in particular here the track for an AudioDestinationNode) being destroyed while a resume message is in flight, it's pretty clear from the pernosco link.

The MediaTrack is unlinked from the graph in MediaTrack::DestroyImpl, and then the member mGraph is accessed in mozilla::MediaTrackGraphImpl::PendingResumeOperation::Abort, and it crashes.

I guess it spikes now because pending operations stay a lot longer in the graph, with the fix for bug 1767549 applied.

Karl, I suppose this is just a case we need to take into account with a local fix, yes?

Comment 3

[Karl Tomlinson (:karlt)]

This is a regression from adding the lock in https://hg.mozilla.org/mozilla-central/rev/669d85b972c357746e9c2d2a04a3e9b6359b753e#l1.59

The assert exists because we know that mLifecycleState is not being modified while this code runs, so there is no need for the monitor.
Debug-only locking is also undesirable because it may make debug builds behave differently from opt builds.

The assert correctly null-checks mGraph.

I suspect the best option here is to use LifecycleStateRef(), which already has run-time thread-safety assertions, and static annotations so that static analysis does not require the monitor.

Comment 4

[Karl Tomlinson (:karlt)]

No need for a test because doesn't change behavior for users and difficult to reproduce.

Comment 5

[Paul Adenot (:padenot)]

Attachment: Bug 1771411 - Use LifecycleStateRef() in an MTG lifecycle assertion. r?karlt

Comment 6

[Karl Tomlinson (:karlt)]

Debug builds only.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/65b59ed4e2c4
Use LifecycleStateRef() in an MTG lifecycle assertion. r=karlt

Comment 9

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/65b59ed4e2c4