1702530 - crash near null [@ mozilla::ProfilerParentTracker::StartTracking]

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P2)

Description

[Tyson Smith [:tsmith]]

Found while trying to reproduce another issue using m-c 20210331-32a9e6e145d6. No test case is available but a Pernosco session will be added shortly.

==17250==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000040 (pc 0x3be1697c0ab5 bp 0x7ffcc301a1b0 sp 0x7ffcc301a190 T0)
==17250==The signal is caused by a READ memory access.
==17250==Hint: address points to the zero page.
    #0 0x3be1697c0ab5 in mozilla::Maybe<mozilla::ProfileBufferGlobalController>::isNothing() const src/objdir-ff-ubsan/dist/include/mozilla/Maybe.h:444:46
    #1 0x3be16974c32d in mozilla::ProfilerParentTracker::StartTracking(mozilla::ProfilerParent*) src/tools/profiler/gecko/ProfilerParent.cpp:411:35
    #2 0x3be16974d7f8 in mozilla::ProfilerParent::Init() src/tools/profiler/gecko/ProfilerParent.cpp:564:3
    #3 0x3be16974d48d in mozilla::ProfilerParent::CreateForProcess(int) src/tools/profiler/gecko/ProfilerParent.cpp:549:10
    #4 0x3be15e5860ef in mozilla::RDDChild::Init() src/dom/media/ipc/RDDChild.cpp:66:30
    #5 0x3be15e592e98 in mozilla::RDDProcessHost::InitAfterConnect(bool) src/dom/media/ipc/RDDProcessHost.cpp:180:19
    #6 0x3be15e598fb7 in mozilla::RDDProcessHost::OnChannelConnected(int)::$_2::operator()() const src/dom/media/ipc/RDDProcessHost.cpp:136:11
    #7 0x3be15e598cee in mozilla::detail::RunnableFunction<mozilla::RDDProcessHost::OnChannelConnected(int)::$_2>::Run() src/objdir-ff-ubsan/dist/include/nsThreadUtils.h:534:5
    #8 0x3be151d8f0f9 in mozilla::RunnableTask::Run() src/xpcom/threads/TaskController.cpp:470:16
    #9 0x3be151d7273a in mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) src/xpcom/threads/TaskController.cpp:754:26
    #10 0x3be151d6f100 in mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) src/xpcom/threads/TaskController.cpp:609:15
    #11 0x3be151d6f509 in mozilla::TaskController::ProcessPendingMTTask(bool) src/xpcom/threads/TaskController.cpp:393:36
    #12 0x3be151d76d6d in mozilla::TaskController::InitializeInternal()::$_4::operator()() const src/xpcom/threads/TaskController.cpp:136:37
    #13 0x3be151d76cdd in mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_4>::Run() src/xpcom/threads/nsThreadUtils.h:534:5
    #14 0x3be151dd13c1 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1155:16
    #15 0x3be151ddc5e9 in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:548:10
    #16 0x3be1543f27cd in bool mozilla::SpinEventLoopUntil<(mozilla::ProcessFailureBehavior)1, (anonymous namespace)::ParentImpl::ShutdownBackgroundThread()::$_15>((anonymous namespace)::ParentImpl::ShutdownBackgroundThread()::$_15&&, nsIThread*) src/objdir-ff-ubsan/dist/include/mozilla/SpinEventLoopUntil.h:93:25
    #17 0x3be1543e07a9 in (anonymous namespace)::ParentImpl::ShutdownBackgroundThread() src/ipc/glue/BackgroundImpl.cpp:1296:7
    #18 0x3be1543e0197 in (anonymous namespace)::ParentImpl::ShutdownObserver::Observe(nsISupports*, char const*, char16_t const*) src/ipc/glue/BackgroundImpl.cpp:1418:3
    #19 0x3be151b1e9af in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) src/xpcom/ds/nsObserverList.cpp:70:19
    #20 0x3be151b4f92e in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) src/xpcom/ds/nsObserverService.cpp:288:19
    #21 0x3be151950376 in mozilla::AdvanceShutdownPhaseInternal(mozilla::ShutdownPhase, bool, char16_t const*, nsCOMPtr<nsISupports> const&) src/xpcom/base/AppShutdown.cpp:315:21
    #22 0x3be151950747 in mozilla::AppShutdown::AdvanceShutdownPhase(mozilla::ShutdownPhase, char16_t const*, nsCOMPtr<nsISupports> const&) src/xpcom/base/AppShutdown.cpp:334:3
    #23 0x3be151ecc51a in mozilla::ShutdownXPCOM(nsIServiceManager*) src/xpcom/build/XPCOMInit.cpp:622:5
    #24 0x3be151ecbf14 in NS_ShutdownXPCOM src/xpcom/build/XPCOMInit.cpp:565:10
    #25 0x3be16a78f57a in ScopedXPCOMStartup::~ScopedXPCOMStartup() src/toolkit/xre/nsAppRunner.cpp:1669:5
    #26 0x3be16a7b507a in mozilla::DefaultDelete<ScopedXPCOMStartup>::operator()(ScopedXPCOMStartup*) const src/objdir-ff-ubsan/dist/include/mozilla/UniquePtr.h:463:5
    #27 0x3be16a7b5008 in mozilla::UniquePtr<ScopedXPCOMStartup, mozilla::DefaultDelete<ScopedXPCOMStartup> >::reset(ScopedXPCOMStartup*) src/objdir-ff-ubsan/dist/include/mozilla/UniquePtr.h:305:7
    #28 0x3be16a7b1a96 in mozilla::UniquePtr<ScopedXPCOMStartup, mozilla::DefaultDelete<ScopedXPCOMStartup> >::operator=(std::nullptr_t) src/objdir-ff-ubsan/dist/include/mozilla/UniquePtr.h:275:5
    #29 0x3be16a7a8ee6 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:5554:16
    #30 0x3be16a7a9433 in XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:5600:21
    #31 0x3be16a7d88c6 in mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/Bootstrap.cpp:45:12
    #32 0x55ad74a9e481 in do_main(int, char**, char**) src/browser/app/nsBrowserApp.cpp:220:22
    #33 0x55ad74a9cec4 in main src/browser/app/nsBrowserApp.cpp:347:16

Comment 1

[Tyson Smith [:tsmith]]

A Pernosco session is available here: https://pernos.co/debug/9xufWVa-KJxHojdrahKCSg/index.html

Comment 2

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Thank you for the report and Pernosco link.
NI: myself, to investigate.

Comment 3

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Alastor, could you please have a look at this? (Since RDD is playing a role.)

Here's the pernosco link where I've stopped my investigation (for now) : https://pernos.co/debug/9xufWVa-KJxHojdrahKCSg/index.html#f{m[CrtB,LX0_,t[AQ,Q2I_,f{e[CrsQ,Cz0_,s{aO+FDWCAA,bAWg,uDp4H/g,oDp9N1w___
Based on this and the stack trace in comment 0, you can see that:

• #24 NS_ShutdownXPCOM -> XPCOM is shutting down.
• #17 ShutdownBackgroundThread -> Threads are getting shut down (this destroyed the Profiler IPC child before this point).
• #16 SpinEventLoopUntil -> Waiting for all actors to have disappeared.
• #6 RDDProcessHost::OnChannelConnected -> RDD connection, is that bad timing?
• #4 RDDChild::Init -> RDD is initializing the IPC Child side.
• #3 ProfilerParent::CreateForProcess -> RDDChild::Init wants to initialize the profiler IPC child.
• After that, the Profiler IPC code registers itself to be cleared on shutdown, which happens immediately, which destroys some object that the Profiler IPC code then tries to use -> UAF.

I think both the Profiler and RDD should not initialize themselves during XPCOM shutdown.
Now I would argue that RDD is probably more at fault here, since it's the one trying to initialize itself and the profiler during shutdown! (And if the Profiler hadn't crashed, do you think the RDD code would have continued without problems after this?)

What do you think? I'd be happy to split the bug, so we each take care of our side.

Comment 4

[Alastor Wu [:alwu]]

Thank for the analysis! Agree, it seems to me that in RDDProcessHost we should know if XPCOM starts shutdown or not. Then treat this as a failure if the RDD connection happens after XPCOM shutdown.

Comment 5

[Alastor Wu [:alwu]]

Attachment: RDD-debug.png

(I tried to save the link of current view in Pernosco, but every time I pasted the link of URL, the page didn't preserve the view so I had to screenshot that)

After digging into that more, I found that entire RDD process creation was done after XPCOM started shutdown, which should not be run at that situation.

Comment 6

[Alastor Wu [:alwu]]

Attachment: Bug 1702530 - do not launch RDD process if XPCOM has started shutdown.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fd4a2d6f63f4
do not launch RDD process if XPCOM has started shutdown. r=mattwoodrow

Comment 8

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/fd4a2d6f63f4