1299516 - Crash in shutdownhang | mozilla::dom::PBlobParent::DestroySubtree

Status: NEW

(Toolkit :: General, defect, P2, critical)

Description

[alex_mayorga]

This bug was filed from the Socorro interface and is 
report bp-2a288f86-70e2-4b4c-b8af-4c7562160831.
=============================================================

¡Hola!

Just crashed like this on Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0 ID:20160831030224 CSet: 506facea63169a29e04eb140663da1730052db64:

bp-2a288f86-70e2-4b4c-b8af-4c7562160831
	31/08/2016	09:25 a.m.

Found 4365 more crashes in the past week at https://crash-stats.mozilla.com/signature/?product=Firefox&signature=shutdownhang%20%7C%20mozilla%3A%3A%60anonymous%20namespace%27%27%3A%3ARunWatchdog dating back to Firefox 37.0.2 so filing a bug and guessing a component.

¡Gracias!
Alex

Crashing Thread (55)
Frame 	Module 	Signature 	Source
0 	xul.dll 	mozilla::`anonymous namespace'::RunWatchdog 	toolkit/components/terminator/nsTerminator.cpp:158
1 	nss3.dll 	PR_NativeRunThread 	nsprpub/pr/src/threads/combined/pruthr.c:397
2 	nss3.dll 	pr_root 	nsprpub/pr/src/md/windows/w95thred.c:95
3 	ucrtbase.dll 	crt_at_quick_exit 	
4 	kernel32.dll 	BaseThreadInitThunk 	
5 	ntdll.dll 	RtlUserThreadStart

Comment 1

[Marcia Knous [:marcia - needinfo? me]]

[Tracking Requested - why for this release]: This has risen to #1 to crash on 49 Beta 8. It has been present in other releases as well.

Comment 2

[Marcia Knous [:marcia - needinfo? me]]

This is also currently the #2 top browser crash on 48.0.2.

Comment 3

[Marcia Knous [:marcia - needinfo? me]]

See also Bug 1299769 where apparently shutdownhang signatures are incorrectly generated.

Comment 4

[Sylvestre Ledru [:Sylvestre]]

David, can you help with this? Thanks

Comment 5

[David Bolter [:davidb] (NeedInfo me for attention)]

Yep. Benjamin is this basically the crash we expect to see when FF takes too long to shutdown and we force a crash? Anything we can investigate?

Comment 6

[Benjamin Smedberg]

These crashes need to be investigated by what the main thread was doing (not the watchdog thread, which is just doing its job crashing on purpose).

e.g. the crash from comment 0 the main thread is firing shutdown observers, which are spinning a nested event loop (so far relatively normal, although I hate the nested-eventloop shutdown set we're using).
Then we're processing a mozilla::ipc::MessageChannel::OnNotifyMaybeChannelError() from a content process that has shut down earlier. I can't be sure whether this is "normal" shutdown or abnormal.
That triggers mozilla::dom::PContentParent::OnChannelError() which tears down the IPDL actor tree. Which shouldn't take a long time.

So either tearing down the actor tree is taking far longer than it should/caused an infinite loop, or we just happened to hit the 60-second shutdown timer at this random point in time.

In case it's not clear, these are hard to debug ;-) One thing that might make it better is to annotate when we started the shutdown timer and at various shutdown milestones after that, so that we can know whether particular phases of shutdown are taking a long time.

A went through a few others and the stacks are broken-enough that I couldn't figure out what's going on. I'd have to load them in VS to see if that has better stacks.

Comment 7

[David Bolter [:davidb] (NeedInfo me for attention)]

Jimm, do you know who might be able to investigate the cause of these shutdown issues? (I think your team has chases stuff like this before?)

Comment 8

[Liz Henry (:lizzard)]

Here's what landed between beta 7 and beta 8: https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_49_0b7_RELEASE&tochange=FIREFOX_49_0b8_RELEASE

Comment 9

[[:philipp]]

there was a ~24 hour window when all shutdown hangs were showing up under the same signature. now that bug 1299769 got fixed and all shutdownhang reports get reprocessed, the particular crash from comment #0 turns out to be [@ shutdownhang | mozilla::dom::PBlobParent::DestroySubtree] which is of quite minor volume and probably doesn't need tracking for 49 anymore.

Comment 10

[Jim Mathies [:jimm]]

I see three reports across 7 days so this isn't critical.

Kanru's team has been looking at shutdown crashes with their stability work. Marking the bug such that it'll get on their radar.

Comment 11

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to Jim Mathies [:jimm] from comment #10)
> I see three reports across 7 days so this isn't critical.
> 
> Kanru's team has been looking at shutdown crashes with their stability work.
> Marking the bug such that it'll get on their radar.

Thanks. Note this was still called "Crash in shutdownhang | mozilla::`anonymous namespace''::RunWatchdog" when I asked, which is a topcrash (although assumed to be many types of shutdown hangs)... is there investigation (via Kanru?) happening as per comment 6 somewhere else?

Do we still want a "Crash in shutdownhang | mozilla::`anonymous namespace''::RunWatchdog" (perhaps meta) bug?

Comment 12

[Liz Henry (:lizzard)]

Sounds like maybe this is from bug 1299769 and the signatures were incorrectly generated.

Comment 13

[Jim Mathies [:jimm]]

I don't see why we would spend time on this, low volume crash appears to be a shutdownhang, which the shutdown hang thread catches.

https://crash-stats.mozilla.com/signature/?signature=shutdownhang%20%7C%20mozilla%3A%3Adom%3A%3APBlobParent%3A%3ADestroySubtree&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_sort=-date&page=1#reports

13 crashes over 7 days

Comment 14

[Jim Mathies [:jimm]]

(In reply to David Bolter [:davidb] from comment #11)
> (In reply to Jim Mathies [:jimm] from comment #10)
> > I see three reports across 7 days so this isn't critical.
> > 
> > Kanru's team has been looking at shutdown crashes with their stability work.
> > Marking the bug such that it'll get on their radar.
> 
> Thanks. Note this was still called "Crash in shutdownhang |
> mozilla::`anonymous namespace''::RunWatchdog" when I asked, which is a
> topcrash (although assumed to be many types of shutdown hangs)... is there
> investigation (via Kanru?) happening as per comment 6 somewhere else?
> 
> Do we still want a "Crash in shutdownhang | mozilla::`anonymous
> namespace''::RunWatchdog" (perhaps meta) bug?

The reports should have the crashing (hung) thread as the main signature. So namespace''::RunWatchdog should show up.

Comment 15

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #14)
> (In reply to David Bolter [:davidb] from comment #11)
> > (In reply to Jim Mathies [:jimm] from comment #10)
> > > I see three reports across 7 days so this isn't critical.
> > > 
> > > Kanru's team has been looking at shutdown crashes with their stability work.
> > > Marking the bug such that it'll get on their radar.
> > 
> > Thanks. Note this was still called "Crash in shutdownhang |
> > mozilla::`anonymous namespace''::RunWatchdog" when I asked, which is a
> > topcrash (although assumed to be many types of shutdown hangs)... is there
> > investigation (via Kanru?) happening as per comment 6 somewhere else?
> > 
> > Do we still want a "Crash in shutdownhang | mozilla::`anonymous
> > namespace''::RunWatchdog" (perhaps meta) bug?
> 
> The reports should have the crashing (hung) thread as the main signature. So
> namespace''::RunWatchdog should show up.

s/should/shouldn't

Comment 16

[David Bolter [:davidb] (NeedInfo me for attention)]

OK. Setting wontfix for 49 (for "Crash in shutdownhang | mozilla::dom::PBlobParent::DestroySubtree"). Thanks.

Comment 17

[alex_mayorga]

¡Hola!

FWIW This is low volume but still a thing on Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0 ID:20161024030205 CSet: 215f9686117673a2c914ed207bc7da9bb8d741ad

bp-9b1bab88-41a4-434b-967d-508922161025
	25/10/2016	08:56 a.m.

¡Gracias!
Alex

Crashing Thread (0)
Frame 	Module 	Signature 	Source
0 	xul.dll 	mozilla::dom::PBlobParent::DestroySubtree(mozilla::ipc::IProtocolManager<mozilla::ipc::IProtocol>::ActorDestroyReason) 	obj-firefox/ipc/ipdl/PBlobParent.cpp:603
1 	xul.dll 	mozilla::dom::PContentParent::DestroySubtree(mozilla::ipc::IProtocolManager<mozilla::ipc::IProtocol>::ActorDestroyReason) 	obj-firefox/ipc/ipdl/PContentParent.cpp:9787
2 	xul.dll 	mozilla::dom::PContentParent::OnChannelError() 	obj-firefox/ipc/ipdl/PContentParent.cpp:9694
3 	xul.dll 	mozilla::dom::ContentParent::OnChannelError() 	dom/ipc/ContentParent.cpp:1634
4 	xul.dll 	mozilla::ipc::MessageChannel::NotifyMaybeChannelError() 	ipc/glue/MessageChannel.cpp:2104
5 	xul.dll 	mozilla::ipc::MessageChannel::OnNotifyMaybeChannelError() 	ipc/glue/MessageChannel.cpp:2133
6 	xul.dll 	mozilla::detail::RunnableMethodImpl<void ( mozilla::ipc::MessageChannel::*)(void), 0, 1>::Run() 	obj-firefox/dist/include/nsThreadUtils.h:764
7 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1082
8 	xul.dll 	nsThread::Shutdown() 	xpcom/threads/nsThread.cpp:941
9 	xul.dll 	mozilla::detail::RunnableMethodImpl<void ( mozilla::storage::Connection::*)(nsIThread*), 1, 0, nsCOMPtr<nsIThread> >::Run() 	obj-firefox/dist/include/nsThreadUtils.h:764
10 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1082
11 	xul.dll 	nsThread::Shutdown() 	xpcom/threads/nsThread.cpp:941
12 	xul.dll 	mozilla::detail::RunnableMethodImpl<void ( mozilla::storage::Connection::*)(nsIThread*), 1, 0, nsCOMPtr<nsIThread> >::Run() 	obj-firefox/dist/include/nsThreadUtils.h:764
13 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1082
14 	xul.dll 	nsThread::Shutdown() 	xpcom/threads/nsThread.cpp:941
15 	xul.dll 	mozilla::DataStorage::Observe(nsISupports*, char const*, char16_t const*) 	security/manager/ssl/DataStorage.cpp:886
16 	xul.dll 	nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) 	xpcom/ds/nsObserverList.cpp:112
17 	xul.dll 	nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) 	xpcom/ds/nsObserverService.cpp:281
18 	xul.dll 	nsXREDirProvider::DoShutdown() 	toolkit/xre/nsXREDirProvider.cpp:1250
19 	xul.dll 	ScopedXPCOMStartup::~ScopedXPCOMStartup() 	toolkit/xre/nsAppRunner.cpp:1396
20 	xul.dll 	mozilla::UniquePtr<ScopedXPCOMStartup, mozilla::DefaultDelete<ScopedXPCOMStartup> >::reset(ScopedXPCOMStartup*) 	obj-firefox/dist/include/mozilla/UniquePtr.h:345
21 	xul.dll 	XREMain::XRE_main(int, char** const, nsXREAppData const*) 	toolkit/xre/nsAppRunner.cpp:4611
22 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:4674
23 	firefox.exe 	do_main 	browser/app/nsBrowserApp.cpp:282
24 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:115
25 	firefox.exe 	__scrt_common_main_seh 	f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:253
26 	kernel32.dll 	BaseThreadInitThunk 	
27 	ntdll.dll 	RtlUserThreadStart

Comment 18

[Nathan Froyd [:froydnj] [away until 28 October]]

I hit this today, and part of the cause may be that Test Pilot's vertical tabs extension triggers some leaky behavior of blobs, which then causes us to accumulate lots of PBlob* things, which then causes the shutdown hang...

Comment 19

[alex_mayorga]

¡Hola!

Just crashed like this on Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0 ID:20161201030205 CSet: cd4cdcc9ad6c45dad8b8d8c0d40e459db2bca8a1

bp-7c54c745-6be7-4452-8a4c-69c902161201
	12/1/16	9:35 AM

As Nathan stated, this install does have Test Pilot's Tab Center and Activity Stream enabled.

¡Gracias!
Alex

Crashing Thread (0)
Frame 	Module 	Signature 	Source
0 	xul.dll 	mozilla::dom::PBlobParent::DestroySubtree(mozilla::ipc::IProtocol::ActorDestroyReason) 	obj-firefox/ipc/ipdl/PBlobParent.cpp:463
1 	xul.dll 	mozilla::dom::PContentParent::DestroySubtree(mozilla::ipc::IProtocol::ActorDestroyReason) 	obj-firefox/ipc/ipdl/PContentParent.cpp:8953
2 	xul.dll 	mozilla::dom::PContentParent::OnChannelError() 	obj-firefox/ipc/ipdl/PContentParent.cpp:8930
3 	xul.dll 	mozilla::dom::ContentParent::OnChannelError() 	dom/ipc/ContentParent.cpp:1317
4 	xul.dll 	mozilla::ipc::MessageChannel::NotifyMaybeChannelError() 	ipc/glue/MessageChannel.cpp:2193
5 	xul.dll 	mozilla::ipc::MessageChannel::OnNotifyMaybeChannelError() 	ipc/glue/MessageChannel.cpp:2222
6 	xul.dll 	mozilla::detail::RunnableMethodImpl<void ( mozilla::ipc::MessageChannel::*)(void), 0, 1>::Run() 	obj-firefox/dist/include/nsThreadUtils.h:814
7 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1213
8 	xul.dll 	nsThread::Shutdown() 	xpcom/threads/nsThread.cpp:983
9 	xul.dll 	mozilla::detail::RunnableMethodImpl<void ( mozilla::storage::Connection::*)(nsIThread*), 1, 0, nsCOMPtr<nsIThread> >::Run() 	obj-firefox/dist/include/nsThreadUtils.h:814
10 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1213
11 	xul.dll 	nsThread::Shutdown() 	xpcom/threads/nsThread.cpp:983
12 	xul.dll 	mozilla::detail::RunnableMethodImpl<void ( mozilla::storage::Connection::*)(nsIThread*), 1, 0, nsCOMPtr<nsIThread> >::Run() 	obj-firefox/dist/include/nsThreadUtils.h:814
13 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1213
14 	xul.dll 	nsThread::Shutdown() 	xpcom/threads/nsThread.cpp:983
15 	xul.dll 	mozilla::DataStorage::Observe(nsISupports*, char const*, char16_t const*) 	security/manager/ssl/DataStorage.cpp:886
16 	xul.dll 	nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) 	xpcom/ds/nsObserverList.cpp:112
17 	xul.dll 	nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) 	xpcom/ds/nsObserverService.cpp:281
18 	xul.dll 	nsXREDirProvider::DoShutdown() 	toolkit/xre/nsXREDirProvider.cpp:1250
19 	xul.dll 	ScopedXPCOMStartup::~ScopedXPCOMStartup() 	toolkit/xre/nsAppRunner.cpp:1409
20 	xul.dll 	mozilla::UniquePtr<ScopedXPCOMStartup, mozilla::DefaultDelete<ScopedXPCOMStartup> >::reset(ScopedXPCOMStartup*) 	obj-firefox/dist/include/mozilla/UniquePtr.h:345
21 	xul.dll 	XREMain::XRE_main(int, char** const, nsXREAppData const*) 	toolkit/xre/nsAppRunner.cpp:4641
22 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:4704
23 	firefox.exe 	do_main 	browser/app/nsBrowserApp.cpp:328
24 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:115
25 	firefox.exe 	__scrt_common_main_seh 	f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:253
Ø 26 	kernel32.dll 	kernel32.dll@0x159cc 	
Ø 27 	ntdll.dll 	ntdll.dll@0x2a2e0

Comment 20

[Julien Cristau [:jcristau]]

Mass wontfix for bugs affecting firefox 52.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/abd22366d81e
Allow MediaEngines to Start() and Stop() without affecting tracks. r=jib, r=padenot