852164 - crash in mozilla::plugins::PluginModuleParent::TerminateChildProcess @ PL_DHashTableOperate

Status: VERIFIED FIXED

Description

[Scoobidiver (away)]

It's #72 top browser crasher in 20.0b5 and occurs mainly on Windows XP.

Signature 	PL_DHashTableOperate | nsTHashtable<nsBaseHashtableET<nsCStringHashKey, nsCString> >::PutEntry(nsACString_internal const&) More Reports Search
UUID	53e07183-695e-405e-9190-299632130318
Date Processed	2013-03-18 14:15:46
Uptime	1923
Last Crash	2.1 weeks before submission
Install Age	36.1 minutes since version was first installed.
Install Time	2013-03-18 13:38:59
Product	Firefox
Version	20.0
Build ID	20130313170052
Release Channel	beta
OS	Windows NT
OS Version	6.1.7601 Service Pack 1
Build Architecture	x86
Build Architecture Info	GenuineIntel family 6 model 23 stepping 10
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0xd
App Notes 	
AdapterVendorID: 0x8086, AdapterDeviceID: 0x2a42, AdapterSubsysID: 17831854, AdapterDriverVersion: 8.15.10.2555
D2D? D2D+ DWrite? DWrite+ D3D10 Layers? D3D10 Layers+ 
Processor Notes 	sp-processor08.phx1.mozilla.com_4808:2008
EMCheckCompatibility	True
Adapter Vendor ID	0x8086
Adapter Device ID	0x2a42
Total Virtual Memory	2147352576
Available Virtual Memory	1108848640
System Memory Use Percentage	64
Available Page File	3495723008
Available Physical Memory	1126715392

Frame 	Module 	Signature 	Source
0 	xul.dll 	PL_DHashTableOperate 	obj-firefox/xpcom/build/pldhash.cpp:578
1 	xul.dll 	nsTHashtable<nsBaseHashtableET<nsCStringHashKey,nsCString> >::PutEntry 	obj-firefox/dist/include/nsTHashtable.h:170
2 	xul.dll 	mozilla::plugins::PluginModuleParent::TerminateChildProcess 	dom/plugins/ipc/PluginModuleParent.cpp:491
3 	xul.dll 	mozilla::plugins::PluginHangUIParent::RecvUserResponse 	dom/plugins/ipc/PluginHangUIParent.cpp:296
4 	xul.dll 	mozilla::plugins::PluginHangUIParent::OnMiniShmEvent 	dom/plugins/ipc/PluginHangUIParent.cpp:351
5 	xul.dll 	mozilla::plugins::MiniShmBase::OnEvent 	dom/plugins/ipc/hangui/MiniShmBase.h:232
6 	xul.dll 	mozilla::plugins::MiniShmParent::OnEvent 	dom/plugins/ipc/MiniShmParent.cpp:199
7 	xul.dll 	mozilla::plugins::MiniShmBase::SOnEvent 	dom/plugins/ipc/hangui/MiniShmBase.h:299
8 	ntdll.dll 	RtlpTpWaitCallback 	
9 	ntdll.dll 	TppWaitpExecuteCallback 	
10 	ntdll.dll 	TppCallbackCheckThreadBeforeCallback 	
11 	kernel32.dll 	BaseThreadInitThunk 	
12 	ntdll.dll 	__RtlUserThreadStart 	
13 	ntdll.dll 	_RtlUserThreadStart

More reports at:
https://crash-stats.mozilla.com/report/list?signature=PL_DHashTableOperate+|+nsTHashtable%3CnsBaseHashtableET%3CnsCStringHashKey%2C+nsCString%3E+%3E%3A%3APutEntry%28nsACString_internal+const%26%29
https://crash-stats.mozilla.com/report/list?signature=PL_DHashTableOperate+|+nsTHashtable%3CnsBaseHashtableET%3CnsISupportsHashKey%2C+nsCString%3E+%3E%3A%3APutEntry%28nsISupports*%29
https://crash-stats.mozilla.com/report/list?signature=PL_DHashTableOperate+|+nsTHashtable%3CnsBaseHashtableET%3CnsPtrHashKey%3CnsPIDOMWindow%3E%2C+nsAutoPtr%3CnsTArray%3Cmozilla%3A%3Adom%3A%3Aworkers%3A%3AWorkerPrivate*%3E+%3E+%3E+%3E%3A%3APutEntry%28nsPIDOMWindow*%29

Comment 1

[Benjamin Smedberg]

aklotz, there are some unexpected stacks here; PluginHangUIParent::RecvUserResponse appears to be called even though the main thread is not blocked on a plugin hang? Perhaps the PluginHangUIParent isn't disabling itself completely when the plugin hang recovers?

Comment 2

[Aaron Klotz [:aklotz]]

Attachment: Potential fix

Adds a check to ensure that a cancellation isn't already pending when processing a user response.

Comment 3

[Aaron Klotz [:aklotz]]

Let's see if this helps.
https://hg.mozilla.org/integration/mozilla-inbound/rev/1cad363e90ae

Comment 4

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/1cad363e90ae

Comment 5

[Aaron Klotz [:aklotz]]

Comment on attachment 726573 [details] [diff] [review]
Potential fix

[Approval Request Comment]
Bug caused by (feature/regressing bug #): New feature, Plugin Hang UI (bug 805591)
User impact if declined: Intermittent crashes, particularly on slower machines
Testing completed (on m-c, etc.): Landed on m-c, also tested manually. Not 100% sure that this completely eliminates the crash but this is a reasonable preventative measure that won't harm anything.
Risk to taking this patch (and alternatives if risky): Low, adds an additional sanity check
String or UUID changes made by this patch: None

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 726573 [details] [diff] [review]
Potential fix

We can take this on Aurora and analyze the data over the coming weeks to see if this does help with volume but it's too late to take a speculative fix on beta and this is not a topcrasher so I think we're OK to wait and let this bake.

Comment 7

[Aaron Klotz [:aklotz]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/939d9a298677

Comment 8

[Scoobidiver (away)]

(In reply to Aaron Klotz [:aklotz] from comment #7)
> https://hg.mozilla.org/releases/mozilla-aurora/rev/939d9a298677
There are still crashes in 21.0 Beta, 22.0a2 and 23.0a1.
See https://crash-stats.mozilla.com/report/list?signature=PL_DHashTableOperate+|+nsTHashtable%3CnsBaseHashtableET%3CnsURIHashKey%2C+nsRefPtr%3CnsXULPrototypeDocument%3E+%3E+%3E%3A%3APutEntry%28nsIURI*%29 and https://crash-stats.mozilla.com/report/list?signature=PL_DHashTableOperate+|+nsTHashtable%3CUnassociatedIconHashKey%3E%3A%3APutEntry%28nsIURI*%29

Comment 9

[Aaron Klotz [:aklotz]]

Attachment: Mutual exclusion fix

This crash is happening when the PCrashReporter sub-actor is torn down at the same time that the Plugin Hang UI calls TerminateChildProcess from another thread.

I think what we need here is to ensure that there is mutual exclusion between destruction of the PluginModuleParent's sub-actors and PluginModuleParent::TerminateChildProcess.

Comment 10

[Benjamin Smedberg]

Comment on attachment 819172 [details] [diff] [review]
Mutual exclusion fix

I'm confused. What is the shared state that this lock is protecting, and why isn't the normal channel locking scheme insufficient?

Comment 11

[Aaron Klotz [:aklotz]]

The shared state is the PCrashReporterParent object. If the channel ends up closing (and thus destroying the PCrashReporterParent sub-actor) concurrently with the crash report annotations being done by the Plugin Hang UI on another thread, we end up with this crash.

Unless I missed something, it doesn't look to me like the channel lock is held when OnChannelClose and OnChannelError are invoked.

Comment 12

[Benjamin Smedberg]

Comment on attachment 819172 [details] [diff] [review]
Mutual exclusion fix

I'm worried about this. The locking surface is pretty huge with OnChannelClose/OnChannelError. Can we instead save CrashReporterParent* as a member (so that we don't have to lock around the IPDL data structures) and lock only in ActorDestroy?

In either case, I'd like a better comment explaining that this protects both CrashReporter() method itself as well as adding annotations, and why this is safe (because plugin child processes never send annotations during normal operation, presumably).

Comment 13

[Aaron Klotz [:aklotz]]

Attachment: Mutual exclusion fix rev 2

This version saves the CrashReporterParent* as a member and only needs to lock in DeallocPCrashReporterParent, instead of OnChannelClose and OnChannelError.

Comment 14

[Aaron Klotz [:aklotz]]

Try: https://tbpl.mozilla.org/?tree=Try&rev=c18e5c591b1e

https://hg.mozilla.org/integration/mozilla-inbound/rev/5f07f8e85831

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5f07f8e85831

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/77ac974d69d6

Comment 17

[Anthony Hughes (:ashughes) [QA]]

Marking verified fixed based on crash-stats.