Closed Bug 828034 Opened 11 years ago Closed 11 years ago

crash in mozilla::ipc::RPCChannel::EnteredCxxStack

Categories

(Core Graveyard :: Plug-ins, defect, P1)

Product:
Core Graveyard
Component:
Plug-ins
Version:
20 Branch
Platform:
All
Windows 7
Type:
defect

Tracking

(firefox19 unaffected, firefox20+ verified, firefox21 fixed)

Status:
VERIFIED FIXED
Milestone:
mozilla21
Tracking Flags:
Tracking Status
firefox19 --- unaffected
firefox20 + verified
firefox21 --- fixed

People

(Reporter: scoobidiver, Assigned: bugzilla)

References

Details

(Keywords: crash, regression, topcrash)

Crash Data

Attachments

(1 file, 1 obsolete file)

Proposed crash fix, rev. 2
3.83 KB, patch
benjamin
: review+
bajaj
: approval-mozilla-aurora+
Details | Diff | Splinter Review 
It first showed up in 20.0a1/20130106. The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d8ca3e1c469e&tochange=20d1a5916ef6

Signature 	mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction, IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow() More Reports Search
UUID	5e4f9196-94c3-4ac3-85a6-6ebc42130108
Date Processed	2013-01-08 19:28:47
Uptime	539
Last Crash	3.0 weeks before submission
Install Age	9.0 minutes since version was first installed.
Install Time	2013-01-08 19:19:16
Product	Firefox
Version	21.0a1
Build ID	20130108033457
Release Channel	nightly
OS	Windows NT
OS Version	6.1.7600
Build Architecture	x86
Build Architecture Info	GenuineIntel family 6 model 23 stepping 10
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x0
App Notes 	
AdapterVendorID: 0x8086, AdapterDeviceID: 0x2a42, AdapterSubsysID: 360b103c, AdapterDriverVersion: 8.15.10.1749
D3D10 Layers? D3D10 Layers- D3D9 Layers? D3D9 Layers- 
EMCheckCompatibility	True
Adapter Vendor ID	0x8086
Adapter Device ID	0x2a42
Total Virtual Memory	2147352576
Available Virtual Memory	1411579904
System Memory Use Percentage	87
Available Page File	415076352
Available Physical Memory	126259200

Frame 	Module 	Signature 	Source
0 	xul.dll 	mozilla::ipc::RPCChannel::EnteredCxxStack 	obj-firefox/dist/include/mozilla/ipc/RPCChannel.h:197
1 	xul.dll 	mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame 	obj-firefox/dist/include/mozilla/ipc/RPCChannel.h:250
2 	xul.dll 	mozilla::ipc::RPCChannel::Call 	ipc/glue/RPCChannel.cpp:136
3 	xul.dll 	mozilla::plugins::PPluginInstanceParent::CallUpdateWindow 	obj-firefox/ipc/ipdl/PPluginInstanceParent.cpp:1076
4 	xul.dll 	nsWindow::OnPaint 	widget/windows/nsWindowGfx.cpp:203
5 	xul.dll 	nsWindow::ProcessMessage 	widget/windows/nsWindow.cpp:4802
6 	xul.dll 	nsWindow::WindowProcInternal 	widget/windows/nsWindow.cpp:4407
7 	xul.dll 	CallWindowProcCrashProtected 	xpcom/base/nsCrashOnException.cpp:32
8 	xul.dll 	nsWindow::WindowProc 	widget/windows/nsWindow.cpp:4359
9 	user32.dll 	InternalCallWinProc 	
10 	user32.dll 	UserCallWinProcCheckWow 	
11 	user32.dll 	CallWindowProcAorW 	
12 	user32.dll 	CallWindowProcW 	
13 	xul.dll 	mozilla::plugins::PluginInstanceParent::PluginWindowHookProc 	dom/plugins/ipc/PluginInstanceParent.cpp:1862
14 	user32.dll 	InternalCallWinProc 	
15 	user32.dll 	UserCallWinProcCheckWow 	
16 	user32.dll 	CallWindowProcAorW 	
17 	user32.dll 	CallWindowProcW 	
18 	xul.dll 	PluginWndProcInternal 	dom/plugins/base/nsPluginNativeWindowWin.cpp:327
19 	xul.dll 	CallWindowProcCrashProtected 	xpcom/base/nsCrashOnException.cpp:32
20 	xul.dll 	PluginWndProc 	dom/plugins/base/nsPluginNativeWindowWin.cpp:356
21 	user32.dll 	InternalCallWinProc 	
22 	user32.dll 	GetRealWindowOwner 	
23 	user32.dll 	DispatchClientMessage 	
24 	user32.dll 	__fnDWORD 	
25 	ntdll.dll 	KiUserCallbackDispatcher 	
26 	ntdll.dll 	KiUserApcDispatcher 	
27 	user32.dll 	DispatchMessageW 	
28 	xul.dll 	nsAppShell::ProcessNextNativeEvent 	widget/windows/nsAppShell.cpp:328
29 	xul.dll 	nsBaseAppShell::OnProcessNextEvent 	widget/xpwidgets/nsBaseAppShell.cpp:280
...

More reports at:
https://crash-stats.mozilla.com/report/list?signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29+|+mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C+mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C+IPC%3A%3AMessage+const*%29+|+mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29
It's #9 top browser crasher in 21.0a1.
Crash Signature: [@ mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction, IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow()] → [@ mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow()] [@ …
tracking-firefox20: --- → ?
Keywords: topcrash
Hardware: x86 → All
KaiRo - would you mind grabbing URLs and correlations?
Flags: needinfo?(kairo)
Keywords: needURLs
Note that we had a crash with the same signature a few months back (Fx15/16) in bug 770805 and then Benjamin found a fix, so CCing him again here.


URLs:
2 	https://apps.facebook.com/onthefarm/?source=FBad&affiliate=toolbar&creative&redirecting_zy_session_expired=1&
2 	about:blank
1 	http://apps.facebook.com/luckygemcasino/?fb_source=bookmark_apps&ref=bookmarks&count=0&fb_bmpos=5_0
1 	http://apps.facebook.com/playhappyfarm/?fb_source=bookmark_apps&ref=bookmarks&count=37&fb_bmpos=2_37
1 	http://www.youtube.com/watch?v=lqKQyy6T63E
...and a long list of other sites that are probably using Flash, including some adult video site, apparently.

There are no correlation reports for this signature.
Flags: needinfo?(kairo)
Keywords: needURLs
See bug 829909, which almost certainly has the same root cause. Regression from bug 805591.
Assignee: nobody → aklotz
Blocks: 805591
Priority: -- → P1
I've tried to reproduce the crashes on both latest Nightly and Aurora, with intense stress testing, but without any luck.

I've also tried to lower dom.ipc.plugins.hangUITimeoutSecs so that the Plugin Hang UI triggers more easily, but still no Firefox crash.

My attempts were with multiple tabs and multiple windows, all with Flash content, both on Windows 7 and Windows 8.
Attached patch Proposed crash fix (obsolete) — — Splinter Review 
I was finally able to obtain some useful information from new correlation reports indicating that nearly half of the recent crashes occurred on 32-bit, single-core machines. I fired up an old dual-core desktop of mine, forced Windows to boot with only one core, and eventually was able to reproduce with WinDbg attached!

The debugger is showing that PluginModuleParent::CleanupFromTimeout is firing as expected. Usually when the IPC channel is closed from here, its status is already indicating an error (i.e. the I/O thread reported the channel error first). OTOH, occasionally the channel state was still showing that everything was OK (indicating that CleanupFromTimeout beat the I/O thread to the punch), so the channel was not closing with error. A crash whose stack matches this signature would follow.

It looks like the correct thing to do here is for CleanupFromTimeout() to call CloseWithError() on the channel instead of doing a regular Close().
Attachment #705626 - Flags: review?(benjamin)
Comment on attachment 705626 [details] [diff] [review]
Proposed crash fix

I don't think this can hurt.
Attachment #705626 - Flags: review?(benjamin) → review+
Sorry, the last revision broke a bunch of tests on try. We need to select which Close* function to call depending on whether the child process was terminated directly from ShouldContinueFromReplyTimeout or from a separate thread via the Plugin Hang UI.
Attachment #705626 - Attachment is obsolete: true
Attachment #706038 - Flags: review?(benjamin)
Attachment #706038 - Flags: review?(benjamin) → review+
Try in progress:
https://tbpl.mozilla.org/?tree=Try&rev=be273a2fcbe9
Keywords: checkin-needed
Whiteboard: [leave-open]
Whiteboard: [leave-open] → [leave open]
Blocks: 829909
Crash Signature: IPC::Message const*) | IPC::Message::Message(IPC::Message const&)] → IPC::Message const*) | IPC::Message::Message(IPC::Message const&)] [@ mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::Ca…
This landed on the 26th, but the Nightly from 27th still crashes, e.g. bp-22d1907c-3776-4cd6-b3f2-1bab72130127 :(
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #12)
> This landed on the 26th
Yes for the date, no for the build. It first landed in 21.0a1/20120128. See http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f18b12139151&tochange=80fed51ae074
(In reply to Scoobidiver from comment #13)
> (In reply to Robert Kaiser (:kairo@mozilla.com) from comment #12)
> > This landed on the 26th
> Yes for the date, no for the build. It first landed in 21.0a1/20120128. See
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=f18b12139151&tochange=80fed51ae074

Oh, interesting. And also, that's a relief. Haven't seen crashes with the 28 build yet, but it's still rather early on that day, so let's see.
Flags: needinfo?(kairo)
Looks like we're good here, and like bug 829909 is good as well. No crashes after the builds from the 27th.
Flags: needinfo?(kairo)
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
status-firefox21: affected → fixed
Target Milestone: --- → mozilla21
Whiteboard: [leave open]
Comment on attachment 706038 [details] [diff] [review]
Proposed crash fix, rev. 2

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 805591
User impact if declined: Intermittent crashes when Plugin Hang UI terminates a plugin
Testing completed (on m-c, etc.): Landed on m-c on Jan 28, no crashes for this signature on Nightly since
Risk to taking this patch (and alternatives if risky): None
String or UUID changes made by this patch: None
Attachment #706038 - Flags: approval-mozilla-aurora?
Attachment #706038 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
checkin-needed for Aurora, please.
Keywords: checkin-needed
Whiteboard: [needs-checkin-aurora]
I don't see any crash reports in Socorro, after 2013-02-01.

Here are the reports for the first and third signature of this bug, within last week, because I couldn't find any reports regarding the second signature.

https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A16%3A31&range_value=1&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29


https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A17%3A09&range_value=1&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29
Yes, as I said in comment #15, we're good on this one.
Status: RESOLVED → VERIFIED
There aren't any new crashes reported in Socorro, for neither one of the 3 signatures of this bug, within last month.  No new crashes after Firefox 20 beta 1, neither.

Reports are available here:

https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A16%3A31&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29



https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20IPC%3A%3AMessage%3A%3AMessage%28IPC%3A%3AMessage%20const%26amp%3B%29&reason_type=contains&date=02%2F28%2F2013%2014%3A24%3A52&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20IPC%3A%3AMessage%3A%3AMessage%28IPC%3A%3AMessage%20const%26%29


https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A17%3A09&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29
QA Contact: manuela.muntean
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: