crash in mozilla::ipc::RPCChannel::EnteredCxxStack

VERIFIED FIXED in Firefox 20

Status

()

Core
Plug-ins
P1
critical
VERIFIED FIXED
4 years ago
4 years ago

People

(Reporter: Scoobidiver (away), Assigned: aklotz)

Tracking

({crash, regression, topcrash})

20 Branch
mozilla21
All
Windows 7
crash, regression, topcrash
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox19 unaffected, firefox20+ verified, firefox21 fixed)

Details

(crash signature)

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

4 years ago
It first showed up in 20.0a1/20130106. The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d8ca3e1c469e&tochange=20d1a5916ef6

Signature 	mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction, IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow() More Reports Search
UUID	5e4f9196-94c3-4ac3-85a6-6ebc42130108
Date Processed	2013-01-08 19:28:47
Uptime	539
Last Crash	3.0 weeks before submission
Install Age	9.0 minutes since version was first installed.
Install Time	2013-01-08 19:19:16
Product	Firefox
Version	21.0a1
Build ID	20130108033457
Release Channel	nightly
OS	Windows NT
OS Version	6.1.7600
Build Architecture	x86
Build Architecture Info	GenuineIntel family 6 model 23 stepping 10
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x0
App Notes 	
AdapterVendorID: 0x8086, AdapterDeviceID: 0x2a42, AdapterSubsysID: 360b103c, AdapterDriverVersion: 8.15.10.1749
D3D10 Layers? D3D10 Layers- D3D9 Layers? D3D9 Layers- 
EMCheckCompatibility	True
Adapter Vendor ID	0x8086
Adapter Device ID	0x2a42
Total Virtual Memory	2147352576
Available Virtual Memory	1411579904
System Memory Use Percentage	87
Available Page File	415076352
Available Physical Memory	126259200

Frame 	Module 	Signature 	Source
0 	xul.dll 	mozilla::ipc::RPCChannel::EnteredCxxStack 	obj-firefox/dist/include/mozilla/ipc/RPCChannel.h:197
1 	xul.dll 	mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame 	obj-firefox/dist/include/mozilla/ipc/RPCChannel.h:250
2 	xul.dll 	mozilla::ipc::RPCChannel::Call 	ipc/glue/RPCChannel.cpp:136
3 	xul.dll 	mozilla::plugins::PPluginInstanceParent::CallUpdateWindow 	obj-firefox/ipc/ipdl/PPluginInstanceParent.cpp:1076
4 	xul.dll 	nsWindow::OnPaint 	widget/windows/nsWindowGfx.cpp:203
5 	xul.dll 	nsWindow::ProcessMessage 	widget/windows/nsWindow.cpp:4802
6 	xul.dll 	nsWindow::WindowProcInternal 	widget/windows/nsWindow.cpp:4407
7 	xul.dll 	CallWindowProcCrashProtected 	xpcom/base/nsCrashOnException.cpp:32
8 	xul.dll 	nsWindow::WindowProc 	widget/windows/nsWindow.cpp:4359
9 	user32.dll 	InternalCallWinProc 	
10 	user32.dll 	UserCallWinProcCheckWow 	
11 	user32.dll 	CallWindowProcAorW 	
12 	user32.dll 	CallWindowProcW 	
13 	xul.dll 	mozilla::plugins::PluginInstanceParent::PluginWindowHookProc 	dom/plugins/ipc/PluginInstanceParent.cpp:1862
14 	user32.dll 	InternalCallWinProc 	
15 	user32.dll 	UserCallWinProcCheckWow 	
16 	user32.dll 	CallWindowProcAorW 	
17 	user32.dll 	CallWindowProcW 	
18 	xul.dll 	PluginWndProcInternal 	dom/plugins/base/nsPluginNativeWindowWin.cpp:327
19 	xul.dll 	CallWindowProcCrashProtected 	xpcom/base/nsCrashOnException.cpp:32
20 	xul.dll 	PluginWndProc 	dom/plugins/base/nsPluginNativeWindowWin.cpp:356
21 	user32.dll 	InternalCallWinProc 	
22 	user32.dll 	GetRealWindowOwner 	
23 	user32.dll 	DispatchClientMessage 	
24 	user32.dll 	__fnDWORD 	
25 	ntdll.dll 	KiUserCallbackDispatcher 	
26 	ntdll.dll 	KiUserApcDispatcher 	
27 	user32.dll 	DispatchMessageW 	
28 	xul.dll 	nsAppShell::ProcessNextNativeEvent 	widget/windows/nsAppShell.cpp:328
29 	xul.dll 	nsBaseAppShell::OnProcessNextEvent 	widget/xpwidgets/nsBaseAppShell.cpp:280
...

More reports at:
https://crash-stats.mozilla.com/report/list?signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29+|+mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C+mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C+IPC%3A%3AMessage+const*%29+|+mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29
(Reporter)

Comment 1

4 years ago
It's #9 top browser crasher in 21.0a1.
Crash Signature: [@ mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction, IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow()] → [@ mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow(…
tracking-firefox20: --- → ?
Keywords: topcrash
Hardware: x86 → All

Comment 2

4 years ago
KaiRo - would you mind grabbing URLs and correlations?
Flags: needinfo?(kairo)
Keywords: needURLs

Comment 3

4 years ago
Note that we had a crash with the same signature a few months back (Fx15/16) in bug 770805 and then Benjamin found a fix, so CCing him again here.


URLs:
2 	https://apps.facebook.com/onthefarm/?source=FBad&affiliate=toolbar&creative&redirecting_zy_session_expired=1&
2 	about:blank
1 	http://apps.facebook.com/luckygemcasino/?fb_source=bookmark_apps&ref=bookmarks&count=0&fb_bmpos=5_0
1 	http://apps.facebook.com/playhappyfarm/?fb_source=bookmark_apps&ref=bookmarks&count=37&fb_bmpos=2_37
1 	http://www.youtube.com/watch?v=lqKQyy6T63E
...and a long list of other sites that are probably using Flash, including some adult video site, apparently.

There are no correlation reports for this signature.
Flags: needinfo?(kairo)
Keywords: needURLs
See bug 829909, which almost certainly has the same root cause. Regression from bug 805591.
Assignee: nobody → aklotz
Blocks: 805591
Priority: -- → P1

Updated

4 years ago
status-firefox19: --- → unaffected
tracking-firefox20: ? → +
I've tried to reproduce the crashes on both latest Nightly and Aurora, with intense stress testing, but without any luck.

I've also tried to lower dom.ipc.plugins.hangUITimeoutSecs so that the Plugin Hang UI triggers more easily, but still no Firefox crash.

My attempts were with multiple tabs and multiple windows, all with Flash content, both on Windows 7 and Windows 8.
Created attachment 705626 [details] [diff] [review]
Proposed crash fix

I was finally able to obtain some useful information from new correlation reports indicating that nearly half of the recent crashes occurred on 32-bit, single-core machines. I fired up an old dual-core desktop of mine, forced Windows to boot with only one core, and eventually was able to reproduce with WinDbg attached!

The debugger is showing that PluginModuleParent::CleanupFromTimeout is firing as expected. Usually when the IPC channel is closed from here, its status is already indicating an error (i.e. the I/O thread reported the channel error first). OTOH, occasionally the channel state was still showing that everything was OK (indicating that CleanupFromTimeout beat the I/O thread to the punch), so the channel was not closing with error. A crash whose stack matches this signature would follow.

It looks like the correct thing to do here is for CleanupFromTimeout() to call CloseWithError() on the channel instead of doing a regular Close().
Attachment #705626 - Flags: review?(benjamin)
Comment on attachment 705626 [details] [diff] [review]
Proposed crash fix

I don't think this can hurt.
Attachment #705626 - Flags: review?(benjamin) → review+
Created attachment 706038 [details] [diff] [review]
Proposed crash fix, rev. 2

Sorry, the last revision broke a bunch of tests on try. We need to select which Close* function to call depending on whether the child process was terminated directly from ShouldContinueFromReplyTimeout or from a separate thread via the Plugin Hang UI.
Attachment #705626 - Attachment is obsolete: true
Attachment #706038 - Flags: review?(benjamin)
Attachment #706038 - Flags: review?(benjamin) → review+
Try in progress:
https://tbpl.mozilla.org/?tree=Try&rev=be273a2fcbe9
Keywords: checkin-needed
Whiteboard: [leave-open]
Whiteboard: [leave-open] → [leave open]
https://hg.mozilla.org/integration/mozilla-inbound/rev/50a7e6e2f6a1
Keywords: checkin-needed
(Reporter)

Updated

4 years ago
Blocks: 829909
(Reporter)

Updated

4 years ago
Crash Signature: [@ mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow(… → [@ mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow(…
https://hg.mozilla.org/mozilla-central/rev/50a7e6e2f6a1

Comment 12

4 years ago
This landed on the 26th, but the Nightly from 27th still crashes, e.g. bp-22d1907c-3776-4cd6-b3f2-1bab72130127 :(
(Reporter)

Comment 13

4 years ago
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #12)
> This landed on the 26th
Yes for the date, no for the build. It first landed in 21.0a1/20120128. See http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f18b12139151&tochange=80fed51ae074

Comment 14

4 years ago
(In reply to Scoobidiver from comment #13)
> (In reply to Robert Kaiser (:kairo@mozilla.com) from comment #12)
> > This landed on the 26th
> Yes for the date, no for the build. It first landed in 21.0a1/20120128. See
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=f18b12139151&tochange=80fed51ae074

Oh, interesting. And also, that's a relief. Haven't seen crashes with the 28 build yet, but it's still rather early on that day, so let's see.
Flags: needinfo?(kairo)

Comment 15

4 years ago
Looks like we're good here, and like bug 829909 is good as well. No crashes after the builds from the 27th.
Flags: needinfo?(kairo)
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Reporter)

Updated

4 years ago
status-firefox21: affected → fixed
Target Milestone: --- → mozilla21
(Reporter)

Updated

4 years ago
Whiteboard: [leave open]
Comment on attachment 706038 [details] [diff] [review]
Proposed crash fix, rev. 2

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 805591
User impact if declined: Intermittent crashes when Plugin Hang UI terminates a plugin
Testing completed (on m-c, etc.): Landed on m-c on Jan 28, no crashes for this signature on Nightly since
Risk to taking this patch (and alternatives if risky): None
String or UUID changes made by this patch: None
Attachment #706038 - Flags: approval-mozilla-aurora?

Updated

4 years ago
Attachment #706038 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
checkin-needed for Aurora, please.
Keywords: checkin-needed
Whiteboard: [needs-checkin-aurora]
https://hg.mozilla.org/releases/mozilla-aurora/rev/b7e0f49442fc
status-firefox20: affected → fixed
Keywords: checkin-needed
Whiteboard: [needs-checkin-aurora]
I don't see any crash reports in Socorro, after 2013-02-01.

Here are the reports for the first and third signature of this bug, within last week, because I couldn't find any reports regarding the second signature.

https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A16%3A31&range_value=1&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29


https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A17%3A09&range_value=1&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29

Comment 20

4 years ago
Yes, as I said in comment #15, we're good on this one.
Status: RESOLVED → VERIFIED
There aren't any new crashes reported in Socorro, for neither one of the 3 signatures of this bug, within last month.  No new crashes after Firefox 20 beta 1, neither.

Reports are available here:

https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A16%3A31&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29



https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20IPC%3A%3AMessage%3A%3AMessage%28IPC%3A%3AMessage%20const%26amp%3B%29&reason_type=contains&date=02%2F28%2F2013%2014%3A24%3A52&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20IPC%3A%3AMessage%3A%3AMessage%28IPC%3A%3AMessage%20const%26%29


https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A17%3A09&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29

Updated

4 years ago
status-firefox20: fixed → verified

Updated

4 years ago
QA Contact: manuela.muntean
You need to log in before you can comment on or make changes to this bug.