Closed Bug 828034 Opened 13 years ago Closed 13 years ago

crash in mozilla::ipc::RPCChannel::EnteredCxxStack

Categories

(Core Graveyard :: Plug-ins, defect, P1)

Product:
Core Graveyard
Component:
Plug-ins
Version:
20 Branch
Platform:
All
Windows 7
Type:
defect

Tracking

(firefox19 unaffected, firefox20+ verified, firefox21 fixed)

Status:
VERIFIED FIXED
Milestone:
mozilla21
Tracking Flags:
Tracking Status
firefox19 --- unaffected
firefox20 + verified
firefox21 --- fixed

People

(Reporter: scoobidiver, Assigned: bugzilla)

References

Details

(Keywords: crash, regression, topcrash)

Crash Data

Attachments

(1 file, 1 obsolete file)

Proposed crash fix, rev. 2
3.83 KB, patch
benjamin
: review+
bajaj
: approval-mozilla-aurora+
Details | Diff | Splinter Review
It first showed up in 20.0a1/20130106. The regression range is: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d8ca3e1c469e&tochange=20d1a5916ef6 Signature mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction, IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow() More Reports Search UUID 5e4f9196-94c3-4ac3-85a6-6ebc42130108 Date Processed 2013-01-08 19:28:47 Uptime 539 Last Crash 3.0 weeks before submission Install Age 9.0 minutes since version was first installed. Install Time 2013-01-08 19:19:16 Product Firefox Version 21.0a1 Build ID 20130108033457 Release Channel nightly OS Windows NT OS Version 6.1.7600 Build Architecture x86 Build Architecture Info GenuineIntel family 6 model 23 stepping 10 Crash Reason EXCEPTION_ACCESS_VIOLATION_READ Crash Address 0x0 App Notes AdapterVendorID: 0x8086, AdapterDeviceID: 0x2a42, AdapterSubsysID: 360b103c, AdapterDriverVersion: 8.15.10.1749 D3D10 Layers? D3D10 Layers- D3D9 Layers? D3D9 Layers- EMCheckCompatibility True Adapter Vendor ID 0x8086 Adapter Device ID 0x2a42 Total Virtual Memory 2147352576 Available Virtual Memory 1411579904 System Memory Use Percentage 87 Available Page File 415076352 Available Physical Memory 126259200 Frame Module Signature Source 0 xul.dll mozilla::ipc::RPCChannel::EnteredCxxStack obj-firefox/dist/include/mozilla/ipc/RPCChannel.h:197 1 xul.dll mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame obj-firefox/dist/include/mozilla/ipc/RPCChannel.h:250 2 xul.dll mozilla::ipc::RPCChannel::Call ipc/glue/RPCChannel.cpp:136 3 xul.dll mozilla::plugins::PPluginInstanceParent::CallUpdateWindow obj-firefox/ipc/ipdl/PPluginInstanceParent.cpp:1076 4 xul.dll nsWindow::OnPaint widget/windows/nsWindowGfx.cpp:203 5 xul.dll nsWindow::ProcessMessage widget/windows/nsWindow.cpp:4802 6 xul.dll nsWindow::WindowProcInternal widget/windows/nsWindow.cpp:4407 7 xul.dll CallWindowProcCrashProtected xpcom/base/nsCrashOnException.cpp:32 8 xul.dll nsWindow::WindowProc widget/windows/nsWindow.cpp:4359 9 user32.dll InternalCallWinProc 10 user32.dll UserCallWinProcCheckWow 11 user32.dll CallWindowProcAorW 12 user32.dll CallWindowProcW 13 xul.dll mozilla::plugins::PluginInstanceParent::PluginWindowHookProc dom/plugins/ipc/PluginInstanceParent.cpp:1862 14 user32.dll InternalCallWinProc 15 user32.dll UserCallWinProcCheckWow 16 user32.dll CallWindowProcAorW 17 user32.dll CallWindowProcW 18 xul.dll PluginWndProcInternal dom/plugins/base/nsPluginNativeWindowWin.cpp:327 19 xul.dll CallWindowProcCrashProtected xpcom/base/nsCrashOnException.cpp:32 20 xul.dll PluginWndProc dom/plugins/base/nsPluginNativeWindowWin.cpp:356 21 user32.dll InternalCallWinProc 22 user32.dll GetRealWindowOwner 23 user32.dll DispatchClientMessage 24 user32.dll __fnDWORD 25 ntdll.dll KiUserCallbackDispatcher 26 ntdll.dll KiUserApcDispatcher 27 user32.dll DispatchMessageW 28 xul.dll nsAppShell::ProcessNextNativeEvent widget/windows/nsAppShell.cpp:328 29 xul.dll nsBaseAppShell::OnProcessNextEvent widget/xpwidgets/nsBaseAppShell.cpp:280 ... More reports at: https://crash-stats.mozilla.com/report/list?signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29+|+mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C+mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C+IPC%3A%3AMessage+const*%29+|+mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29
It's #9 top browser crasher in 21.0a1.
Crash Signature: [@ mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction, IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow()] → [@ mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow()] [@ …
tracking-firefox20: --- → ?
Keywords: topcrash
Hardware: x86 → All
KaiRo - would you mind grabbing URLs and correlations?
Flags: needinfo?(kairo)
Keywords: needURLs
Note that we had a crash with the same signature a few months back (Fx15/16) in bug 770805 and then Benjamin found a fix, so CCing him again here. URLs: 2 https://apps.facebook.com/onthefarm/?source=FBad&affiliate=toolbar&creative&redirecting_zy_session_expired=1& 2 about:blank 1 http://apps.facebook.com/luckygemcasino/?fb_source=bookmark_apps&ref=bookmarks&count=0&fb_bmpos=5_0 1 http://apps.facebook.com/playhappyfarm/?fb_source=bookmark_apps&ref=bookmarks&count=37&fb_bmpos=2_37 1 http://www.youtube.com/watch?v=lqKQyy6T63E ...and a long list of other sites that are probably using Flash, including some adult video site, apparently. There are no correlation reports for this signature.
Flags: needinfo?(kairo)
Keywords: needURLs
See bug 829909, which almost certainly has the same root cause. Regression from bug 805591.
Assignee: nobody → aklotz
Blocks: 805591
Priority: -- → P1
I've tried to reproduce the crashes on both latest Nightly and Aurora, with intense stress testing, but without any luck. I've also tried to lower dom.ipc.plugins.hangUITimeoutSecs so that the Plugin Hang UI triggers more easily, but still no Firefox crash. My attempts were with multiple tabs and multiple windows, all with Flash content, both on Windows 7 and Windows 8.
Attached patch Proposed crash fix (obsolete) — — Splinter Review
I was finally able to obtain some useful information from new correlation reports indicating that nearly half of the recent crashes occurred on 32-bit, single-core machines. I fired up an old dual-core desktop of mine, forced Windows to boot with only one core, and eventually was able to reproduce with WinDbg attached! The debugger is showing that PluginModuleParent::CleanupFromTimeout is firing as expected. Usually when the IPC channel is closed from here, its status is already indicating an error (i.e. the I/O thread reported the channel error first). OTOH, occasionally the channel state was still showing that everything was OK (indicating that CleanupFromTimeout beat the I/O thread to the punch), so the channel was not closing with error. A crash whose stack matches this signature would follow. It looks like the correct thing to do here is for CleanupFromTimeout() to call CloseWithError() on the channel instead of doing a regular Close().
Attachment #705626 - Flags: review?(benjamin)
Comment on attachment 705626 [details] [diff] [review] Proposed crash fix I don't think this can hurt.
Attachment #705626 - Flags: review?(benjamin) → review+
Sorry, the last revision broke a bunch of tests on try. We need to select which Close* function to call depending on whether the child process was terminated directly from ShouldContinueFromReplyTimeout or from a separate thread via the Plugin Hang UI.
Attachment #705626 - Attachment is obsolete: true
Attachment #706038 - Flags: review?(benjamin)
Attachment #706038 - Flags: review?(benjamin) → review+
Try in progress: https://tbpl.mozilla.org/?tree=Try&rev=be273a2fcbe9
Keywords: checkin-needed
Whiteboard: [leave-open]
Whiteboard: [leave-open] → [leave open]
Blocks: 829909
Crash Signature: IPC::Message const*) | IPC::Message::Message(IPC::Message const&)] → IPC::Message const*) | IPC::Message::Message(IPC::Message const&)] [@ mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::Ca…
This landed on the 26th, but the Nightly from 27th still crashes, e.g. bp-22d1907c-3776-4cd6-b3f2-1bab72130127 :(
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #12) > This landed on the 26th Yes for the date, no for the build. It first landed in 21.0a1/20120128. See http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f18b12139151&tochange=80fed51ae074
(In reply to Scoobidiver from comment #13) > (In reply to Robert Kaiser (:kairo@mozilla.com) from comment #12) > > This landed on the 26th > Yes for the date, no for the build. It first landed in 21.0a1/20120128. See > http://hg.mozilla.org/mozilla-central/ > pushloghtml?fromchange=f18b12139151&tochange=80fed51ae074 Oh, interesting. And also, that's a relief. Haven't seen crashes with the 28 build yet, but it's still rather early on that day, so let's see.
Flags: needinfo?(kairo)
Looks like we're good here, and like bug 829909 is good as well. No crashes after the builds from the 27th.
Flags: needinfo?(kairo)
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
status-firefox21: affected → fixed
Target Milestone: --- → mozilla21
Whiteboard: [leave open]
Comment on attachment 706038 [details] [diff] [review] Proposed crash fix, rev. 2 [Approval Request Comment] Bug caused by (feature/regressing bug #): bug 805591 User impact if declined: Intermittent crashes when Plugin Hang UI terminates a plugin Testing completed (on m-c, etc.): Landed on m-c on Jan 28, no crashes for this signature on Nightly since Risk to taking this patch (and alternatives if risky): None String or UUID changes made by this patch: None
Attachment #706038 - Flags: approval-mozilla-aurora?
Attachment #706038 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
checkin-needed for Aurora, please.
Keywords: checkin-needed
Whiteboard: [needs-checkin-aurora]
I don't see any crash reports in Socorro, after 2013-02-01. Here are the reports for the first and third signature of this bug, within last week, because I couldn't find any reports regarding the second signature. https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A16%3A31&range_value=1&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29 https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A17%3A09&range_value=1&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29
Yes, as I said in comment #15, we're good on this one.
Status: RESOLVED → VERIFIED
There aren't any new crashes reported in Socorro, for neither one of the 3 signatures of this bug, within last month. No new crashes after Firefox 20 beta 1, neither. Reports are available here: https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A16%3A31&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29 https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20IPC%3A%3AMessage%3A%3AMessage%28IPC%3A%3AMessage%20const%26amp%3B%29&reason_type=contains&date=02%2F28%2F2013%2014%3A24%3A52&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20IPC%3A%3AMessage%3A%3AMessage%28IPC%3A%3AMessage%20const%26%29 https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26amp%3B%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29&reason_type=contains&date=02%2F08%2F2013%2015%3A17%3A09&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29
QA Contact: manuela.muntean
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: