[e10s] Flash context menu causes CPOW deadlock with AdBlock Plus

RESOLVED FIXED in Firefox 39

Status

()

defect
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: handyman, Assigned: blassey)

Tracking

(Blocks 1 bug)

Trunk
mozilla39
x86
macOS
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(e10sm6+, firefox39 fixed)

Details

Attachments

(4 attachments)

Bringing up the Flash context menu usually causes CPOW deadlock when ABP is enabled.  The context menu appears but the user cannot highlight or select any options.  Not only is the rest of the browser behavior stalled (e.g. you cannot switch tabs) but much of OSX is.  I believe all OS keyboard input is blocked (so I can’t use the debugger during this period).  When the process-hang timeout occurs, the browser retakes control and removes the context menu, displays the ‘slow tab’ message, and continues as if nothing was wrong.  Notably, you can still interact with the tab, its plugin, and now the context menu.

I tried to reproduce this with a test plugin but couldn’t.  The bug is infrequent enough that I’m not sure if the plugin itself matters.

The behavior seems to appear somewhat frequently in e10s windows but I may have seen it outside of e10s.

I was able to debug this by ssh-ing into the machine to debug (since the bug causes all keyboard events to be ignored on the host).  I was even able to get call stacks for each process (main, content, plugin) involved in the circular firing squad.  From the stacks (attached), its clear why there is deadlock.

STR:

1. Open an e10s window with ABP installed and enabled
2. Go to http://homestarrunner.com
3. Right-click the cartoon

Expected Result:
Context menu appears and you can select items from it.

Actual Result:
The context menu appears but you cannot navigate or select it.  Also, the browser (and half the OS) locks up.  Browser chrome is unresponsive.  This lasts until a timeout happens (about 15 seconds?), at which point the yellow ‘tab slow’ message appears, the context menu vanishes, and all responsiveness returns.
> Not only is the rest of the browser behavior stalled (e.g. you
> cannot switch tabs) but much of OSX is.  I believe all OS keyboard
> input is blocked (so I can’t use the debugger during this period).

This kind of nastiness sounds related to event taps -- a very buggy
facility in OS X that we nevertheless make some use of:

https://hg.mozilla.org/mozilla-central/annotate/3d3f1b07ef0f/widget/cocoa/nsToolkit.mm#l225
https://hg.mozilla.org/mozilla-central/annotate/3d3f1b07ef0f/widget/cocoa/nsToolkit.mm#l156

https://hg.mozilla.org/mozilla-central/annotate/3d3f1b07ef0f/widget/cocoa/nsChildView.mm#l6307
https://hg.mozilla.org/mozilla-central/annotate/3d3f1b07ef0f/widget/cocoa/nsChildView.mm#l6290

By any chance do you have APZ support on?  This seems unlikely, but I
thought I'd ask.  To turn it on you set layer.async-pan-zoom.enabled
to true (in about:config).
(In reply to Steven Michaud from comment #3)
> > Not only is the rest of the browser behavior stalled (e.g. you
> > cannot switch tabs) but much of OSX is.  I believe all OS keyboard
> > input is blocked (so I can’t use the debugger during this period).
> 
> This kind of nastiness sounds related to event taps

This may be, but it probably won't bring us further to fixing the bug. The fact that native menus block input to all apps isn't something we can or should fix, regardless of whether menus use event taps for that behavior or something else.
We should fix the fact that event processing in the event loop that the menu spawns is blocked because we deadlock in PPluginModuleChild::CallProcessSomeEvents.
By the way, I checked if Flash also uses event taps (by looking at its symbol imports) -- it doesn't.
Assignee: nobody → davidp99

Updated

4 years ago
Blocks: e10s-plugins
I can reproduce this (using the STR from comment #0) in today's m-c nightly.

But I no longer think the weird behavior in other apps has anything to do with event taps.

Interestingly, for as long as the "hang" lasts, the Flash context menu appears in *all* desktop spaces, above *all* apps.  I'd guess that it's acting something like a modal dialog.
(In reply to David Parks [:handyman] from comment #2)
> Created attachment 8561294 [details]
> Plugin Process Stack

I got a slightly different stack for the plugin process 
    2553 Thread_1366172   DispatchQueue_1: com.apple.main-thread  (serial)
    + 2553 start  (in plugin-container) + 52  [0x1000019c4]
    +   2553 main  (in plugin-container) + 101  [0x100001cd5]  plugin-container.cpp:211
    +     2553 XRE_InitChildProcess  (in XUL) + 1292  [0x1026fc63c]  nsEmbedFunctions.cpp:580
    +       2553 MessageLoop::Run()  (in XUL) + 76  [0x10073349c]  message_loop.cc:233
    +         2553 base::MessagePumpCFRunLoopBase::Run(base::MessagePump::Delegate*)  (in XUL) + 134  [0x10073e696]  message_pump_mac.mm:213
    +           2553 base::MessagePumpNSApplication::DoRun(base::MessagePump::Delegate*)  (in XUL) + 276  [0x10073ebb4]  message_pump_mac.mm:663
    +             2553 -[NSApplication run]  (in AppKit) + 553  [0x7fff95cc799c]
    +               2553 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]  (in AppKit) + 122  [0x7fff95cd389b]
    +                 2553 _DPSNextEvent  (in AppKit) + 1434  [0x7fff95cd424e]
    +                   2553 _BlockUntilNextEventMatchingListInModeWithFilter  (in HIToolbox) + 65  [0x7fff9118c5bc]
    +                     2553 ReceiveNextEventCommon  (in HIToolbox) + 479  [0x7fff9118c7b7]
    +                       2553 RunCurrentEventLoopInMode  (in HIToolbox) + 226  [0x7fff9118ca0d]
    +                         2553 CFRunLoopRunSpecific  (in CoreFoundation) + 309  [0x7fff8a931e75]
    +                           2553 __CFRunLoopRun  (in CoreFoundation) + 831  [0x7fff8a9323ef]
    +                             2553 __CFRunLoopDoSources0  (in CoreFoundation) + 242  [0x7fff8a932c62]
    +                               2553 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__  (in CoreFoundation) + 17  [0x7fff8a9415b1]
    +                                 2553 base::MessagePumpCFRunLoopBase::RunWorkSource(void*)  (in XUL) + 39  [0x10073e187]  message_pump_mac.mm:277
    +                                   2553 MessageLoop::DoWork()  (in XUL) + 235  [0x100733d1b]  message_loop.cc:447
    +                                     2553 MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&)  (in XUL) + 129  [0x1007339b1]  message_loop.cc:361
    +                                       2553 mozilla::ipc::MessageChannel::OnMaybeDequeueOne()  (in XUL) + 207  [0x100749b9f]  MessageChannel.cpp:1122
    +                                         2553 mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message const&)  (in XUL) + 124  [0x10074cb2c]  MessageChannel.cpp:1136
    +                                           2553 mozilla::ipc::MessageChannel::DispatchInterruptMessage(IPC::Message const&, unsigned long)  (in XUL) + 399  [0x10074cd1f]  MessageChannel.cpp:1278
    +                                             2553 mozilla::plugins::PPluginModuleChild::OnCallReceived(IPC::Message const&, IPC::Message*&)  (in XUL) + 368  [0x100828640]  .PPluginModuleChild.cpp:1038
    +                                               2553 mozilla::plugins::PPluginInstanceChild::OnCallReceived(IPC::Message const&, IPC::Message*&)  (in XUL) + 7101  [0x10081815d]  .PPluginInstanceChild.cpp:2212
    +                                                 2553 mozilla::plugins::PluginInstanceChild::AnswerNPP_HandleEvent(mozilla::plugins::NPRemoteEvent const&, short*)  (in XUL) + 187  [0x101ce627b]  PluginInstanceChild.cpp:820
    +                                                   2553 ???  (in FlashPlayer-10.6)  load address 0x10e7fc000 + 0x3b3596  [0x10ebaf596]
    +                                                     2553 ???  (in FlashPlayer-10.6)  load address 0x10e7fc000 + 0x42a61d  [0x10ec2661d]
    +                                                       2553 ???  (in FlashPlayer-10.6)  load address 0x10e7fc000 + 0x3457bf  [0x10eb417bf]
    +                                                         2553 ???  (in FlashPlayer-10.6)  load address 0x10e7fc000 + 0x3f8c8e  [0x10ebf4c8e]
    +                                                           2553 ???  (in FlashPlayer-10.6)  load address 0x10e7fc000 + 0x3dc658  [0x10ebd8658]
    +                                                             2553 ???  (in FlashPlayer-10.6)  load address 0x10e7fc000 + 0x4222da  [0x10ec1e2da]
    +                                                               2553 mozilla::plugins::child::_popupcontextmenu(_NPP*, _NPNSMenu*)  (in XUL) + 166  [0x101cebac6]  PluginModuleChild.cpp:1821
    +                                                                 2553 mozilla::plugins::PluginUtilsOSX::ShowCocoaContextMenu(void*, int, int, void*, void (*)(void*))  (in XUL) + 270  [0x101d056de]  PluginUtilsOSX.mm:243
    +                                                                   2553 -[NSMenu popUpMenuPositioningItem:atLocation:inView:]  (in AppKit) + 354  [0x7fff96250af5]
    +                                                                     2553 -[NSCarbonMenuImpl _popUpMenuPositioningItem:atCocoaIndex:atLocation:inView:]  (in AppKit) + 395  [0x7fff960c987d]
    +                                                                       2553 _NSSLMPopUpCarbonMenu3  (in AppKit) + 4153  [0x7fff95f5b161]
    +                                                                         2553 _HandlePopUpMenuSelection7  (in HIToolbox) + 596  [0x7fff911bf2f6]
    +                                                                           2553 PopUpMenuSelectCore(MenuData*, Point, double, Point, unsigned short, unsigned int, Rect const*, unsigned short, unsigned int, Rect const*, Rect const*, __CFString const*, OpaqueMenuRef**, unsigned short*)  (in HIToolbox) + 1526  [0x7fff911c00d6]
    +                                                                             2553 TrackMenuCommon(MenuSelectData&, unsigned char*)  (in HIToolbox) + 1545  [0x7fff911d7c61]
    +                                                                               2553 IsUserStillTracking(MenuSelectData*, unsigned char*)  (in HIToolbox) + 186  [0x7fff911d805a]
    +                                                                                 2553 AcquireNextEventInMode  (in HIToolbox) + 51  [0x7fff911da5cd]
    +                                                                                   2553 ReceiveNextEventCommon  (in HIToolbox) + 479  [0x7fff9118c7b7]
    +                                                                                     2553 RunCurrentEventLoopInMode  (in HIToolbox) + 226  [0x7fff9118ca0d]
    +                                                                                       2553 CFRunLoopRunSpecific  (in CoreFoundation) + 309  [0x7fff8a931e75]
    +                                                                                         2553 __CFRunLoopRun  (in CoreFoundation) + 1525  [0x7fff8a9326a5]
    +                                                                                           2553 __CFRunLoopDoTimers  (in CoreFoundation) + 298  [0x7fff8a9e85aa]
    +                                                                                             2553 __CFRunLoopDoTimer  (in CoreFoundation) + 1151  [0x7fff8a976f1f]
    +                                                                                               2553 __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__  (in CoreFoundation) + 20  [0x7fff8a9773e4]
    +                                                                                                 2553 __NSFireTimer  (in Foundation) + 96  [0x7fff968f8714]
    +                                                                                                   2553 mozilla::plugins::PPluginModuleChild::CallProcessSomeEvents()  (in XUL) + 365  [0x10082414d]  .PPluginModuleChild.cpp:226
    +                                                                                                     2553 mozilla::ipc::MessageChannel::Call(IPC::Message*, IPC::Message*)  (in XUL) + 670  [0x10074bebe]  CondVar.h:79
    +                                                                                                       2553 PR_WaitCondVar  (in libnss3.dylib) + 253  [0x10028a66d]  ptsynch.c:385
    +                                                                                                         2553 _pthread_cond_wait  (in libsystem_pthread.dylib) + 727  [0x7fff8ed54c3b]
    +                                                                                                           2553 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7fff8f62b716]
and the call stack for the content process:


    2651 Thread_4054616   DispatchQueue_1: com.apple.main-thread  (serial)
    + 2651 start  (in plugin-container) + 52  [0x1000019c4]
    +   2651 main  (in plugin-container) + 101  [0x100001cd5]  plugin-container.cpp:211
    +     2651 XRE_InitChildProcess  (in XUL) + 1292  [0x1026fc63c]  nsEmbedFunctions.cpp:580
    +       2651 MessageLoop::Run()  (in XUL) + 76  [0x1007335ac]  message_loop.cc:233
    +         2651 XRE_RunAppShell  (in XUL) + 207  [0x1026fca3f]  nsEmbedFunctions.cpp:743
    +           2651 nsAppShell::Run()  (in XUL) + 326  [0x101f50e06]  nsAppShell.mm:651
    +             2651 -[NSApplication run]  (in AppKit) + 553  [0x7fff95cc799c]
    +               2651 -[GeckoNSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]  (in XUL) + 82  [0x101f4fe52]  nsAppShell.mm:118
    +                 2651 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]  (in AppKit) + 122  [0x7fff95cd389b]
    +                   2651 _DPSNextEvent  (in AppKit) + 1434  [0x7fff95cd424e]
    +                     2651 _BlockUntilNextEventMatchingListInModeWithFilter  (in HIToolbox) + 65  [0x7fff9118c5bc]
    +                       2651 ReceiveNextEventCommon  (in HIToolbox) + 479  [0x7fff9118c7b7]
    +                         2651 RunCurrentEventLoopInMode  (in HIToolbox) + 226  [0x7fff9118ca0d]
    +                           2651 CFRunLoopRunSpecific  (in CoreFoundation) + 309  [0x7fff8a931e75]
    +                             2651 __CFRunLoopRun  (in CoreFoundation) + 831  [0x7fff8a9323ef]
    +                               2651 __CFRunLoopDoSources0  (in CoreFoundation) + 242  [0x7fff8a932c62]
    +                                 2651 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__  (in CoreFoundation) + 17  [0x7fff8a9415b1]
    +                                   2651 nsAppShell::ProcessGeckoEvents(void*)  (in XUL) + 302  [0x101f5078e]  nsAppShell.mm:377
    +                                     2651 nsBaseAppShell::NativeEventCallback()  (in XUL) + 99  [0x101f08533]  nsBaseAppShell.cpp:98
    +                                       2651 NS_ProcessPendingEvents(nsIThread*, unsigned int)  (in XUL) + 78  [0x1004e102e]  nsThreadUtils.cpp:207
    +                                         2651 nsThread::ProcessNextEvent(bool, bool*)  (in XUL) + 863  [0x1004c1bbf]  nsThread.cpp:855
    +                                           2651 mozilla::ipc::DoWorkRunnable::Run()  (in XUL) + 44  [0x10074f03c]  MessagePump.cpp:233
    +                                             2651 MessageLoop::DoWork()  (in XUL) + 235  [0x100733e2b]  message_loop.cc:447
    +                                               2651 MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&)  (in XUL) + 129  [0x100733ac1]  message_loop.cc:361
    +                                                 2651 mozilla::ipc::MessageChannel::OnMaybeDequeueOne()  (in XUL) + 207  [0x100749caf]  MessageChannel.cpp:1124
    +                                                   2651 mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message const&)  (in XUL) + 137  [0x10074cc59]  MessageChannel.cpp:1140
    +                                                     2651 mozilla::ipc::MessageChannel::DispatchAsyncMessage(IPC::Message const&)  (in XUL) + 126  [0x10074d76e]  MessageChannel.cpp:1213
    +                                                       2651 mozilla::dom::PContentChild::OnMessageReceived(IPC::Message const&)  (in XUL) + 160  [0x10094cc30]  .PContentChild.cpp:4777
    +                                                         2651 mozilla::dom::PBrowserChild::OnMessageReceived(IPC::Message const&)  (in XUL) + 21509  [0x1008e0725]  .PBrowserChild.cpp:2785
    +                                                           2651 non-virtual thunk to mozilla::dom::TabChild::RecvRealMouseButtonEvent(mozilla::WidgetMouseEvent const&)  (in XUL) + 13  [0x101d8938d]  TabChild.cpp:2172
    +                                                             2651 mozilla::dom::TabChild::RecvRealMouseButtonEvent(mozilla::WidgetMouseEvent const&)  (in XUL) + 175  [0x101d891ff]  TabChild.cpp:2170
    +                                                               2651 mozilla::layers::APZCCallbackHelper::DispatchWidgetEvent(mozilla::WidgetGUIEvent&)  (in XUL) + 68  [0x100ccaef4]  APZCCallbackHelper.cpp:433
    +                                                                 2651 mozilla::widget::PuppetWidget::DispatchEvent(mozilla::WidgetGUIEvent*, nsEventStatus&)  (in XUL) + 320  [0x101f01b40]  PuppetWidget.cpp:324
    +                                                                   2651 nsView::HandleEvent(mozilla::WidgetGUIEvent*, bool)  (in XUL) + 138  [0x101ef25ea]  nsView.cpp:1085
    +                                                                     2651 nsViewManager::DispatchEvent(mozilla::WidgetGUIEvent*, nsView*, nsEventStatus*)  (in XUL) + 463  [0x101ef407f]  nsViewManager.cpp:776
    +                                                                       2651 PresShell::HandleEvent(nsIFrame*, mozilla::WidgetGUIEvent*, bool, nsEventStatus*)  (in XUL) + 8754  [0x1021d0692]  nsPresShell.cpp:7727
    +                                                                         2651 PresShell::HandlePositionedEvent(nsIFrame*, mozilla::WidgetGUIEvent*, nsEventStatus*)  (in XUL) + 371  [0x1021d0bf3]  nsPresShell.cpp:7927
    +                                                                           2651 PresShell::HandleEventInternal(mozilla::WidgetEvent*, nsEventStatus*)  (in XUL) + 1051  [0x1021d102b]  nsPresShell.cpp:8093
    +                                                                             2651 PresShell::DispatchEventToDOM(mozilla::WidgetEvent*, nsEventStatus*, nsPresShellEventCB*)  (in XUL) + 340  [0x1021d1ee4]  nsPresShell.cpp:8198
    +                                                                               2651 mozilla::EventDispatcher::Dispatch(nsISupports*, nsPresContext*, mozilla::WidgetEvent*, nsIDOMEvent*, nsEventStatus*, mozilla::EventDispatchingCallback*, nsTArray<mozilla::dom::EventTarget*>*)  (in XUL) + 3483  [0x1019c334b]  EventDispatcher.cpp:633
    +                                                                                 2651 mozilla::EventTargetChainItem::HandleEventTargetChain(nsTArray<mozilla::EventTargetChainItem>&, mozilla::EventChainPostVisitor&, mozilla::EventDispatchingCallback*, mozilla::ELMCreationDetector&)  (in XUL) + 425  [0x1019c2139]  EventDispatcher.cpp:299
    +                                                                                   2651 mozilla::EventListenerManager::HandleEventInternal(nsPresContext*, mozilla::WidgetEvent*, nsIDOMEvent**, mozilla::dom::EventTarget*, nsEventStatus*)  (in XUL) + 1054  [0x1019c6ede]  EventListenerManager.cpp:1124
    +                                                                                     2651 mozilla::EventListenerManager::HandleEventSubType(mozilla::EventListenerManager::Listener*, nsIDOMEvent*, mozilla::dom::EventTarget*)  (in XUL) + 139  [0x1019c67cb]  EventListenerManager.cpp:976
    +                                                                                       2651 nsPluginInstanceOwner::HandleEvent(nsIDOMEvent*)  (in XUL) + 798  [0x101cd816e]  nsPluginInstanceOwner.cpp:1652
    +                                                                                         2651 nsPluginInstanceOwner::ProcessMouseDown(nsIDOMEvent*)  (in XUL) + 175  [0x101cd7e2f]  nsPluginInstanceOwner.cpp:1581
    +                                                                                           2651 nsPluginInstanceOwner::ProcessEvent(mozilla::WidgetGUIEvent const&)  (in XUL) + 1709  [0x101cd7b7d]  nsPluginInstanceOwner.cpp:1939
    +                                                                                             2651 nsNPAPIPluginInstance::HandleEvent(void*, short*, NSPluginCallReentry)  (in XUL) + 127  [0x101cd1abf]  nsNPAPIPluginInstance.cpp:681
    +                                                                                               2651 mozilla::plugins::PluginModuleParent::NPP_HandleEvent(_NPP*, void*)  (in XUL) + 135  [0x101cf59c7]  PluginModuleParent.cpp:1562
    +                                                                                                 2651 mozilla::plugins::PluginInstanceParent::NPP_HandleEvent(void*)  (in XUL) + 394  [0x101cf268a]  PluginInstanceParent.cpp:1330
    +                                                                                                   2651 mozilla::plugins::PPluginInstanceParent::CallNPP_HandleEvent(mozilla::plugins::NPRemoteEvent const&, short*)  (in XUL) + 347  [0x10081b25b]  .PPluginInstanceParent.cpp:451
    +                                                                                                     2651 mozilla::ipc::MessageChannel::Call(IPC::Message*, IPC::Message*)  (in XUL) + 670  [0x10074bfde]  CondVar.h:79
    +                                                                                                       2651 PR_WaitCondVar  (in libnss3.dylib) + 227  [0x10028a653]  ptsynch.c:264
    +                                                                                                         2651 _pthread_cond_wait  (in libsystem_pthread.dylib) + 727  [0x7fff8ed54c3b]
    +                                                                                                           2651 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7fff8f62b716]


the line that results in a sync CPOW call is "elementDoc = this.currentElement.ownerDocument;"

Bill, my assumption is that the content process is blocked on the process. Is there anyway to detect that from the chrome process? If so, we could just throw rather than make the sync call and get hung.
Flags: needinfo?(wmccloskey)
I think I see what the problem is. When we put up the context menu, the plugin needs to ask someone to spin a nested event loop for it. Right now it's asking the chrome process. That's wrong. It should be asking the content process. If it asks the content process, then we'll run a nested event loop in the content process, immediately answer the CPOW, and then everything will work normally. Any plugin-relevant events will be passed from the chrome process to the content process to the plugin.

David, can you try changing the coder here:
http://mxr.mozilla.org/mozilla-central/source/dom/plugins/ipc/PluginModuleChild.cpp#1823
so that it says |InstCast(instance)->Manager()| instead of |PluginModuleChild::GetChrome()|? That will pass the PluginModuleChild for the content process into the code that asks for a nested event loop rather than the PluginModuleChild for chrome.
Flags: needinfo?(wmccloskey)
yup, that did the trick
Assignee: davidp99 → blassey.bugs
Attachment #8584558 - Flags: review?(wmccloskey)
Attachment #8584558 - Flags: review?(wmccloskey) → review+

Comment 12

4 years ago
https://hg.mozilla.org/mozilla-central/rev/eab17adb9a35
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla39
You need to log in before you can comment on or make changes to this bug.