554447 - [OOPP] With IPC enabled, Sprint's android page doesn't show animation & crashes plugin on window-resize

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Daniel Holbert [:dholbert]]

STEPS TO REPRODUCE:
 1. Load http://now.sprint.com/android/
 2. After loading splash page, you'll arrive at a page with 3 lower tabs:
      Meet Android TM|See the Phones|Featured Apps
 3. Resize Firefox (larger or smaller; doesn't matter)

ACTUAL RESULTS:
  - After step 2: The top of the page (above the tabs) is mostly blank
  - After step 3: Firefox hangs for 5 sec, and then I get "The Shockwave Flash plugin has crashed. Reload the page to try again."

EXPECTED RESULTS:
  - There should be an animation with spinning phones in the blank area, after step 2.
  - There shouldn't be a crash after step 3.

I get ACTUAL RESULTS from:
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a4pre) Gecko/20100323 Minefield/3.7a4pre

I get EXPECTED RESULTS with that build if I disable 'dom.ipc.plugins.enabled', and also with a 1.9.2 nightly:
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3pre) Gecko/20100323 Namoroka/3.6.3pre

(This should probably be "blocking-lorentz?" but I'm not sure what fields to use for that.)
I'm using Ubuntu 10.04 beta, btw.  about:plugins says I have "Shockwave Flash 10.0 r45"

Comment 1

[Daniel Holbert [:dholbert]]

FWIW, this has the same regression range as bug 554453.

Quoting bug 554453 comment #4:
> LAST GOOD BUILD:
> Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a4pre) Gecko/20100318
> Minefield/3.7a4pre
> Built from http://hg.mozilla.org/mozilla-central/rev/2cc5ad2cf917
> 
> FIRST BAD BUILD:
> Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a4pre) Gecko/20100319
> Minefield/3.7a4pre
> Built from http://hg.mozilla.org/mozilla-central/rev/5108c4c2c043
> 
> Regression pushlog:
> http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=2cc5ad2cf917&tochange=5108c4c2c043
> 
> Looks like cjones had a number of OOPP-related changesets during that period --
> not sure which one would be responsible.

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Repro'd with flash 10.0 r45 (but not r42 :S).

Comment 3

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Oh lordy.  I was just thinking about this problem yesterday, abstractly.  Not any more!

IO thread:

#3  0x00007f8bc58326f6 in PR_Lock (lock=0x11b2930) at /home/cjones/mozilla/mozilla-central/nsprpub/pr/src/pthreads/ptsynch.c:206
#4  0x00007f8bc7907172 in mozilla::Mutex::Lock (this=0x11b1d18) at BlockingResourceBase.cpp:261
#5  0x00007f8bc6485af9 in MutexAutoLock (this=0x7fff08877d70, aLock=...) at ../../dist/include/mozilla/Mutex.h:180
#6  0x00007f8bc783abde in mozilla::ipc::RPCChannel::OnMessageReceived (this=0x11b1cf8, msg=...) at /home/cjones/mozilla/mozilla-central/ipc/glue/RPCChannel.cpp:671
#7  0x00007f8bc7a6013d in IPC::Channel::ChannelImpl::ProcessIncomingMessages (this=0x11b5e00) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/chrome/common/ipc_channel_posix.cc:548
#8  0x00007f8bc7a609bd in IPC::Channel::ChannelImpl::OnFileCanReadWithoutBlocking (this=0x11b5e00, fd=3) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/chrome/common/ipc_channel_posix.cc:722
#9  0x00007f8bc7a4daf3 in base::MessagePumpLibevent::OnLibeventNotification (fd=3, flags=2, context=0x11b5e00) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_libevent.cc:210
#10 0x00007f8bc79cec22 in event_process_active (base=0x11aa600) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/third_party/libevent/event.c:385
#11 0x00007f8bc79cef4b in event_base_loop (base=0x11aa600, flags=1) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/third_party/libevent/event.c:522
#12 0x00007f8bc7a4e070 in base::MessagePumpLibevent::Run (this=0x11a80d0, delegate=0x7fff08878320) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_libevent.cc:330


Main thread:

#0  0x00007f8bc94a204b in write () from /lib/libpthread.so.0
#1  0x00007f8bc7a67bd2 in base::MessagePumpForUI::ScheduleWork (this=0x11b4060) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_glib.cc:307
#2  0x00007f8bc79f3e58 in MessageLoop::PostTask_Helper (this=0x7f8bbcb7ee20, from_here=..., task=0x7f8ba0ab1c20, delay_ms=0, nestable=true) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:306
#3  0x00007f8bc79f3c0d in MessageLoop::PostTask (this=0x7f8bbcb7ee20, from_here=..., task=0x7f8ba0ab1c20) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:249
#4  0x00007f8bc7839b17 in mozilla::ipc::RPCChannel::EnqueuePendingMessages (this=0x11b1cf8) at /home/cjones/mozilla/mozilla-central/ipc/glue/RPCChannel.cpp:358
#5  0x00007f8bc783a7ea in mozilla::ipc::RPCChannel::ExitedCxxStack (this=0x11b1cf8) at /home/cjones/mozilla/mozilla-central/ipc/glue/RPCChannel.cpp:603
#6  0x00007f8bc783b467 in ~CxxStackFrame (this=0x7f8bbcb7e7b0, __in_chrg=<value optimized out>) at ../../dist/include/mozilla/ipc/RPCChannel.h:259
#7  0x00007f8bc7839dc2 in mozilla::ipc::RPCChannel::OnMaybeDequeueOne (this=0x11b1cf8) at /home/cjones/mozilla/mozilla-central/ipc/glue/RPCChannel.cpp:403
#8  0x00007f8bc783f93a in DispatchToMethod<mozilla::ipc::RPCChannel, void (mozilla::ipc::RPCChannel::*)()> (obj=0x11b1cf8, method=0x7f8bc7839b46 <mozilla::ipc::RPCChannel::OnMaybeDequeueOne()>, arg=...) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/tuple.h:383
#9  0x00007f8bc783f7e2 in RunnableMethod<mozilla::ipc::RPCChannel, void (mozilla::ipc::RPCChannel::*)(), Tuple0>::Run (this=0x11b2af0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/task.h:307
#10 0x00007f8bc783b521 in mozilla::ipc::RPCChannel::RefCountedTask::Run (this=0x11b2b30) at ../../dist/include/mozilla/ipc/RPCChannel.h:402
#11 0x00007f8bc783b624 in mozilla::ipc::RPCChannel::DequeueTask::Run (this=0x7f8ba04daf10) at ../../dist/include/mozilla/ipc/RPCChannel.h:427
#12 0x00007f8bc79f3f9a in MessageLoop::RunTask (this=0x7f8bbcb7ee20, task=0x7f8ba04daf10) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:336
#13 0x00007f8bc79f400a in MessageLoop::DeferOrRunPendingTask (this=0x7f8bbcb7ee20, pending_task=...) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:344
#14 0x00007f8bc79f4408 in MessageLoop::DoWork (this=0x7f8bbcb7ee20) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:444
#15 0x00007f8bc7a679ad in base::MessagePumpForUI::HandleDispatch (this=0x11b4060) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_glib.cc:264
#16 0x00007f8bc7a67027 in WorkSourceDispatch (source=0x11b41b0, unused_func=0, unused_data=0x0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_glib.cc:109
#17 0x00007f8bc2455bce in g_main_dispatch (context=0x11b40d0) at /build/buildd/glib2.0-2.22.3/glib/gmain.c:1960


The IO thread and main thread are deadlocked.  The main thread is trying to wake up the IO thread by writing to its wakeup pipe while holding RPCChannel.mMutex, while the IO thread is trying to acquire that mutex.  But the write() is blocking, presumably because the wakeup pipe's buffer is full.

There are some things in the regression range that might have exacerbated this, bug 554466 among them (which I've fixed), but we've been vulnerable to this basically forever.  I'll confirm that that fix makes this "go away" then fix this bug for realz.  We always need to PostTask both IO->main and main->IO without holding the mutex.

Comment 4

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

This doesn't explain the "plugin crashed" UI coming.  I suspect that this is happening both in the plugin process and browser process, and I happened to catch a browser lockup.

Comment 5

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to comment #4)
> This doesn't explain the "plugin crashed" UI coming.  I suspect that this is
> happening both in the plugin process and browser process, and I happened to
> catch a browser lockup.

Oops sorry, these stacks were from the plugin process.

Comment 6

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Yep, bug 554466 makes this go away.  Checking in now.

Comment 7

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to comment #3)
> Oh lordy.  I was just thinking about this problem yesterday, abstractly.  Not
> any more!
> 

dholbert, I'm considering dup'ing this and bug 554453 on bug 554466, because bug 554466 was especially horrible (allowed the event queue size to diverge from the number of received messages in bad cases).  The deadlock is still a (potentially) bad bug, but we'd need to be processing orders of magnitude more messages to hit it.  OK with you if I spin it off?

Comment 8

[Benjamin Smedberg]

Spin or morph, yes. With the workaround in place this doesn't block beta, in any case.

Comment 9

[Daniel Holbert [:dholbert]]

(In reply to comment #7)
> OK with you if I spin it off?

Fine by me -- do whatever makes the most sense. :)

Meanwhile I'll make sure this is fixed in today's nightly (as it looks like it should be, given that bug 554466 landed last night)...

Comment 10

[Daniel Holbert [:dholbert]]

Yup, steps to reproduce are WFM in today's nightly. Leaving open so that cjones can spin off any now-masked parts of this bug and then dupe this, per comment 7.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a4pre) Gecko/20100324 Minefield/3.7a4pre

Comment 12

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Filed followup bug 554685.