592002 - OOPP crash with Flash Player due to out of turn RPC replies [hang | mozilla::plugins::PPluginScriptableObjectParent::CallHasProperty(mozilla::plugins::PPluginIdentifierParent*, bool*)]

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Charles]

We have a situation where the Flash Player is reported as crashed by Firefox when OOPP is enabled but when we disable this feature there is no problem with playing the content.  Please follow the below instructions for reproduction steps.

1. go to http://www.tbs.com/tveverywhere/ and let the page fully load
2. click "get access" - which loads the Adobe TV Service SWF
3. Select Comcast - which redirects to the Comcast login  (Upper right corner
4. After the CC page fully loads, click the browser back button

Comment 1

[Charles]

This is a serious issue for us and we'd like to work with you to find solution to this ASAP.

Comment 2

[Jet Villegas (inactive)]

More info:

- when OOPP is disabled with dom.ipc.plugins.enabled.libflashplayer.so=false or dom.ipc.plugins.enabled=false the issue is no longer reproducible. The browser behaves normally.
- when there's no timeout specified : dom.ipc.plugins.timeoutSecs=-1 , the issue still appears as in hangs indefinitely

Comment 3

[Benjamin Smedberg]

I can't reproduce this (on Windows 7). What version of Firefox and what OS are you testing with? When Flash "crashes", do you get the option to submit a crash report? Do you have the crash report links from about:crashes ?

Comment 4

[juan becerra [:juanb]]

I'm able to see this problem on a Windows 7 VM, using 10.1.82.76, on Mozilla/5.0 (Windows NT 6.1; rv:2.0b5pre) Gecko/20100830 Firefox/4.0b5pre

Disabling OOPP by flipping the pref makes the hang go away.

This is the signature for the hang: http://crash-stats.mozilla.com/report/index/2ad36a7c-8f8c-4363-9a65-ad81d2100830

Comment 5

[Charles]

I just downloaded the nightly and was able to reproduce on Windows 7.

Comment 6

[Jet Villegas (inactive)]

http://crash-stats.mozilla.com/report/index/8b9c520e-1f21-45a4-8e8f-13e3b2100827
http://crash-stats.mozilla.com/report/index/930dc170-ad54-493b-ac1c-2a5e92100827
http://crash-stats.mozilla.com/report/index/0515e41e-794f-4a4c-8354-a53c82100827
http://crash-stats.mozilla.com/report/index/e63afad5-2b3f-469a-92c1-b60c92100827

Comment 7

[Benjamin Smedberg]

ok, I have this in recording.

Comment 8

[Joel Huff]

If it's helpful to provide context for the architecture of the website and swfs used in this site, there's a group at Adobe and Turner that can be available to share more information.  Please let me know if that's desired, I can set it up.

The teams are also eager to explore ideas for workarounds, since there have been delays to roll-out of some important partner integrations.  If there are early ideas, we're happy to explore them.

Comment 10

[Benjamin Smedberg]

>	xul.dll!CrashReporter::CreatePairedMinidumps(childPid=0xcb673af0, childBlamedThread=0x0792c6c0, pairGUID=0x075b72c8, childDump=0x7153b3b0, parentDump=0x001bce88)  Line 1789	C++
 	xul.dll!mozilla::plugins::PluginModuleParent::ShouldContinueFromReplyTimeout()  Line 248	C++
 	xul.dll!mozilla::ipc::SyncChannel::ShouldContinueFromTimeout()  Line 262	C++
 	xul.dll!mozilla::ipc::RPCChannel::Call(msg=0x0192c6c0, reply=0x001bceb8)  Line 210	C++
 	xul.dll!mozilla::plugins::PPluginScriptableObjectParent::CallHasProperty(aId=0x0a212060, aHasProperty=0x001bcf0f)  Line 289	C++
 	xul.dll!mozilla::plugins::PluginScriptableObjectParent::ScriptableHasProperty(aObject=0x06858d30, aName=0x0783f6e0)  Line 312	C++
 	xul.dll!NPObjWrapper_NewResolve(cx=0x0271cfb0, obj=0x06b51840, id=0x0783f6e0, flags=0x00000001, objp=0x001bcf88)  Line 1655	C++
 	xul.dll!CallResolveOp(cx=0x0a126510, start=0x06b517b0, obj=0x06b51840, id=0x0783f6e0, flags=0x00000001, objp=0x001bcfe0, propp=0x001bcfec, recursedp=0x001bcfdf)  Line 4348	C++
 	xul.dll!js_GetPropertyHelper(cx=, obj=, id=, getHow=, vp=)  Line 4779	C++
 	xul.dll!js_GetMethod(cx=0x00000000, obj=0x00000000, id=0x0783f6e0, getHow=0x00000003, vp=0x001bd108)  Line 4876	C++
 	xul.dll!js::Interpret(cx=0x0271cfb0)  Line 4112	C++
 	xul.dll!js::Execute(cx=0x0271cfb0, chain=0x07847f90, script=0x0a20a0c0, down=0x00000000, flags=0x00000000, result=0x001bd6b0)  Line 881	C++
 	xul.dll!JS_EvaluateUCScriptForPrincipals(cx=0x0271cfb0, obj=0x07847f90, principals=0x09e73b04, chars=0x07cf7598, length=0x00000043, filename=0x0765af18, lineno=0x00000000, rval=0x001bd6b0)  Line 4801	C++
 	xul.dll!nsJSContext::EvaluateStringWithValue(aScript={...}, aScopeObject=0x07847f90, aPrincipal=0x09e73b00, aURL=0x0765af18, aLineNo=0x00000000, aVersion=0x00000000, aRetValue=0x001bd738, aIsUndefined=0x00000000)  Line 1612	C++
 	xul.dll!mozilla::plugins::parent::_evaluate(npp=0x064b79c8, npobj=0x068af4d0, script=0x001bd8b8, result=0x001bd8c0)  Line 1641	C++
 	xul.dll!mozilla::plugins::PluginScriptableObjectParent::AnswerNPN_Evaluate(aScript={...}, aResult=0x001bd93c, aSuccess=0x001bd920)  Line 1234	C++
 	xul.dll!mozilla::plugins::PPluginScriptableObjectParent::OnCallReceived(__msg={...}, __reply=0x00000000)  Line 692	C++
 	xul.dll!mozilla::plugins::PPluginModuleParent::OnCallReceived(__msg={...}, __reply=0x00000000)  Line 596	C++
 	xul.dll!mozilla::ipc::RPCChannel::DispatchIncall(call={...})  Line 511	C++
 	xul.dll!mozilla::ipc::RPCChannel::Incall(call={...}, stackDepth=0x00000000)  Line 497	C++
 	xul.dll!mozilla::ipc::RPCChannel::OnMaybeDequeueOne()  Line 434	C++
 	xul.dll!MessageLoop::RunTask(task=0x00000000)  Line 344	C++
 	xul.dll!MessageLoop::DeferOrRunPendingTask(pending_task={...})  Line 354	C++
 	xul.dll!MessageLoop::DoWork()  Line 451	C++
 	xul.dll!mozilla::ipc::DoWorkRunnable::Run()  Line 71	C++
 	xul.dll!nsThread::ProcessNextEvent(mayWait=0x00000001, result=0x001bdaec)  Line 553	C++
 	xul.dll!mozilla::ipc::MessagePump::Run(aDelegate=0x0084f0c0)  Line 135	C++
 	xul.dll!MessageLoop::RunInternal()  Line 219	C++
 	xul.dll!MessageLoop::RunHandler()  Line 203	C++
 	xul.dll!PL_DHashTableOperate(table=0x00000000, key=0x00000000, op=0xcb672c64)  Line 625	C
 	xul.dll!MessageLoop::Run()  Line 177	C++
 	xul.dll!nsBaseAppShell::Run()  Line 181	C++
 	xul.dll!nsAppShell::Run()  Line 249	C++
 	xul.dll!nsAppStartup::Run()  Line 192	C++
 	xul.dll!XRE_main(argc=0x00000001, argv=0x0082a0a8, aAppData=0x008153c0)  Line 3661	C++

The script being evaluated is try { __flash__toXML(swfLoaded()) ; } catch (e) { "<undefined/>"; }

Can somebody point me to where swfLoaded and __flash__toXML are coming from? According to the developer console, swfLoaded is a boolean, not a function.

Comment 11

[Benjamin Smedberg]

plugin stack at roughly the same time:

 	USER32.dll!_MsgWaitForMultipleObjects@20() 	
 	xul.dll!mozilla::ipc::RPCChannel::WaitForNotify()  Line 888	C++
 	xul.dll!mozilla::ipc::RPCChannel::Call(msg=0xfffffdc0, reply=0x0012c62c)  Line 201	C++
 	xul.dll!mozilla::plugins::PPluginScriptableObjectChild::CallNPN_Evaluate(aScript={...}, aResult=0x0012c684, aSuccess=0x0012c66f)  Line 124	C++
 	xul.dll!mozilla::plugins::PluginScriptableObjectChild::Evaluate(aScript=0x0012c6f8, aResult=0x0012c6e8)  Line 1093	C++
 	xul.dll!mozilla::plugins::child::_evaluate(aNPP=0x009451ac, aObject=0x00916520, aScript=0x0012c6f8, aResult=0x0012c6e8)  Line 1244	C++
 	npswf32.dll!F956976105____________(expression=0x0706b358)  Line 1290	
 	npswf32.dll!F400962363________________(splayer=0x03deb000, pCallerCxt=0x058102b8, expression=0x0706b358, checker={...})  Line 463	
 	npswf32.dll!F_395357591_____________________________(expression=0x06bbe310)  Line 129	
 	npswf32.dll!F2034062961____________________________(env=0x06bbe340, argc=0x06a3e4d8, argv=0x06bbe490)  Line 22381	
 	npswf32.dll!F1097034002__________________________(mname=0x01944b78)  Line 63	
 	npswf32.dll!F_1385117125_____________________(multiname=)  Line 65	
 	npswf32.dll!F_961968953_____________________(multiname=0x01a4ee20)  Line 1009	
 	npswf32.dll!F1113283486___________(env=0x04b167c0, name=0x00000000, slot=0x0012c8d0)  Line 210	
 	npswf32.dll!F_1473839636_______________________(env=0x04b167c0, argc=0x00000000, ap=0x0012c8d0)  Line 202	
 	npswf32.dll!F_741595251_________________(ite=0x0193f5e0, argc=0x00000000, ap=0x0012c8d0, iid=0x00d47e38)  Line 203	
 	npswf32.dll!F2082886063________________(ite=0x06a6df88, argc=0x00000000, ap=0x0012c92c, iid=0x06748000)  Line 216	
 	npswf32.dll!F_1473839636_______________________(env=0x04b85800, argc=0x0012c9c4, ap=0x67cbc9d8)  Line 202	
 	npswf32.dll!F_1628558063_______________________(argc=0x00000001, in=0x0012c9a0, argv=0x04c30698, ms=0x04647f50)  Line 503	
 	npswf32.dll!F335247132_____________________________(env=0x67cbec56, argc=0x04c2f7c0, atomv=0x06a6df70)  Line 487	
 	019c1358()	
 	npswf32.dll!F_413154176_____________________________(env=0x06a6df70, argc=0x00000001, args=0x0012ca90)  Line 6382	
 	npswf32.dll!F_1955641692__________________________(env=0x06a6df70, argc=0x00000001, args=0x0012ca90)  Line 249	
 	npswf32.dll!F_2111908792_________________(argc=0x00000001, argv=0x0012ca90)  Line 162	
 	npswf32.dll!F929940168_____________________________________(event=, core=, currentTargetIsStage=)  Line 1711	
 	npswf32.dll!F_1234708036_______________________________________(event=0x00000002, stack=0x00000000)  Line 1510	
 	npswf32.dll!F271937579___________________________________________(event=0x019c1358)  Line 1227	
 	npswf32.dll!F_1163461272___________________________(env=0x019c1358, argc=0x00000001, argv=0x0012cc7c)  Line 22853	
 	npswf32.dll!F_1047388189____________________(argc=0x00000001, argv=0x0012cc7c)  Line 119	
 	npswf32.dll!F905642122_________________________________(env=0x01a79fe8, ctor=0x00000001, argc=0x0012ccc0, atomv=0x0012ccc0)  Line 153	
 	npswf32.dll!F1228711069__________________(argc=0x01a79fe8, ap=0x00000001, ms=0x0012cdac)  Line 311	
 	npswf32.dll!F_413154176_____________________________(env=0x01a79fe8, argc=0x00000001, args=0x0012cdac)  Line 6382	
 	npswf32.dll!F_1955641692__________________________(env=0x01a79fe8, argc=0x00000001, args=0x0012cdac)  Line 249	
 	npswf32.dll!F_2111908792_________________(argc=0x00000001, argv=0x0012cdac)  Line 162	
 	npswf32.dll!F929940168_____________________________________(event=0x01b7d740, core=0x0413f0d1, currentTargetIsStage=true)  Line 1711	
 	0012ccc0()	
 	npswf32.dll!F_413154176_____________________________(env=0x01a79fe8, argc=0x00000001, args=0x0012cdac)  Line 6382	
 	npswf32.dll!F_1955641692__________________________(env=0x01a79fe8, argc=0x00000001, args=0x0012cdac)  Line 249	
 	npswf32.dll!F_2111908792_________________(argc=0x00000001, argv=0x0012cdac)  Line 162	
 	npswf32.dll!F929940168_____________________________________(event=, core=, currentTargetIsStage=)  Line 1711	
 	npswf32.dll!F_1234708036_______________________________________(event=0x019c1268, stack=0x0012cecc)  Line 1417	
 	npswf32.dll!F271937579___________________________________________(event=0x019c1268)  Line 1227	
 	npswf32.dll!F_1163461272___________________________(env=0x019c1268, argc=0x00000001, argv=0x0012cf80)  Line 22853	
 	npswf32.dll!F_1047388189____________________(argc=0x00000001, argv=0x0012cf80)  Line 119	
 	npswf32.dll!F905642122_________________________________(env=0x06a3cf58, ctor=0x00000001, argc=0x0012d000, atomv=0x00d4c4e6)  Line 153	
 	npswf32.dll!F_1473839636_______________________(env=0x06a3ce50, argc=0x00000001, ap=0x0012d06c)  Line 202	
 	npswf32.dll!F_1473839636_______________________(env=0x06748000, argc=0x03a87ad8, ap=0x0012d114)  Line 202	
 	npswf32.dll!F_1955641692__________________________(env=0x01a79f58, argc=0x00000000, args=0x0012d110)  Line 249	
 	npswf32.dll!F_1473839636_______________________(env=0x06a6df58, argc=0x00000000, ap=0x0012d154)  Line 202	
 	npswf32.dll!F_1473839636_______________________(env=0x01a4dac0, argc=0x00000001, ap=0x0012d190)  Line 202	
 	npswf32.dll!F1228711069__________________(argc=0x00000001, ap=0x0012d190, ms=0x04b1e2e0)  Line 311	
 	npswf32.dll!F335247132_____________________________(env=0x04c1a208, argc=0x00000001, atomv=0x0012cf80)  Line 487	
 	0199c85b()	
 	npswf32.dll!F_1473839636_______________________(env=0x06a3cf58, argc=0x00000001, ap=0x0012d000)  Line 202	
 	npswf32.dll!F2082886063________________(ite=0x06a3ce50, argc=0x00000001, ap=0x0012d06c, iid=0x066fc6d7)  Line 216	
 	npswf32.dll!F_1473839636_______________________(env=0x06748000, argc=0x03a87ad8, ap=0x0012d114)  Line 202	
 	npswf32.dll!F_1955641692__________________________(env=0x01a79f58, argc=0x00000000, args=0x0012d110)  Line 249	
 	npswf32.dll!F_1473839636_______________________(env=0x06a6df58, argc=0x00000000, ap=0x0012d154)  Line 202	
 	npswf32.dll!F_1473839636_______________________(env=0x01a4dac0, argc=0x00000001, ap=0x0012d190)  Line 202	
 	npswf32.dll!F1228711069__________________(argc=0x00000001, ap=0x0012d190, ms=0x04b1e2e0)  Line 311	
 	npswf32.dll!F335247132_____________________________(env=0x01a4dac0, argc=0x00000001, atomv=0x0012d27c)  Line 487	
 	npswf32.dll!F_413154176_____________________________(env=0x01a4dac0, argc=0x00000001, args=0x0012d27c)  Line 6382	
 	npswf32.dll!F_1955641692__________________________(env=0x01a4dac0, argc=0x00000001, args=0x0012d27c)  Line 249	
 	npswf32.dll!F_2111908792_________________(argc=0x00000001, argv=0x0012d27c)  Line 162	
 	npswf32.dll!F929940168_____________________________________(event=0x019c1100, core=0x06748000, currentTargetIsStage=false)  Line 1711	
 	npswf32.dll!F_1234708036_______________________________________(event=0x019c1100, stack=0x0012d39c)  Line 1417	
 	npswf32.dll!F271937579___________________________________________(event=0x019c1100)  Line 1227	
 	npswf32.dll!F_1163461272___________________________(env=0x019c1100, argc=0x00000001, argv=0x0012d458)  Line 22853	
 	npswf32.dll!F_1047388189____________________(argc=0x00000001, argv=0x0012d458)  Line 119	
 	npswf32.dll!F905642122_________________________________(env=0x01a4d9d0, ctor=0x00000000, argc=0x0012d4b4, atomv=0x06748000)  Line 153	
 	npswf32.dll!F_1473839636_______________________(env=0x01a4da90, argc=0x00000001, ap=0x0012d4f0)  Line 202	
 	npswf32.dll!F1228711069__________________(argc=0x00000001, ap=0x0012d4f0, ms=0x04b1e430)  Line 311	
 	npswf32.dll!F335247132_____________________________(env=0x04c1a208, argc=0x00000001, atomv=0x0012d458)  Line 487	
 	01a77ca7()	
 	npswf32.dll!F_1473839636_______________________(env=0x01a4da90, argc=0x00000001, ap=0x0012d4f0)  Line 202	
 	npswf32.dll!F1228711069__________________(argc=0x00000001, ap=0x0012d4f0, ms=0x04b1e430)  Line 311	
 	npswf32.dll!F335247132_____________________________(env=0x01a4da90, argc=0x00000001, atomv=0x0012d5e0)  Line 487	
 	npswf32.dll!F_413154176_____________________________(env=0x01a4da90, argc=0x00000001, args=0x0012d5e0)  Line 6382	
 	npswf32.dll!F_1955641692__________________________(env=0x01a4da90, argc=0x00000001, args=0x0012d5e0)  Line 249	
 	npswf32.dll!F_2111908792_________________(argc=0x00000001, argv=0x0012d5e0)  Line 162	
 	npswf32.dll!F929940168_____________________________________(event=0x06de1fd0, core=0x06748000, currentTargetIsStage=false)  Line 1711	
 	npswf32.dll!F_1234708036_______________________________________(event=0x06de1fd0, stack=0x0012d700)  Line 1417	
 	npswf32.dll!F271937579___________________________________________(event=0x06de1fd0)  Line 1227	
 	npswf32.dll!F_1163461272___________________________(env=0x03a89f50, argc=0x0012d7d4, argv=0x04c6e0eb)  Line 22853	
 	npswf32.dll!F1882488737_______________________()  Line 101	
 	npswf32.dll!F1228711069__________________(argc=0x00000001, ap=0x0012d800, ms=0x06a39238)  Line 311	
 	npswf32.dll!F335247132_____________________________(env=0x04c1a208, argc=0x00000001, atomv=0x0012d7bc)  Line 487	
 	04c6e0eb()	
 	npswf32.dll!F1228711069__________________(argc=0x00000001, ap=0x0012d800, ms=0x06a39238)  Line 311	
 	npswf32.dll!F335247132_____________________________(env=0x01a79ad8, argc=0x00000001, atomv=0x0012d8f4)  Line 487	
 	npswf32.dll!F_413154176_____________________________(env=0x01a79ad8, argc=0x00000001, args=0x0012d8f4)  Line 6382	
 	npswf32.dll!F_1955641692__________________________(env=0x01a79ad8, argc=0x00000001, args=0x0012d8f4)  Line 249	
 	npswf32.dll!F_2111908792_________________(argc=0x00000001, argv=0x0012d8f4)  Line 162	
 	npswf32.dll!F929940168_____________________________________(event=0x06a23bd8, core=0x06748000, currentTargetIsStage=false)  Line 1711	
 	npswf32.dll!F_1234708036_______________________________________(event=0x06a23bd8, stack=0x0012da14)  Line 1417	
 	npswf32.dll!F271937579___________________________________________(event=0x06a23bd8)  Line 1227	
 	npswf32.dll!F_1163461272___________________________(env=0x06bceb38, argc=0x00000005, argv=0x0012db04)  Line 22853	
 	npswf32.dll!F_1473839636_______________________(env=0x035fb192, argc=0x06a24900, ap=0x0012dc14)  Line 202	
 	npswf32.dll!F_1001905598_________________(name=0x47a1ba88)  Line 117	
 	npswf32.dll!F_476147114_________________()  Line 84	
 	ntdll.dll!_ZwQueryPerformanceCounter@8() 	
 	npswf32.dll!F_2111908792_________________(argc=0x00000001, argv=0x0012dc14)  Line 162	
 	npswf32.dll!F929940168_____________________________________(event=0x035fc2b0, core=0x06bcebb0, currentTargetIsStage=true)  Line 1711	
 	00000004()	
 	npswf32.dll!F_1001905598_________________(name=0x47a1ba88)  Line 117	
 	npswf32.dll!F_476147114_________________()  Line 84	
 	ntdll.dll!_ZwQueryPerformanceCounter@8() 	
 	npswf32.dll!F_2111908792_________________(argc=0x00000001, argv=0x0012dc14)  Line 162	
 	npswf32.dll!F929940168_____________________________________(event=, core=, currentTargetIsStage=)  Line 1711	
 	npswf32.dll!F_1234708036_______________________________________(event=0x06a23e08, stack=0x0012dd28)  Line 1417	
 	npswf32.dll!F_1234708036_______________________________________(event=0x06a23e08, stack=0x0012dd28, preventDefault=false)  Line 1377	
 	npswf32.dll!F2136329884____________________________________________________(stack=0x0012dd28, preventDefault=false, clsId=0x0000013e, fmt=0x67dc1a58, ...)  Line 1334	
 	npswf32.dll!F_1823564447________________________________________(type=0x035fb190, bubbles=false, cancelable=false, code=0x000007f0, url=0x06bc7e68)  Line 3856	
 	npswf32.dll!F_497257586______________________()  Line 1212	
 	npswf32.dll!F_1017954872_________________________()  Line 532	
 	npswf32.dll!NPP_URLNotify(instance=0x009451ac, url=0x0091f338, reason=0x0001, notifyData=0x0000000d)  Line 1817	
 	xul.dll!mozilla::plugins::StreamNotifyChild::NPP_URLNotify(reason=0x0001)  Line 1877	C++
 	xul.dll!mozilla::plugins::StreamNotifyChild::Recv__delete__(reason=0x0001)  Line 1867	C++
 	xul.dll!mozilla::plugins::PStreamNotifyChild::OnMessageReceived(__msg={...})  Line 87	C++
 	xul.dll!mozilla::plugins::PPluginModuleChild::OnMessageReceived(__msg={...})  Line 430	C++
 	xul.dll!mozilla::ipc::AsyncChannel::OnDispatchMessage(msg={...})  Line 262	C++
 	xul.dll!mozilla::ipc::RPCChannel::Call(msg=0x009531e0, reply=0x0012dff4)  Line 246	C++
 	xul.dll!mozilla::plugins::PPluginScriptableObjectChild::CallNPN_Evaluate(aScript={...}, aResult=0x0012e04c, aSuccess=0x0012e037)  Line 124	C++
 	xul.dll!mozilla::plugins::PluginScriptableObjectChild::Evaluate(aScript=0x0012e0c0, aResult=0x0012e0b0)  Line 1093	C++
>	xul.dll!mozilla::plugins::child::_evaluate(aNPP=0x0094526c, aObject=0x009167e0, aScript=0x0012e0c0, aResult=0x0012e0b0)  Line 1244	C++
 	npswf32.dll!F956976105____________(expression=0x0695bd30)  Line 1290	
 	npswf32.dll!F400962363________________(splayer=0x035f2000, pCallerCxt=0x068da380, expression=0x0695bd30, checker={...})  Line 463	
 	npswf32.dll!F_395357591_____________________________(expression=0x046c6e68)  Line 129	
 	npswf32.dll!F2034062961____________________________(env=0x046c6e80, argc=0x03b274c0, argv=0x046c6fa0)  Line 22381	
 	npswf32.dll!F1097034002__________________________(mname=0x066e1d70)  Line 63	
 	npswf32.dll!F_1385117125_____________________(multiname=)  Line 65	
 	npswf32.dll!F_961968953_____________________(multiname=0x0674da30)  Line 1009	
 	npswf32.dll!F1113283486___________(env=0x03d56000, name=0x0012ec20, slot=0x0012e334)  Line 210	
 	npswf32.dll!F_1955641692__________________________(env=0x06783838, argc=0x00000000, args=0x0012e32c)  Line 249	
 	npswf32.dll!F_1473839636_______________________(env=0x01b76f88, argc=0x00000000, ap=0x0012e390)  Line 202	
 	npswf32.dll!F_1473839636_______________________(env=0x01b767d8, argc=0x00000000, ap=0x0012e3e0)  Line 202	
 	npswf32.dll!F1228711069__________________(argc=0x00000000, ap=0x0012e3e0, ms=0x046cfd60)  Line 311	
 	npswf32.dll!F335247132_____________________________(env=0x04b03510, argc=0x00000000, atomv=0x00000000)  Line 487	
 	8003ea01()	

What's interesting here is that stream-delete/NPP_URLNotify (which are async messages) is nesting in the outer NPN_Evaluate. I suspect the RPC stacks are mismatched here causing the PPluginScriptableObjectParent::CallHasProperty message to not be delivered properly. I'd really like to look all the way up the stack here, too, since it stops before we get to main().

Comment 12

[Benjamin Smedberg]

The "outer" evaluate is evaluating this:

try { __flash__toXML(function(){ return document.location.href.toString(); }()) ; } catch (e) { "<undefined/>"; }

Comment 13

[Benjamin Smedberg]

So to summarize, the RPC stacks are:

Parent:
* receives NPN_Evaluate #2
* calls HasProperty

Child:
* calls NPN_Evaluate #1
* receives async streamnotifychild__delete__ (NPP_URLNotify)
* calls NPN_Evaluate #2
=stuck, incoming HasProperty never delivered

At this point, the child RPCChannel has the following state:

-		(mozilla::ipc::RPCChannel*) 0x00902820	0x00902820 {mPending=[0x00000000]() mStack=[0x00000002]({name_=0x691fe5dc "PPluginScriptableObject::Msg_NPN_Evaluate" },{name_=0x691fe5dc "PPluginScriptableObject::Msg_NPN_Evaluate" }) mOutOfTurnReplies=[0x00000001]((0xfffffdc4,{name_=0x691d2490 "???" })) ...}	mozilla::ipc::RPCChannel *
+		mozilla::ipc::SyncChannel	{kNoTimeout=0x80000000 mTopFrame=0x0012c5ac sStaticTopFrame=0x0012c5ac ...}	mozilla::ipc::SyncChannel
		mPending	[0x00000000]()	std::queue<IPC::Message,std::deque<IPC::Message,std::allocator<IPC::Message> > >
-		mStack	[0x00000002]({name_=0x691fe5dc "PPluginScriptableObject::Msg_NPN_Evaluate" },{name_=0x691fe5dc "PPluginScriptableObject::Msg_NPN_Evaluate" })	std::stack<IPC::Message,std::deque<IPC::Message,std::allocator<IPC::Message> > >
+		[0]	{name_=0x691fe5dc "PPluginScriptableObject::Msg_NPN_Evaluate" }	IPC::Message
+		[1]	{name_=0x691fe5dc "PPluginScriptableObject::Msg_NPN_Evaluate" }	IPC::Message
-		mOutOfTurnReplies	[0x00000001]((0xfffffdc4,{name_=0x691d2490 "???" }))	std::map<unsigned int,IPC::Message,std::less<unsigned int>,std::allocator<std::pair<unsigned int const ,IPC::Message> > >
+		[0]	(0xfffffdc4,{name_=0x691d2490 "???" })	std::pair<unsigned int const ,IPC::Message>
-		mDeferred	[0x00000001]({name_=0x691d2490 "???" })	std::stack<IPC::Message,std::deque<IPC::Message,std::allocator<IPC::Message> > >
+		[0]	{name_=0x691d2490 "???" }	IPC::Message
		mRemoteStackDepthGuess	0x00000001	unsigned int
		mBlockedOnParent	false	bool
-		mCxxStackFrames	[0x00000003]({mDirection=OUT_MESSAGE mMsg=0x0012df44 },{mDirection=IN_MESSAGE mMsg=0x0012df5c },{mDirection=OUT_MESSAGE mMsg=0x0012c57c })	std::vector<mozilla::ipc::RPCChannel::RPCFrame,std::allocator<mozilla::ipc::RPCChannel::RPCFrame> >
+		[0]	{mDirection=OUT_MESSAGE mMsg=0x0012df44 }	mozilla::ipc::RPCChannel::RPCFrame
+		[1]	{mDirection=IN_MESSAGE mMsg=0x0012df5c }	mozilla::ipc::RPCChannel::RPCFrame
+		[2]	{mDirection=OUT_MESSAGE mMsg=0x0012c57c }	mozilla::ipc::RPCChannel::RPCFrame
		mSawRPCOutMsg	true	bool

I'm pretty sure that the NPN_Evaluate #1 has already been fully processed, and that's the mOutOfTurnReplies. I'm not sure why there's something in mDeferred, so I suspect we're deferring a message we shouldn't be. More to come.

Comment 14

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Attachment: Test

This test abstracts the ops bsmedberg reports, seems to trigger the same bug.  Will investigate after dinner.

Comment 15

[Benjamin Smedberg]

In terms of a workaround: I expect the easiest thing to do is avoid calling NPN_Evaluate from the stream completion event (NPP_URLNotify). The stream in question is http://entitlement.auth.adobe.com/adobe-services/trackBrowserSession

Probably the easiest thing to do here is do whatever you would have done the next time around the event loop (setTimeout(0) in browser-speak, I'm sure AS has something equivalent).

Comment 16

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Attachment: Out-of-turn RPC replies should be taken into consideration when checking for races, or else we can defer in-calls when we shouldn't

Comment 17

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Attachment: Test

Comment 18

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

http://hg.mozilla.org/mozilla-central/rev/fdf5a45c0af8
http://hg.mozilla.org/mozilla-central/rev/7581f4b695a6

Comment 19

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Comment on attachment 470977 [details] [diff] [review]
Out-of-turn RPC replies should be taken into consideration when checking for races, or else we can defer in-calls when we shouldn't

This is going to cause our hang detector to kick in spuriously.  Would be nice to have.

Comment 20

[Charles]

Great news Chris thank you!  When will this land in trunk and does anyone here have an idea on the next patch update will be?

Comment 21

[Benjamin Smedberg]

3.6.10 is the earliest release in which this will be fixed, so probably 4-8 weeks. clegnitto can speak to the exact schedule.

Comment 22

[j.j.]

> When will this land in trunk and does anyone here
> have an idea on the next patch update will be?

Just landed on trunk. Follow the latest meeting notes to be up to date with 3.6.10 release.

 https://wiki.mozilla.org/Platform#Meetings

Comment 23

[Adrian Silaghi]

(In reply to comment #15)
> In terms of a workaround: I expect the easiest thing to do is avoid calling
> NPN_Evaluate from the stream completion event (NPP_URLNotify). The stream in
> question is
> http://entitlement.auth.adobe.com/adobe-services/trackBrowserSession
> 
> Probably the easiest thing to do here is do whatever you would have done the
> next time around the event loop (setTimeout(0) in browser-speak, I'm sure AS
> has something equivalent).

In AS we were already redispatching the event multiple times, untill we ended up calling something via ExternalInterface. So somewhere in the event chain I have put a 50ms timer, and redispatch the event on the timer event. Since then we have not been able to reproduce the crash with the original reported url ! Seems that introducing a very short timer in AS ensures we broke out of the current event loop and deffer processing and we avoid the ou-of-turn RPC replies.

Comment 24

[Daniel Veditz [:dveditz]]

Comment on attachment 470977 [details] [diff] [review]
Out-of-turn RPC replies should be taken into consideration when checking for races, or else we can defer in-calls when we shouldn't

Approved for 1.9.2.10, a=dveditz for release-drivers

Comment 25

[Daniel Veditz [:dveditz]]

Comment on attachment 470978 [details] [diff] [review]
Test

Please land the test on the branch, too.

Comment 26

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/7256bd3d3aa0
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/e083b305df0c

Note that the IPDL C++ unit tests have been broken on 1.9.2 for apparently a "long time".  I have a patch to get them building, will file a bug.

Comment 27

[Al Billings [:abillings - ex-MoCo]]

(In reply to comment #25)
> Comment on attachment 470978 [details] [diff] [review]
> Test
> 
> Please land the test on the branch, too.

Is this test running and passing on 1.9.2 enough to say that this is fixed really? Chris' comment muddies the waters here but I see it running in the logs.

Comment 28

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Hm, it shouldn't be running, I never got approval to land the test that fixes the tests.  The test for this bug is wholly deterministic, fails before this patch, and passes after along with all the other IPDL tests (with the patch to get the tests running also applied).  I'm as confident as I can be that this bug is fixed.

Comment 29

[Al Billings [:abillings - ex-MoCo]]

Can we get approval for you to land the fix for the tests?