858900 - crash in doInvoke @ XPCCallContext::Init

Status: RESOLVED WORKSFORME

(Core Graveyard :: Plug-ins, defect, P2)

Description

[Scoobidiver (away)]

I don't know if it's an XPConnect or plugins bug.

Comments talk about Java (Pogo), Silverlight (Sharepoint Onlines, Netflix), Webex (Vinsolutions).

Signature 	XPCCallContext::Init(XPCContext::LangType, int, JSObject*, JSObject*, XPCCallContext::WrapperInitOptions, long, unsigned int, JS::Value*, JS::Value*) More Reports Search
UUID	e84dfd65-9c33-40a2-9353-c435d2130406
Date Processed	2013-04-06 03:17:50
Uptime	8162
Install Age	3.2 days since version was first installed.
Install Time	2013-04-02 23:40:10
Product	Firefox
Version	20.0
Build ID	20130326150557
Release Channel	release
OS	Mac OS X
OS Version	10.7.5 11G63
Build Architecture	amd64
Build Architecture Info	family 6 model 42 stepping 7
Crash Reason	EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
Crash Address	0x28
App Notes 	
AdapterVendorID: 0x1002, AdapterDeviceID: 0x6740GL Context? GL Context+ GL Layers? GL Layers+ 
Processor Notes 	sp-processor03.phx1.mozilla.com_17102:2008; exploitablity tool: ERROR: unable to analyze dump
EMCheckCompatibility	True
Adapter Vendor ID	0x1002
Adapter Device ID	0x6740

Frame 	Module 	Signature 	Source
0 	XUL 	XPCCallContext::Init 	js/xpconnect/src/xpcprivate.h:1032
1 	XUL 	XPCCallContext::XPCCallContext 	js/xpconnect/src/XPCCallContext.cpp:31
2 	XUL 	XPC_WN_Helper_NewResolve 	js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1054
3 	XUL 	js::baseops::GetProperty 	js/src/jsobj.cpp:3077
4 	XUL 	js::Interpret 	js/src/jsobjinlines.h:174
5 	XUL 	js::RunScript 	js/src/jsinterp.cpp:346
6 	XUL 	js::InvokeKernel 	js/src/jsinterp.cpp:404
7 	XUL 	js_fun_call 	js/src/jsinterp.h:112
8 	XUL 	js::InvokeKernel 	js/src/jscntxtinlines.h:373
9 	XUL 	js::Interpret 	js/src/jsinterp.cpp:2366
10 	XUL 	js::RunScript 	js/src/jsinterp.cpp:346
11 	XUL 	js::InvokeKernel 	js/src/jsinterp.cpp:404
12 	XUL 	js::Invoke 	js/src/jsinterp.h:112
13 	XUL 	JS_CallFunctionValue 	js/src/jsapi.cpp:5817
14 	XUL 	doInvoke 	dom/plugins/base/nsJSNPRuntime.cpp:704
15 	XUL 	nsJSObjWrapper::NP_InvokeDefault 	dom/plugins/base/nsJSNPRuntime.cpp:738
16 	XUL 	mozilla::plugins::parent::_invokeDefault 	dom/plugins/base/nsNPAPIPlugin.cpp:1505
17 	XUL 	mozilla::plugins::PluginScriptableObjectParent::AnswerInvokeDefault 	dom/plugins/ipc/PluginScriptableObjectParent.cpp:817
18 	XUL 	mozilla::plugins::PPluginScriptableObjectParent::OnCallReceived 	obj-firefox/x86_64/ipc/ipdl/PPluginScriptableObjectParent.cpp:896
19 	XUL 	mozilla::plugins::PPluginModuleParent::OnCallReceived 	obj-firefox/x86_64/ipc/ipdl/PPluginModuleParent.cpp:1288
20 	XUL 	mozilla::ipc::RPCChannel::DispatchIncall 	ipc/glue/RPCChannel.cpp:486
21 	XUL 	mozilla::ipc::RPCChannel::OnMaybeDequeueOne 	ipc/glue/RPCChannel.cpp:398
22 	XUL 	RunnableMethod<mozilla::ipc::RPCChannel, bool 	ipc/chromium/src/base/tuple.h:383
23 	XUL 	MessageLoop::DeferOrRunPendingTask 	ipc/chromium/src/base/message_loop.cc:333
24 	XUL 	MessageLoop::DoWork 	ipc/chromium/src/base/message_loop.cc:441
25 	XUL 	mozilla::ipc::DoWorkRunnable::Run 	ipc/glue/MessagePump.cpp:42
26 	XUL 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:627
27 	XUL 	NS_ProcessPendingEvents_P 	obj-firefox/x86_64/xpcom/build/nsThreadUtils.cpp:188
28 	XUL 	nsBaseAppShell::NativeEventCallback 	widget/xpwidgets/nsBaseAppShell.cpp:97
29 	XUL 	nsAppShell::ProcessGeckoEvents 	widget/cocoa/nsAppShell.mm:387
30 	CoreFoundation 	CoreFoundation@0x124f1 	
31 	CoreFoundation 	CoreFoundation@0x11d5d 	
32 	CoreFoundation 	CoreFoundation@0x38b49 	
33 	libsystem_c.dylib 	libsystem_c.dylib@0x4d160 	

More reports at:
https://crash-stats.mozilla.com/report/list?signature=XPCCallContext%3A%3AInit%28XPCContext%3A%3ALangType%2C+int%2C+JSObject*%2C+JSObject*%2C+XPCCallContext%3A%3AWrapperInitOptions%2C+long%2C+unsigned+int%2C+JS%3A%3AValue*%2C+JS%3A%3AValue*%29

Comment 1

[Benjamin Smedberg]

Is there a nightly regression range?

This is a GCC signature, and I expect that there will be a different matching signature on Windows, probably the signature for bug 622280. But almost none of those appear to be from plugins; and most (not all) of these do appear to be from plugins.

For one that's not, see https://crash-stats.mozilla.com/report/index/556fe793-76e3-470d-b0d8-1cd2d2130405 where nsDocLoader::FireOnLocationChange appears to be calling a wrappedjs.

Note that when plugins are involved, this is a function being invoked by plugins at the top of the stack, so there is no prior script on the stack.

bholley, what are the conditions where http://hg.mozilla.org/releases/mozilla-release/annotate/c90d44bfa96c/js/xpconnect/src/XPCCallContext.cpp#l116 could fail? I'm pretty sure we're crashing with a null mXPCContext at http://hg.mozilla.org/releases/mozilla-release/annotate/c90d44bfa96c/js/xpconnect/src/XPCCallContext.cpp#l117

Comment 2

[Scoobidiver (away)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> Is there a nightly regression range?
It first showed up in 19.0a2/20121214 and 20.0a1/20121217 but is discontinuous across builds.

Comment 3

[Bobby Holley (:bholley)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> bholley, what are the conditions where
> http://hg.mozilla.org/releases/mozilla-release/annotate/c90d44bfa96c/js/
> xpconnect/src/XPCCallContext.cpp#l116 could fail?

So, XPCContext::GetXPCContext just returns JS_GetSecondContextPrivate. So the real issue here is that the associated private in the JSContext is null. This gets set up via a callback (ContextCallback) on the JSRuntime in the XPCJSRuntime constructor, immediately after we create the JSRuntime. So one of three things could be happening:

1 - This JSContext belongs to a different JSRuntime without the callback (i.e. workers).

2 - This JSContext was hit by memory corruption.

3 - This JSContext was destroyed, which caused JSCONTEXT_DESTROY to be passed to ContextCallback, which deletes the associated XPCContext.

My money is on (3). I've always been skeptical that we're guaranteed to wait for a JSContext to fall entirely out of use before destroying it. In particular, the JSContext destruction happens in XPConnect, which walks up the list of XPCCallContexts to see if the JSContext is still being used anywhere. But there are increasingly many cases these days (well, since QuickStubs, really) where we don't create an XPCCallContext at all. So it seems like we should really tie lifetime to the context stack instead. But given that I'm trying to get rid of JSContexts entirely, I was always a little reticent to dig in too deep.

As for moving forward on this bug, there are two things we can do:
* Add diagnostics to poison the JS private when the context is destroyed to confirm the hypothesis that we're hitting (3).
* Try to overhaul JSContext lifetime management to be more sane.

What do you think, benjamin?

Comment 4

[Benjamin Smedberg]

Attachment: Poison JSContext

Poison like so?

I really can't speak to JSContext lifetime, but perhaps we should at least make sure that there is an XPCCallContext for the plugin call in question here? i.e. nsJSNPRuntime.cpp:doInvoke http://hg.mozilla.org/mozilla-central/annotate/9d5f05a6d497/dom/plugins/base/nsJSNPRuntime.cpp#l631 should have a synthetic XPCCallContext or something like it, just to make sure that the JSContext tracking is correct?

Comment 5

[Bobby Holley (:bholley)]

Comment on attachment 735368 [details] [diff] [review]
Poison JSContext

This would work, sure - though I don't think I can technically review it. I was going to suggest just setting the private to 0xdeadbeef in the XPCContext destructor, but this is probably better and more likely to catch other problems.

Comment 6

[Bobby Holley (:bholley)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #4)

> I really can't speak to JSContext lifetime, but perhaps we should at least
> make sure that there is an XPCCallContext for the plugin call in question
> here? i.e. nsJSNPRuntime.cpp:doInvoke
> http://hg.mozilla.org/mozilla-central/annotate/9d5f05a6d497/dom/plugins/base/
> nsJSNPRuntime.cpp#l631 should have a synthetic XPCCallContext or something
> like it, just to make sure that the JSContext tracking is correct?

Yeah, I mean, that's broken as the code stands, but there are a bunch of other things that are broken as well. Let me cook up a patch to scan the JSContext instead of the XPCCallContext stack and see if anything explodes.

Comment 7

[Luke Wagner [:luke]]

Comment on attachment 735368 [details] [diff] [review]
Poison JSContext

Neat

Comment 8

[Bobby Holley (:bholley)]

Let's get the diagnostics patch checked in, while I simultaneously work on bug 860085.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7fc45d104920

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/7fc45d104920

Comment 11

[Benjamin Smedberg]

scoobidiver, did you see other signatures replace this one after this landed? I'm very interested in getting better stacks out of it.

Comment 12

[Scoobidiver (away)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #11)
> scoobidiver, did you see other signatures replace this one after this
> landed? 
There's only one crash in the trunk. So if the signature has morphed it's impossible to detect it. You'll need to uplift the patch to Beta to check (one crash also in Aurora) and in addition it's not guaranteed (only 3 crashes in 21.0b1).