Intermittent PROCESS-CRASH application crashed [@ XPCWrappedNative::GetNewOrUsed(xpcObjectHelper&, XPCWrappedNativeScope*, XPCNativeInterface*, XPCWrappedNative**)][@ js::GetGlobalForObjectCrossCompartment(JSObject*)]

RESOLVED DUPLICATE of bug 986843

Status

()

Core
XPConnect
RESOLVED DUPLICATE of bug 986843
4 years ago
4 years ago

People

(Reporter: KWierso, Assigned: Bobby Holley (parental leave - send mail for anything urgent))

Tracking

({crash, intermittent-failure})

Trunk
mozilla30
All
Android
crash, intermittent-failure
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [leave open], crash signature)

Attachments

(2 attachments)

(Reporter)

Description

4 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=33126554&tree=Fx-Team
slave: panda-0328



14:32:43     INFO -  REFTEST TEST-START | http://10.12.130.22:30328/jsreftest/tests/jsreftest.html?test=ecma/Math/15.8.2.2.js
14:32:43     INFO -  REFTEST TEST-LOAD | http://10.12.130.22:30328/jsreftest/tests/jsreftest.html?test=ecma/Math/15.8.2.2.js | 443 / 2106 (21%)
14:32:43     INFO -  INFO | automation.py | Application ran for: 0:05:10.829212
14:32:43     INFO -  INFO | zombiecheck | Reading PID log: /tmp/tmpSb2g3fpidlog
14:32:43     INFO -  /data/anr/traces.txt not found
14:32:44     INFO -  mozcrash INFO | Downloading symbols from: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/fx-team-android/1389906679/fennec-29.0a1.en-US.android-arm.crashreporter-symbols.zip
14:33:31  WARNING -  PROCESS-CRASH | http://10.12.130.22:30328/jsreftest/tests/jsreftest.html?test=ecma/Math/15.8.2.2.js | application crashed [@ XPCWrappedNative::GetNewOrUsed(xpcObjectHelper&, XPCWrappedNativeScope*, XPCNativeInterface*, XPCWrappedNative**)]
14:33:31     INFO -  Crash dump filename: /tmp/tmps2aIT8/6fcbbac9-6bbe-5ca9-2ba6b063-21061672.dmp
14:33:31     INFO -  Operating system: Android
14:33:31     INFO -                    0.0.0 Linux 3.2.0+ #2 SMP PREEMPT Thu Nov 29 08:06:57 EST 2012 armv7l pandaboard/pandaboard/pandaboard:4.0.4/IMM76I/5:eng/test-keys
14:33:31     INFO -  CPU: arm
14:33:31     INFO -       2 CPUs
14:33:31     INFO -  Crash reason:  SIGILL
14:33:31     INFO -  Crash address: 0x63191256
14:33:31     INFO -  Thread 13 (crashed)
14:33:31     INFO -   0  libxul.so!XPCWrappedNative::GetNewOrUsed(xpcObjectHelper&, XPCWrappedNativeScope*, XPCNativeInterface*, XPCWrappedNative**) [nsAutoPtr.h:52058b81ff1f : 906 + 0x6]
14:33:31     INFO -       r4 = 0x6bf8e744    r5 = 0x5d3e9448    r6 = 0x6ae862c0    r7 = 0x5bdd4a10
14:33:31     INFO -       r8 = 0x6bf8e744    r9 = 0x00000001   r10 = 0x5d3e9488    fp = 0x6ae862c0
14:33:31     INFO -       sp = 0x5d3e9378    lr = 0x62c187fb    pc = 0x63191256
14:33:31     INFO -      Found by: given as instruction pointer in context
The fact that Android's crashing here in "any damn test at all" makes me think xpconnect (or, xpconnect should blame Fennec, either one) rather than the owner of the particular test.
Component: JavaScript Engine → XPConnect
Version: unspecified → Trunk
Comment hidden (Treeherder Robot)
Duplicate of this bug: 969734
Duplicate of this bug: 969729
Comment hidden (Treeherder Robot)
Of the four crashes I looked at, all but one (the one in comment 0 of this bug) are crashing at 0x14, so it looks like there's a null deref at this line:
  http://mxr.mozilla.org/mozilla-central/source/js/xpconnect/src/XPCWrappedNative.cpp#378

Well, it looks like we're making a recursive call back into GNOU, then hitting a null deref at this line:
  302     Native2WrappedNativeMap* map = Scope->GetWrappedNativeMap();

So I guess GetObjectScope of parent is returning null, then because it is different than scope, we recurse and crash.  I'm not sure how that should be fixed, though.
Summary: Intermittent PROCESS-CRASH | jsreftest.html?test=ecma/Math/15.8.2.2.js | application crashed [@ XPCWrappedNative::GetNewOrUsed(xpcObjectHelper&, XPCWrappedNativeScope*, XPCNativeInterface*, XPCWrappedNative**)] → Intermittent PROCESS-CRASH application crashed [@ XPCWrappedNative::GetNewOrUsed(xpcObjectHelper&, XPCWrappedNativeScope*, XPCNativeInterface*, XPCWrappedNative**)]
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(In reply to Andrew McCreight [:mccr8] from comment #6)
> Of the four crashes I looked at, all but one (the one in comment 0 of this
> bug) are crashing at 0x14, so it looks like there's a null deref at this
> line:
>  
> http://mxr.mozilla.org/mozilla-central/source/js/xpconnect/src/
> XPCWrappedNative.cpp#378
> 
> Well, it looks like we're making a recursive call back into GNOU, then
> hitting a null deref at this line:
>   302     Native2WrappedNativeMap* map = Scope->GetWrappedNativeMap();
> 
> So I guess GetObjectScope of parent is returning null, then because it is
> different than scope, we recurse and crash.  I'm not sure how that should be
> fixed, though.

Any ideas, Bobby? :)
Flags: needinfo?(bobbyholley)
Comment hidden (Treeherder Robot)
https://tbpl.mozilla.org/php/getParsedLog.php?id=34432605&tree=Mozilla-Inbound
Hm, that's pretty bad. GetObjectScope should more or less never return null modulo stuff with old-school JSD machinery, which I doubt is getting invoked on b2g here. I'm worried that someone is passing in an object from a different thread or something. :-(
Flags: needinfo?(bobbyholley)
Armv6 Fennec, rather than b2g.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(In reply to Bobby Holley (:bholley) from comment #18)
> Hm, that's pretty bad. GetObjectScope should more or less never return null
> modulo stuff with old-school JSD machinery, which I doubt is getting invoked
> on b2g here. I'm worried that someone is passing in an object from a
> different thread or something. :-(

Who was this comment directed at?
Comment hidden (Treeherder Robot)
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #46)
> (In reply to Bobby Holley (:bholley) from comment #18)
> > Hm, that's pretty bad. GetObjectScope should more or less never return null
> > modulo stuff with old-school JSD machinery, which I doubt is getting invoked
> > on b2g here. I'm worried that someone is passing in an object from a
> > different thread or something. :-(
> 
> Who was this comment directed at?

The void. This isn't something I can figure out by inspection.
This is happening too frequently to leave hanging. Any suggestions for who can help?
I can try landing some diagnostics. Moment.
Created attachment 8375703 [details] [diff] [review]
Diagnostics. v1
Attachment #8375703 - Flags: review?(continuation)
Comment on attachment 8375703 [details] [diff] [review]
Diagnostics. v1

Review of attachment 8375703 [details] [diff] [review]:
-----------------------------------------------------------------

I guess spamming the log in opt builds right before we crash is okay.
Attachment #8375703 - Flags: review?(continuation) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/1cad5f5cc647
Comment hidden (Treeherder Robot)
(Reporter)

Comment 55

4 years ago
https://hg.mozilla.org/mozilla-central/rev/1cad5f5cc647
Assignee: nobody → bobbyholley
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla30
Comment hidden (Treeherder Robot)
(Reporter)

Comment 57

4 years ago
I doubt the diagnostics fixed this.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
And morphing:
https://tbpl.mozilla.org/php/getParsedLog.php?id=34852023&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=34856253&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=34847298&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=34861692&tree=Mozilla-Inbound
Summary: Intermittent PROCESS-CRASH application crashed [@ XPCWrappedNative::GetNewOrUsed(xpcObjectHelper&, XPCWrappedNativeScope*, XPCNativeInterface*, XPCWrappedNative**)] → Intermittent PROCESS-CRASH application crashed [@ XPCWrappedNative::GetNewOrUsed(xpcObjectHelper&, XPCWrappedNativeScope*, XPCNativeInterface*, XPCWrappedNative**)][@ js::GetGlobalForObjectCrossCompartment(JSObject*)]
https://tbpl.mozilla.org/php/getParsedLog.php?id=34864320&tree=Mozilla-Central

Do the new stacks help, Bobby? :)

Also CCing the other sheriffs to be aware of how the stacks are morphing on this now so we don't end up filing more dupes.
Flags: needinfo?(bobbyholley)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
https://tbpl.mozilla.org/php/getParsedLog.php?id=34914436&tree=Mozilla-Inbound
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Hm. So the android logs don't appear to contain my printfs. But the fact that we're sometimes (and sometimes not) crashing when trying to invoke:

js::GetObjectClass(js::GetGlobalForObjectCrossCompartment(parent))

Implies that |parent| probably points to garbage.

It would be really helpful to know which PreCreate hook we're dealing with here, but it's tricky without the printfs. Ryan, do you have any idea why the printfs here aren't showing up in the log?

http://mxr.mozilla.org/mozilla-central/source/js/xpconnect/src/XPCWrappedNative.cpp#380
Flags: needinfo?(bobbyholley) → needinfo?(ryanvm)
Geoff's the guy you want for Android harness questions.
Flags: needinfo?(ryanvm) → needinfo?(gbrown)
You just need to change "printf" to "printf_stderr". 

Output will appear in the logcat output, in the test log.
Flags: needinfo?(gbrown)
Created attachment 8378704 [details] [diff] [review]
More diagnostics. v1
Attachment #8378704 - Flags: review?(continuation)
Comment on attachment 8378704 [details] [diff] [review]
More diagnostics. v1

Review of attachment 8378704 [details] [diff] [review]:
-----------------------------------------------------------------

::: js/xpconnect/src/XPCWrappedNative.cpp
@@ +382,5 @@
> +                printf_stderr("IsMainThread: %u\n", (uint32_t) NS_IsMainThread());
> +                char* className = nullptr;
> +                sciWrapper.GetCallback()->GetClassName(&className);
> +                printf_stderr("SH Class Name: %s\n", className);
> +                nsMemory::Free(className);

probably not too important to free the memory right before a crash but okay. ;)

@@ +383,5 @@
> +                char* className = nullptr;
> +                sciWrapper.GetCallback()->GetClassName(&className);
> +                printf_stderr("SH Class Name: %s\n", className);
> +                nsMemory::Free(className);
> +                printf_stderr("IsMainThread: %u\n", (uint32_t) NS_IsMainThread());

You don't need to print IsMainThread twice, do you?
Attachment #8378704 - Flags: review?(continuation) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/c1c6b4f9e19c
Whiteboard: [leave open]
https://hg.mozilla.org/mozilla-central/rev/c1c6b4f9e19c
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(In reply to TBPL Robot from comment #80)

New diagnostics are working now:

02-20 11:46:00.558 I/Gecko   ( 1727): Uh oh, hit an object without a scope! Crashing shortly.
02-20 11:46:00.558 I/Gecko   ( 1727): IsMainThread: 1
02-20 11:46:00.558 I/Gecko   ( 1727): SH Class Name: Window
02-20 11:46:00.558 I/Gecko   ( 1727): plannedParent object class: ChromeWindow
02-20 11:46:00.558 I/Gecko   ( 1727): plannedParent Global class: ChromeWindow
02-20 11:46:00.558 I/Gecko   ( 1727): parent Object class: Window
02-20 11:46:01.510 I/DEBUG   (  937): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
02-20 11:46:01.510 I/DEBUG   (  937): Build fingerprint: 'nvidia/harmony/harmony/harmony:2.2/FRF91/20110202.102810:eng/test-keys'
02-20 11:46:01.510 I/DEBUG   (  937): pid: 1727, tid: 1743  >>> org.mozilla.fennec <<<
02-20 11:46:01.510 I/DEBUG   (  937): signal 11 (SIGSEGV), fault addr 00000000
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
I am ruminating on this, FWIW.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
So, it seems pretty clear that we're trying to get an XPCWN for an inner nsGlobalWindow whose mJSObject points to memory that has been partially trashed.

We have two sets of crashes. This analysis assumes that they're the same situation, and that the symptoms vary with the contents of the garbage memory.

The logs show us calling into nsWindowSH::PreCreate, which just stashes mJSObject into the outparam.

After this, we call GetObjectScope, which returns obj->compartment()->data->scope. In half the cases, this is null, and we print the logging and explicitly crash. In the other half, it's non-null, but probably garbage, since we appear to crash shortly thereafter trying to dereference it with GetWrappedNativeMap.

In the first case (and probably in the second), the JSObject returned from PreCreate has an intact JSClass whose ->name is "Window", suggesting that this JSObject* once did point to a valid global. But calling js::GetGlobalForObjectCrossCompartment (which effectively does *obj->compartment()->global_) returns a null ref.

So this all suggests that the global and compartment have been recently collected. If that were true, the finalize hook _should_ have caused us to null out mJSObject. This leaves 3 possibilities:
(1) The global was finalized, but the JS engine didn't properly trigger the finalize hook.
(2) The global was finalized, but the XPConnect finalize hook didn't manage to get all the way to nulling out mJSObject on the global.
(3) The global was not finalized (and the analysis above is wrong).

We could answer (2) by adding some release-mode assertions along the finalize path. I could add that kind of patch, if we thought it would be useful.

We could answer (1) vs (3) by logging, but we'd need some way to trigger this on a try push, since we couldn't do that kind of logging in the opt builds we ship. Ryan, do you have any thoughts on how to trigger this reliably on a try build?

Andrew, any thoughts on all of this?
Flags: needinfo?(ryanvm)
Flags: needinfo?(philringnalda)
Er.
Flags: needinfo?(philringnalda) → needinfo?(continuation)
(In reply to Bobby Holley (:bholley) from comment #92)
> Ryan, do you have any thoughts on how to trigger this
> reliably on a try build?

Not sure about the "reliably" part. Usually, we just retrigger until we hit it. This is frequent enough that I would suspect we could retrigger our way to it on Try without too much effort.

Looking at the TBPLbot comments here, I'd start with Armv6 reftest-2.
try: -b o -p android-armv6 -u plain-reftest-2 -t none
Flags: needinfo?(ryanvm)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Hmm.  I don't have any particularly concrete thoughts here.  We have had problems where an object ends up in an arena with an incorrect finalize kind, so we don't run the finalizer.

I wonder if the globals were finalizing could be flagged somewhere when they are unlinked or something, in the XPC hashtables?  I'm just trying to think of some way we could try to guess when something should have been finalized.  I'm not sure how that would work, though.
Flags: needinfo?(continuation)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
https://tbpl.mozilla.org/php/getParsedLog.php?id=35292567&tree=Mozilla-Inbound
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Duplicate of this bug: 980284
Any updates here, Bobby? Still hitting us pretty frequently :)
Flags: needinfo?(bobbyholley)
It is on my list. I'm just swamped. Hopefully soon.
Flags: needinfo?(bobbyholley)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
I have some cycles to look at this again. Here's a try push with some release-mode assertions and some logging:

https://tbpl.mozilla.org/?tree=Try&rev=4fcb09814e7d
https://tbpl.mozilla.org/php/getParsedLog.php?id=35947789&tree=Mozilla-Inbound
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Ryan, do you recommend heavy retriggering of the runs in comment 144? I just want to check before I blow some infra dollars.
Flags: needinfo?(ryanvm)
I think that unless there's a better way to trigger the crashes, it's the only real option we've got. I just got the ball rolling for you :)
Flags: needinfo?(ryanvm)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Ryan, should we retrigger more, or try a different configuration?
Flags: needinfo?(ryanvm)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(In reply to Bobby Holley (:bholley) from comment #162)
> Ryan, should we retrigger more, or try a different configuration?

Comment 168 is from your try push, so apparently retrigger more is the answer.
Flags: needinfo?(ryanvm)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Ok. So it appears that a Window's compartment is being destroyed without WrappedNativeFinalize being called. Andrew, do you have an idea of where to look next?
Flags: needinfo?(continuation)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
I don't have any particular ideas, sorry.  The logging patch you posted in IRC looks good.  I think in general you are more in deep GC territory here so you are probably going to get more useful help from jonco or terrence.
Flags: needinfo?(continuation)
Here's a lot more logging:

https://tbpl.mozilla.org/?tree=Try&rev=0e1bd08b6975
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
https://tbpl.mozilla.org/?tree=Try&rev=6c7268e6b685
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
All hail the mysterious and fickle Android compiler.

https://tbpl.mozilla.org/?tree=Try&rev=d5788721b33a
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Depends on: 986843
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)

Updated

4 years ago
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 986843
Comment hidden (Treeherder Robot)
You need to log in before you can comment on or make changes to this bug.