Closed Bug 950653 Opened 12 years ago Closed 11 years ago

Intermittent b2ginstance.py | application crashed [@ JSAutoCompartment::JSAutoCompartment]

Categories

(Core :: JavaScript Engine, defect)

Product:
Core
Component:
JavaScript Engine
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla33
Tracking Flags:
Tracking Status
firefox32 --- fixed
firefox33 --- fixed
firefox-esr24 --- unaffected
firefox-esr31 --- unaffected

People

(Reporter: cbook, Assigned: bholley)

References

(
URL
)

Details

(Keywords: crash, intermittent-failure)

Crash Data

b2g_emulator_vm b2g-inbound opt test crashtest-1 on 2013-12-16 03:30:20 PST for push 18e49736c16e slave: tst-linux64-ec2-149 https://tbpl.mozilla.org/php/getParsedLog.php?id=32020347&tree=B2g-Inbound PROCESS-CRASH | b2ginstance.py | application crashed [@ JSAutoCompartment::JSAutoCompartment] 03:35:25 WARNING - PROCESS-CRASH | b2ginstance.py | application crashed [@ JSAutoCompartment::JSAutoCompartment] 03:35:25 INFO - Crash dump filename: /tmp/tmpZ_pv4W/3609ef92-ee89-e24e-496b48da-1de05e54.dmp 03:35:25 INFO - Operating system: Android 03:35:25 INFO - 0.0.0 Linux 2.6.29-00297-ge2ba18d #4 Tue Sep 24 09:35:47 UTC 2013 armv7l Android/full/generic:4.0.4.0.4.0.4/OPENMASTER/eng.cltbld.20131216.045710:eng/test-keys 03:35:25 INFO - CPU: arm 03:35:25 INFO - 0 CPUs 03:35:25 INFO - Crash reason: SIGSEGV 03:35:25 INFO - Crash address: 0x70 03:35:25 INFO - Thread 0 (crashed) 03:35:25 INFO - 0 libxul.so!JSAutoCompartment::JSAutoCompartment [jscompartment.h:18e49736c16e : 147 + 0x0] 03:35:25 INFO - r4 = 0xbeb6ebf0 r5 = 0x402e0c10 r6 = 0x46703e10 r7 = 0x00000000 03:35:25 INFO - r8 = 0xbeb6ed70 r9 = 0xbeb6ed70 r10 = 0x41f50b14 fp = 0xbeb6ede0 03:35:25 INFO - sp = 0xbeb6ebe0 lr = 0x411d39c9 pc = 0x41abe9f2 03:35:25 INFO - Found by: given as instruction pointer in context 03:35:25 INFO - 1 libxul.so!mozJSComponentLoader::Import(nsACString_internal const&, JS::Value const&, JSContext*, unsigned char, JS::Value*) [Maybe.h : 61 + 0x7] 03:35:25 INFO - r4 = 0x40220270 r5 = 0x402e0c10 r6 = 0x46703e10 r7 = 0x00000000 03:35:25 INFO - r8 = 0xbeb6ed70 r9 = 0xbeb6ed70 r10 = 0x41f50b14 fp = 0xbeb6ede0 03:35:25 INFO - sp = 0xbeb6ebe8 pc = 0x411d39c9 03:35:25 INFO - Found by: call frame info 03:35:25 INFO - 2 libxul.so!nsXPCComponents_Utils::Import(nsACString_internal const&, JS::Value const&, JSContext*, unsigned char, JS::Value*) [XPCComponents.cpp:18e49736c16e : 2826 + 0x11] 03:35:25 INFO - r4 = 0x411d392d r5 = 0xbeb6ed40 r6 = 0x46703e10 r7 = 0x40220270 03:35:25 INFO - r8 = 0x00000005 r9 = 0xbeb6ed70 r10 = 0x41f50b14 fp = 0xbeb6ede0 03:35:25 INFO - sp = 0xbeb6ec40 pc = 0x411a33fd 03:35:25 INFO - Found by: call frame info 03:35:25 INFO - 3 libxul.so!NS_InvokeByIndex [xptcinvoke_arm.cpp:18e49736c16e : 164 + 0x23] 03:35:25 INFO - r4 = 0x411a33c1 r5 = 0xbeb6ec68 r6 = 0xbeb6ec90 r7 = 0xbeb6ec98 03:35:25 INFO - r8 = 0x00000005 r9 = 0xbeb6ed70 r10 = 0x41f50b14 fp = 0xbeb6ede0 03:35:25 INFO - sp = 0xbeb6ec68 pc = 0x40c085ff 03:35:25 INFO - Found by: call frame info 03:35:25 INFO - 4 libxul.so!XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) [XPCWrappedNative.cpp:18e49736c16e : 2566 + 0xd] 03:35:25 INFO - r4 = 0x422fa3d4 r5 = 0xbeb6ed70 r6 = 0xbeb6ed08 r7 = 0x00000010 03:35:25 INFO - r8 = 0x00000003 r9 = 0x0000001a r10 = 0x41f50b14 fp = 0xbeb6ede0 03:35:25 INFO - sp = 0xbeb6ecc8 pc = 0x411c5a37 03:35:25 INFO - Found by: call frame info 03:35:25 INFO - 5 libxul.so!XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) [XPCWrappedNativeJSOps.cpp:18e49736c16e : 1300 + 0x7] 03:35:25 INFO - r4 = 0x40220270 r5 = 0xbeb6eeb8 r6 = 0x00000001 r7 = 0x00000001 03:35:25 INFO - r8 = 0x402cc8a4 r9 = 0x44930880 r10 = 0x44930880 fp = 0xbeb6f3f8 03:35:25 INFO - sp = 0xbeb6ee98 pc = 0x411c9b53 03:35:25 INFO - Found by: call frame info 03:35:25 INFO - 6 libxul.so!js::Invoke [jscntxtinlines.h:18e49736c16e : 220 + 0x9]
Bobby, I see XPConnect stuff on the stack. Any ideas what might be going on here? It's a B2G emulator startup crash.
Flags: needinfo?(bobbyholley)
OS: Linux → Gonk (Firefox OS)
Hardware: x86 → ARM
I don't have the bandwidth - still trying to find time to look at bug 960828. Maybe gabor can take a look?
Flags: needinfo?(bobbyholley) → needinfo?(gkrizsanits)
I'm flooded as hell as well but will try to find some time for this...
Flags: needinfo?(gkrizsanits)
Flags: needinfo?(poirot.alex)
Flags: needinfo?(poirot.alex)
Naveed, can you help us find an owner? This has gotten to be an extremely frequent crash over the past few days.
Severity: normal → critical
Flags: needinfo?(nihsanullah)
Flags: needinfo?(nihsanullah)
Note that every TBPLbot comment here represents 3-5 individual failures each on average due to us starring multiple instances at the same time.
jandem: this B2G test crash in JSAutoCompartment started spiking March ~4: https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=950653 Do you know of any JS changes from March 3–4 that might have provoked this crash? https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=4cb766685b73&tochange=529b86b92b1d 14:53:33 WARNING - PROCESS-CRASH | b2ginstance.py | application crashed [@ JSAutoCompartment::JSAutoCompartment] 14:53:33 INFO - Crash dump filename: /tmp/tmpWPc6sZ/7f7e135f-a3be-14c0-1129898a-680c0678.dmp 14:53:33 INFO - Operating system: Android 14:53:33 INFO - 0.0.0 Linux 2.6.29-00302-g586075d #31 Mon Feb 24 10:28:23 PST 2014 armv7l Android/full/generic:4.0.4.0.4.0.4/OPENMASTER/eng.cltbld.20140307.155909:eng/test-keys 14:53:33 INFO - CPU: arm 14:53:33 INFO - 0 CPUs 14:53:33 INFO - Crash reason: SIGSEGV 14:53:33 INFO - Crash address: 0xdadadae6 14:53:33 INFO - Thread 0 (crashed) 14:53:33 INFO - 0 libxul.so!JSAutoCompartment::JSAutoCompartment [Shape.h:99e60b1adf71 : 714 + 0x2] 14:53:33 INFO - r4 = 0x447d21f0 r5 = 0xbe97c7e0 r6 = 0x00000001 r7 = 0xbe97c7b8 14:53:33 INFO - r8 = 0xbe97c7f0 r9 = 0x00000000 r10 = 0x45b79e30 fp = 0x40220530 14:53:33 INFO - sp = 0xbe97c798 lr = 0x422e12c9 pc = 0x422e12e6 14:53:33 INFO - Found by: given as instruction pointer in context 14:53:33 INFO - 1 libxul.so!mozJSComponentLoader::Import(nsACString_internal const&, JS::Handle<JS::Value>, JSContext*, unsigned char, JS::MutableHandle<JS::Value>) [Maybe.h : 61 + 0x7] 14:53:33 INFO - r4 = 0x464d3080 r5 = 0x00000000 r6 = 0x00000001 r7 = 0xbe97c7b8 14:53:33 INFO - r8 = 0xbe97c7f0 r9 = 0x00000000 r10 = 0x45b79e30 fp = 0x40220530 14:53:33 INFO - sp = 0xbe97c7b0 pc = 0x4152025f 14:53:33 INFO - Found by: call frame info 14:53:33 INFO - 2 libxul.so!nsXPCComponents_Utils::Import(nsACString_internal const&, JS::Handle<JS::Value>, JSContext*, unsigned char, JS::MutableHandle<JS::Value>) [XPCComponents.cpp:99e60b1adf71 : 2735 + 0x11] 14:53:33 INFO - r4 = 0x41520105 r5 = 0x45b79e30 r6 = 0x464d3080 r7 = 0xbe97ca50 14:53:33 INFO - r8 = 0xbe97c890 r9 = 0x00000008 r10 = 0x43077e54 fp = 0xbe97cb08 14:53:33 INFO - sp = 0xbe97c840 pc = 0x414c95fb 14:53:33 INFO - Found by: call frame info 14:53:33 INFO - 3 libxul.so!NS_InvokeByIndex [xptcinvoke_arm.cpp:99e60b1adf71 : 164 + 0x1b] 14:53:33 INFO - r4 = 0x414c95c9 r5 = 0x00000005 r6 = 0x466f5240 r7 = 0xbe97c898 14:53:33 INFO - r8 = 0xbe97c890 r9 = 0x00000008 r10 = 0x43077e54 fp = 0xbe97cb08 14:53:33 INFO - sp = 0xbe97c868 pc = 0x40ccf59f 14:53:33 INFO - Found by: call frame info 14:53:33 INFO - 4 libxul.so!CallMethodHelper::Call() [XPCWrappedNative.cpp:99e60b1adf71 : 2406 + 0xd] 14:53:33 INFO - r4 = 0xbe97ca10 r5 = 0x00000001 r6 = 0x00000003 r7 = 0x00000003 14:53:33 INFO - r8 = 0xbe97c920 r9 = 0x0000001a r10 = 0x43077e54 fp = 0xbe97cb08 14:53:33 INFO - sp = 0xbe97c8b8 pc = 0x41503143 14:53:33 INFO - Found by: call frame info 14:53:33 INFO - 5 libxul.so!XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) [XPCWrappedNative.cpp:99e60b1adf71 : 1714 + 0x5] 14:53:33 INFO - r4 = 0x00000000 r5 = 0xbe97cb08 r6 = 0xbe97ca10 r7 = 0xbe97ca30 14:53:33 INFO - r8 = 0xbe97ca38 r9 = 0xbe97cb8c r10 = 0x00000001 fp = 0x445da400 14:53:33 INFO - sp = 0xbe97ca08 pc = 0x41504565 14:53:33 INFO - Found by: call frame info 14:53:33 INFO - 6 libxul.so!XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) [XPCWrappedNativeJSOps.cpp:99e60b1adf71 : 1285 + 0x7]
Flags: needinfo?(jdemooij)
Flags: needinfo?(jdemooij)
Blocks: 998476
No longer blocks: 998476
Crash Signature: [@JSAutoCompartment::JSAutoCompartment] → [@JSAutoCompartment::JSAutoCompartment] [@ JSAutoCompartment::JSAutoCompartment | mozJSComponentLoader::Import(nsACString_internal const&, JS::Handle<JS::Value>, JSContext*, unsigned char, JS::MutableHandle<JS::Value>)]
Hi Bobby, I was suggested to perhaps talk to you to see if we might be able to find someone to work on this bug? Could you take a look at it and see who I might be able to ask to get some traction on this bug please?
Flags: needinfo?(bobbyholley)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #201) > Hi Bobby, I was suggested to perhaps talk to you to see if we might be able > to find someone to work on this bug? Could you take a look at it and see > who I might be able to ask to get some traction on this bug please? Isn't this a dupe of bug 980537 per comment 170? We haven't hit it since the day that bug was backported (see bug 980537 comment 36).
Flags: needinfo?(bobbyholley)
Similar to what kairo mentioned in bug 983976, it is a lower occurrance so I'll close it for now unless it appears more often or we can find reproducible steps.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
bholley: this intermittent crash seems to have spiked today. The stack trace is the same, but the crash addresses have changed. Does the fact that the crash address is sometimes jemalloc uninitialized memory, sometimes JS_FREE_PATTERN, or sometimes a null pointer suggest anything? It seems a little strange that pointers so many types of memory allocations would be passing through the same code path. The recent crash addresses are 0xa5a5a5d1, which is 0xa5a5a5a5 + 44. firebot says 0xa5a5a5a5 is jemalloc allocated uninitialized junk memory. But crash reports from earlier today also included 0x2c (44) from a NULL pointer access. In comment 106 from March, jandem noted that the crash address then was 0xdadadae6, which was 0xdadadada + 12 == offsetof(BaseShape, compartment_). 0xdadadada is JS_FREE_PATTERN (deallocated JSArena - GC()d). 20:03:17 INFO - Crash reason: SIGSEGV 20:03:17 INFO - Crash address: 0xa5a5a5d1 20:03:17 INFO - Thread 0 (crashed) 20:03:17 INFO - 0 libxul.so!JSAutoCompartment::JSAutoCompartment [jscompartment.h:a6bf99f60178 : 150 + 0x0] 20:03:17 INFO - r4 = 0x449a8be0 r5 = 0xbe850078 r6 = 0x00000001 r7 = 0xbe850050 20:03:17 INFO - r8 = 0xbe850088 r9 = 0x00000000 r10 = 0x44b4a280 fp = 0x40238690 20:03:17 INFO - sp = 0xbe850030 lr = 0x4250be15 pc = 0x4250be24 20:03:17 INFO - Found by: given as instruction pointer in context 20:03:17 INFO - 1 libxul.so!mozJSComponentLoader::Import(nsACString_internal const&, JS::Handle<JS::Value>, JSContext*, unsigned char, JS::MutableHandle<JS::Value>) [Maybe.h : 61 + 0x7] 20:03:17 INFO - r4 = 0x44615220 r5 = 0x00000000 r6 = 0x00000001 r7 = 0xbe850050 20:03:17 INFO - r8 = 0xbe850088 r9 = 0x00000000 r10 = 0x44b4a280 fp = 0x40238690 20:03:17 INFO - sp = 0xbe850048 pc = 0x416c91ab 20:03:17 INFO - Found by: call frame info 20:03:17 INFO - 2 libxul.so!nsXPCComponents_Utils::Import(nsACString_internal const&, JS::Handle<JS::Value>, JSContext*, unsigned char, JS::MutableHandle<JS::Value>) [XPCComponents.cpp:a6bf99f60178 : 2724 + 0x19] 20:03:17 INFO - r4 = 0x416c9059 r5 = 0x44b4a280 r6 = 0x44615220 r7 = 0xbe8502e0 20:03:17 INFO - r8 = 0x00000000 r9 = 0x00000008 r10 = 0x0000001a fp = 0xbe850390 20:03:17 INFO - sp = 0xbe8500d8 pc = 0x41672eb5 20:03:17 INFO - Found by: call frame info 20:03:17 INFO - 3 libxul.so!NS_InvokeByIndex [xptcinvoke_arm.cpp:a6bf99f60178 : 164 + 0x1b] 20:03:17 INFO - r4 = 0x41672e79 r5 = 0x00000005 r6 = 0x450dc940 r7 = 0xbe850130 20:03:17 INFO - r8 = 0xbe850128 r9 = 0x00000008 r10 = 0x0000001a fp = 0xbe850390 20:03:17 INFO - sp = 0xbe850100 pc = 0x40d6c9e3 20:03:17 INFO - Found by: call frame info 20:03:17 INFO - 4 libxul.so!CallMethodHelper::Call() [XPCWrappedNative.cpp:a6bf99f60178 : 2397 + 0xd] 20:03:17 INFO - r4 = 0xbe8502a0 r5 = 0x00000001 r6 = 0x00000003 r7 = 0x00000003 20:03:17 INFO - r8 = 0xbe8501b0 r9 = 0x432a37e8 r10 = 0x0000001a fp = 0xbe850390 20:03:17 INFO - sp = 0xbe850150 pc = 0x416aacd7 20:03:17 INFO - Found by: call frame info 20:03:17 INFO - 5 libxul.so!XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) [XPCWrappedNative.cpp:a6bf99f60178 : 1705 + 0x5] 20:03:17 INFO - r4 = 0x00000000 r5 = 0xbe850390 r6 = 0xbe8502a0 r7 = 0xbe8502c0 20:03:17 INFO - r8 = 0xbe8502c8 r9 = 0xbe850404 r10 = 0xbe850ac8 fp = 0x00000000 20:03:17 INFO - sp = 0xbe850298 pc = 0x416abfdf 20:03:17 INFO - Found by: call frame info 20:03:17 INFO - 6 libxul.so!XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) [XPCWrappedNativeJSOps.cpp:a6bf99f60178 : 1273 + 0x7]
Flags: needinfo?(bobbyholley)
Given that this really started spiking yesterday, we should be able to bisect when this regressed.
The TBPLbot history shows a pretty clear regression on b-i starting yesterday, followed by merging around from there.
If someone could bisect this, even to a range of changesets, I might be able to pick out the culprit.
Flags: needinfo?(bobbyholley)
It's not definitive yet, but I'm pretty sure it's focusing down to this push: https://hg.mozilla.org/integration/b2g-inbound/rev/09ebd0af27a4 I've got a backout going on Try as well. If it confirms, I'll backout.
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #331) > It's not definitive yet, but I'm pretty sure it's focusing down to this push: > https://hg.mozilla.org/integration/b2g-inbound/rev/09ebd0af27a4 > > I've got a backout going on Try as well. If it confirms, I'll backout. You rock, Ryan. :-)
Bug 903291 is looking pretty convincing as the culprit: https://tbpl.mozilla.org/?tree=B2g-Inbound&jobname=emulator.*debug&fromchange=6edd25369826&tochange=09ebd0af27a4 No crashes on the Try run yet. Barring any unforeseen changes, I'll push a backout with my next set of merges.
Bug 903291 has been backed out on m-c. We should see these crashes taper off as it merges around and the already-running jobs cycle through.
So this now appears to be mostly happening in opt builds. We really need a better way to make these crashes more easy to differentiate from each other and we've now had multiple spikes in this bug for various reasons.
bholley: are there any code changes or assertions we could add to mozJSComponentLoader or JSAutoCompartment to help differentiate these various crashes? These latest crashes have crash addresses of 0x2d or 0x6c, which are different offsets from 0x00000000 than the earlier (unrelated?) crash addresses of 0xa5a5a5d1 and 0xdadadae6.
Flags: needinfo?(bobbyholley)
(In reply to Chris Peterson (:cpeterson) from comment #387) > bholley: are there any code changes or assertions we could add to > mozJSComponentLoader or JSAutoCompartment to help differentiate these > various crashes? > > These latest crashes have crash addresses of 0x2d or 0x6c, which are > different offsets from 0x00000000 than the earlier (unrelated?) crash > addresses of 0xa5a5a5d1 and 0xdadadae6. I think we should wait until nbp finishes his investigation in bug 1015027.
Flags: needinfo?(bobbyholley)
Summary: Intermittent PROCESS-CRASH | b2ginstance.py | application crashed [@ JSAutoCompartment::JSAutoCompartment] → Intermittent b2ginstance.py | application crashed [@ JSAutoCompartment::JSAutoCompartment]
I am investigating this.
Assignee: nobody → bobbyholley
Depends on: 1026860
Seems likely that bug 1026860 fixed this. Ryan, does it seem to have fixed any of the other crashes that you were seeing? We should probably backport that patch. Which branches are affected?
Flags: needinfo?(ryanvm)
For sure Aurora (v2.0). Hard to say if b2g30 (v1.4) is affected or not at this point (it certainly has been in the past). Probably would be good to take there as well if the risk in doing so is low enough. And yes, things seem less crash-prone than they'd been :)
Flags: needinfo?(ryanvm)
No longer depends on: 1015027
Looks like no instances since bug 1026860 was uplifted. Seems to be fixed.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
status-firefox32: affected → fixed
status-firefox33: --- → fixed
Target Milestone: --- → mozilla33
You need to log in before you can comment on or make changes to this bug.