871290 - crash in js::ion::Assembler::TraceJumpRelocations

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Scoobidiver (away)]

With the below stack trace, it first showed up in 23.0a1/20130509. The regression range might be (low volume):
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=b980d32c366f&tochange=ea059733677c

Signature 	js::ion::Assembler::TraceJumpRelocations(JSTracer*, js::ion::IonCode*, js::ion::CompactBufferReader&) More Reports Search
UUID	99075260-7dae-485d-8fe8-93b112130512
Date Processed	2013-05-12 16:46:15
Uptime	6182
Last Crash	2.6 weeks before submission
Install Age	1.7 hours since version was first installed.
Install Time	2013-05-12 15:03:19
Product	FennecAndroid
Version	23.0a1
Build ID	20130512030908
Release Channel	nightly
OS	Android
OS Version	0.0.0 Linux 3.0.8-g44ffd6a #1 SMP PREEMPT Fri Jul 6 03:57:28 CDT 2012 armv7l verizon/targa_vzw/cdma_targa:4.1.2/JZO54K/20121116:user/release-keys
Build Architecture	arm
Build Architecture Info	ARMv0
Crash Reason	SIGSEGV
Crash Address	0xe584bffc
App Notes 	
AdapterDescription: 'Imagination Technologies -- PowerVR SGX 540 -- OpenGL ES 2.0 build 1.8@796887 -- Model: XT875, Product: targa_vzw, Manufacturer: Motorola, Hardware: mapphone_cdma'
GL Layers! EGL? EGL+ GL Context? GL Context+ GL Layers+ 
Motorola XT875
verizon/targa_vzw/cdma_targa:4.1.2/JZO54K/20121116:user/release-keys
Processor Notes 	sp-processor07_phx1_mozilla_com_14837:2012; exploitability tool: ERROR: unable to analyze dump
EMCheckCompatibility	True
Adapter Vendor ID	Imagination Technologies
Adapter Device ID	PowerVR SGX 540
Device	Motorola XT875
Android API Version	16 (REL)
Android CPU ABI	armeabi-v7a

Frame 	Module 	Signature 	Source
0 	libxul.so 	js::ion::Assembler::TraceJumpRelocations 	IonCode.h:115
1 	libxul.so 	js::ion::IonCode::trace 	Ion.cpp:482
2 	libxul.so 	js::GCMarker::processMarkStackOther 	Marking.cpp:1129
3 	libxul.so 	ScanShape 	Marking.cpp:838
4 	libxul.so 	js::GCMarker::drainMarkStack 	Marking.cpp:1363
5 	libxul.so 	IncrementalCollectSlice 	jsgc.cpp:3780 

More reports at:
https://crash-stats.mozilla.com/report/list?signature=js%3A%3Aion%3A%3AAssembler%3A%3ATraceJumpRelocations%28JSTracer*%2C+js%3A%3Aion%3A%3AIonCode*%2C+js%3A%3Aion%3A%3ACompactBufferReader%26%29

Comment 1

[Scoobidiver (away)]

It's #5 crasher in 23.0a2 and #7 in 24.0a1.

Comment 2

[Brad Lassey [:blassey] (use needinfo?)]

Naveed, we need an assignee for this

Comment 3

[Jan de Mooij [:jandem]]

I can take a look soon.

Comment 4

[Scoobidiver (away)]

It's #55 crasher in 21.0, #88 in 22.0b3, #3 in 23.0a2, and #5 in 24.0a1.

Comment 5

[Scoobidiver (away)]

It's now even #1 top crasher in 23.0a2.

Two comments talk about scrolling:
"scrolling on reddit.."
"Scrolling (slowly) while reading an article on atlantic.com. I do not know if the page had finished loading. There was enough there for the page to look complete and allow me to read the article."

Comment 6

[Jan de Mooij [:jandem]]

Scoobidiver: there are no crashes with more recent 24.0a1 builds, right? The last build I see is from May but I'm not sure if I'm using Socorro correctly.

Comment 7

[Scoobidiver (away)]

(In reply to Jan de Mooij [:jandem] from comment #6)
> Scoobidiver: there are no crashes with more recent 24.0a1 builds, right? The
> last build I see is from May but I'm not sure if I'm using Socorro correctly.
You're partially right. The latest crash with the js::ion::Assembler::TraceJumpRelocations signature happened in 24.0a1/20130525062525 but the latest one with the MarkInternal<js::ion::IonCode> signature (added to this bug because it has only one more frame - ARMv6 version?) occurred in 24.0a1/20130605031156. See https://crash-stats.mozilla.com/report/list?signature=MarkInternal%3Cjs%3A%3Aion%3A%3AIonCode%3E

Maybe it's fixed in the trunk and not in Aurora. In that case, the working range might be (assumption: fixed after 24.0a1/20130605, 3-day range):
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8f9ba85eb61c&tochange=6006abb39d8e

Comment 8

[Scoobidiver (away)]

(In reply to Scoobidiver from comment #7)
> Maybe it's fixed in the trunk and not in Aurora.
It seems the signature has morphed again. See https://crash-stats.mozilla.com/report/list?signature=js%3A%3Agc%3A%3AMarkIonCode%28JSTracer*%2C+js%3A%3AEncapsulatedPtr%3Cjs%3A%3Aion%3A%3AIonCode%2C+unsigned+int%3E*%2C+char+const*%29

Comment 9

[Jan de Mooij [:jandem]]

I discussed this with mjrosenb and decoder on IRC. decoder's fuzzer hit this crash on ARM, so hopefully he will be able to get us a testcase soon.

mjrosenb wrote a patch that may fix the problem, hopefully once we have a testcase we can see if that patch fixes it and land it.

Comment 10

[Marty Rosenberg [:mjrosenb]]

Attachment: /home/mjrosenb/patches/bug871290-r0.patch

couldn't think of a more descriptive name.  this passed jit-tests, so it shouldn't cause any regressions.  I haven't tested on armv6 yet.

Comment 11

[Marty Rosenberg [:mjrosenb]]

landed: https://hg.mozilla.org/integration/mozilla-inbound/rev/7f58c0937b1d

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/7f58c0937b1d

Comment 13

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

:mjrosenb, can you nom it for 23 please?

Comment 14

[Marty Rosenberg [:mjrosenb]]

Comment on attachment 761074 [details] [diff] [review]
/home/mjrosenb/patches/bug871290-r0.patch

om nom nom
[Approval Request Comment]
Bug caused by (feature/regressing bug #): 805241
User impact if declined: crashes on malicious web sites
Testing completed (on m-c, etc.): sitting on m-c for a few days now
Risk to taking this patch (and alternatives if risky): there may be some features that we won't be able to add in c++ due to a slight restriction on what data can be accessed-- I can't imagine this will ever affect anything we could want to write
String or IDL/UUID changes made by this patch:

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/ce308537fba9

Comment 16

[Scoobidiver (away)]

(In reply to Marty Rosenberg [:mjrosenb] from comment #14)
> Testing completed (on m-c, etc.): sitting on m-c for a few days now
Unfortunately at the same time, the patch of bug 678037 landed. It had been hidden by its side effects for a few builds. See https://crash-stats.mozilla.com/report/list?signature=js%3A%3Agc%3A%3AMarkGCThingRoot%28JSTracer*%2C+void**%2C+char+const*%29

Comment 17

[Scoobidiver (away)]

Because of bug 884300, there's now a full stack trace. See bp-c6fcb362-6504-4777-9e14-ccb9f2130621:
Frame 	Module 	Signature 	Source
0 	libxul.so 	js::ion::Assembler::TraceJumpRelocations 	js/src/ion/IonCode.h:115
1 	libxul.so 	js::ion::IonCode::trace 	js/src/ion/Ion.cpp:482
2 	libxul.so 	js::GCMarker::processMarkStackOther 	js/src/gc/Marking.cpp:1129
3 	libxul.so 	js::GCMarker::drainMarkStack 	js/src/gc/Marking.cpp:1363
4 	libxul.so 	IncrementalCollectSlice 	js/src/jsgc.cpp:3780
5 	libxul.so 	GCCycle 	js/src/jsgc.cpp:4422
6 	libxul.so 	Collect 	js/src/jsgc.cpp:4581
7 	libxul.so 	js::GC 	js/src/jsgc.cpp:4491
8 	libxul.so 	mozilla::dom::workers::WorkerPrivate::GarbageCollectInternal 	dom/workers/WorkerPrivate.cpp:4111
9 	libxul.so 	GarbageCollectRunnable::WorkerRun 	dom/workers/WorkerPrivate.cpp:1400
10 	libxul.so 	mozilla::dom::workers::WorkerRunnable::Run 	dom/workers/WorkerPrivate.cpp:1638
11 	libxul.so 	mozilla::dom::workers::WorkerPrivate::DoRunLoop 	dom/workers/WorkerPrivate.cpp:2762
12 	libxul.so 	WorkerThreadRunnable::Run 	dom/workers/RuntimeService.cpp:534
13 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:627
14 	libxul.so 	NS_ProcessNextEvent 	obj-firefox/xpcom/build/nsThreadUtils.cpp:238
15 	libxul.so 	nsThread::ThreadFunc 	xpcom/threads/nsThread.cpp:265
16 	libnss3.so 	_pt_root 	nsprpub/pr/src/pthreads/ptthread.c:204
17 	libc.so 	libc.so@0xe3da 	
18 	libc.so 	libc.so@0xdac6

Comment 18

[Scoobidiver (away)]

It's #3 top crasher in 23.0b1.
Since 24.0a1/20130615 when the patch landed with others, it's a lower volume crash, only four crashes.

(In reply to Jan de Mooij [:jandem] from comment #9)
> decoder's fuzzer hit this crash on ARM, so hopefully he will be able to get us a 
> testcase soon.
Can we get that testcase?

Comment 19

[Christian Holler (:decoder)]

We don't have a testcase for this. All crashes that I saw were not reproducible. I talked with Marty about adding a debug-only function for testing, that makes such conditions more reproducible but we don't have this yet.

Comment 20

[Tracy Walker [:tracy]]

a bunch of single crashes.  let me know if you'd like more.

1 	http://auto.mail.ru/market/msk/chevrolet/cruze/20839473/
1 	http://albertmcwilliams.com/post/54015994028/youre-going-to-kill-someone
1 	http://m.yahoo.com/w/legobpengine/finance/details/?.b=quotessearch%2F%3F.b%3Dquotessearch%2F%3F.b%3Ddetails%2F%3F.b%3Dindex&.sy=GLD&.ss=GLD&.ts=1372686670&.intl=us&.lang=en
1 	http://www.salvationhistory.com/library/category/the_word_of_god
1 	http://www.hullcityindependent.net/?page=news&news_id=1251
1 	http://m.youtube.com/watch?v=ORs6C32ivjg&feature=youtu.be&desktop_uri=%2Fwatch%3Fv%3DORs6C32ivjg%26feature%3Dyoutu.be
1 	http://www.google.com.ng/search?q=one+click+root+v1.0+crack&oq=one+click+root+v1.0+crack&gs_l=mobile-heirloom-serp.12...0.0.1.3725.0.0.0.0.0.0.0.0..0.0...0.0...1c..16.mobile-heirloom-serp.nvoy4jV4PfU
1 	http://www.bbc.co.uk/news/world-asia-23112180
1 	https://www.google.com/search?q=jessica+chastain&tbm=isch&biw=1453&bih=867&sei=Be_SUazbIqP0yQGD74DYBg#

Comment 21

[Scoobidiver (away)]

A comment says: "I was actually trying to force a crash by repeatedly clicking the back button out of an email link of a link ."

Comment 22

[Scoobidiver (away)]

Still #1 top crasher in 23.0b2 but no crashes in 24.0a2 and one in 25.0a1.

Comment 23

[Robert Kaiser]

(In reply to Scoobidiver from comment #22)
> Still #1 top crasher in 23.0b2 but no crashes in 24.0a2 and one in 25.0a1.

Hasn't this landed on 23 as well?

Comment 24

[Scoobidiver (away)]

(In reply to Robert Kaiser (:[e-mail]) from comment #23)
> Hasn't this landed on 23 as well?
The patch in this bug hasn't fixed the issue (see comment 16). It's another unknown patch that has fixed it.

Comment 25

[Christian Holler (:decoder)]

If we don't know what "fixed" this then it's not even clear that it's fixed at all because this bug is highly fragile, minor changes can make repros fail easily.

Marty, some time ago you mentioned adding a debug function to find problems like this (pool problems I think) easier in the shell. Can we add that and try more fuzzing then?

Comment 26

[Tracy Walker [:tracy]]

the last build to crash in js::ion::Assembler::TraceJumpRelocations(JSTracer*, js::ion::IonCode*, js::ion::CompactBufferReader&) are from build 20130701142603. 

There are crashes [@ MarkInternal<js::ion::IonCode>] with the latest betas.

Should a separate bug be split off for the crashing signature.

Comment 27

[Scoobidiver (away)]

(In reply to Scoobidiver from comment #22)
> no crashes in 24.0a2 and one in 25.0a1.
It has been back since 25.0a1/20130710.

Comment 28

[Lukas Blakk [:lsblakk] use ?needinfo]

Not a topcrasher for FF23, removing tracking and keyword.

Comment 29

[Scoobidiver (away)]

(In reply to [e-mail] [:lsblakk] from comment #28)
> Not a topcrasher for FF23, removing tracking and keyword.
???? It's #2 top crasher in 23.0b4.

Comment 30

[Lukas Blakk [:lsblakk] use ?needinfo]

My mistake, I was looking at Firefox, not Fennec stats

Comment 31

[Jan de Mooij [:jandem]]

Marty, do we have any idea what's going on here? Is there anything we can do to help the fuzzers? We need to fix this.

Comment 32

[Scoobidiver (away)]

A comment says: "Hate to say it but FFox has been acting ''wonky'' since the Google keyboard (touch screen) driver update. Intermittent: crashes, only loading the left (slightly more than) half of the screen, freezing... What I'm doing, what site at the time, doesn't seem to matter."

Comment 33

[Jan de Mooij [:jandem]]

Marty and I are debugging this now on Try (see also bug 894251).

Comment 34

[Jan de Mooij [:jandem]]

(In reply to Jan de Mooij [:jandem] from comment #33)
> Marty and I are debugging this now on Try (see also bug 894251).

We tracked down this one, will be fixed in bug 894251. This should hopefully stop the intermittent crashes we're currently seeing on inbound/central. The bug has been there since Ion was first released in 18 but is unlikely to cause the beta topcrashes -- still, we really want to land that fix on beta and see if it helps some.

Comment 35

[Jan de Mooij [:jandem]]

(In reply to Jan de Mooij [:jandem] from comment #34)
> The bug has been there since Ion was first released in 18 but is unlikely to
> cause the beta topcrashes -- 

After looking at the stack traces again I'm not so sure anymore these are unrelated. Almost all stack traces point to GCs from a DOM worker thread. Workers have their own JSRuntime and hence get their own pre-compiled/shared stubs, so it's possible creating many workers makes us more likely to hit bug 894251.

So I hope this is bug 894251, but it's hard to know for sure at this point.

Comment 36

[Scoobidiver (away)]

(In reply to Jan de Mooij [:jandem] from comment #35)
> So I hope this is bug 894251, but it's hard to know for sure at this point.
There have been no crashes since 25.0a1/20130727 but it previously skipped up to two builds so let's wait a little more to confirm.

Comment 37

[Scoobidiver (away)]

Here is a recent crash report: bp-74b4f5ef-0415-4527-967a-7e84c2130730.

Comment 38

[Jan de Mooij [:jandem]]

(In reply to Scoobidiver from comment #37)
> Here is a recent crash report: bp-74b4f5ef-0415-4527-967a-7e84c2130730.

That's a different signature though: TraceJumpRelocations vs TraceDataRelocations.

Comment 39

[Marty Rosenberg [:mjrosenb]]

> That's a different signature though: TraceJumpRelocations vs
> TraceDataRelocations.

Oh wow, I didn't even notice that.  Is it possible that this is related to the improper use of ma_mov (as opposed to ma_movPatchable) in one of the data paths?

Comment 40

[Kevin Brosnan [Ex-Mozilla]]

23+ ship has sailed. Need to re-triage this.

Comment 41

[Robert Kaiser]

(In reply to Kevin Brosnan [:kbrosnan] from comment #40)
> 23+ ship has sailed. Need to re-triage this.

Er, why? This should be fixed for 23.0b10.

Comment 42

[Kevin Brosnan [Ex-Mozilla]]

I was just mass resetting flags after FxA triage. We decided that most open bugs marked 23+ were not going to make 23. 

Looking at crash stats this looks to be fixed.

Comment 43

[Scoobidiver (away)]

Let's consider the top crasher fixed although there are still some crashes in Nightly.