737928 - crash in js::detail::RegExpCode::compile when restarting after installing ABP

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine, defect)

Description

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

This bug was filed from the Socorro interface and is 
report bp-b43bddc3-e2b6-496a-9fe0-e4ada2120320 .
============================================================= 
Crashing Thread
Frame 	Module 	Signature [Expand] 	Source
0 	libxul.so 	JSC::Yarr::wordcharCreate 	Utility.h:581
1 	libxul.so 	JSC::Yarr::byteCompile 	js/src/yarr/YarrPattern.h:392
2 	libxul.so 	js::detail::RegExpCode::compile 	js/src/vm/RegExpObject.cpp:280
3 	libxul.so 	js::RegExpObject::createShared 	js/src/vm/RegExpObject.cpp:525
4 	libxul.so 	js::CloneRegExpObject 	js/src/vm/RegExpObject-inl.h:82
5 	libxul.so 	js::Interpret 	js/src/jsinterp.cpp:2850
6 	libxul.so 	js::RunScript 	js/src/jsinterp.cpp:469
7 	libxul.so 	js::Execute 	js/src/jsinterp.cpp:667
8 	libxul.so 	JS_ExecuteScript 	js/src/jsapi.cpp:5232
9 	libxul.so 	JS_ExecuteScriptVersion 	js/src/jsapi.cpp:5240
10 	libxul.so 	mozJSComponentLoader::GlobalForLocation 	js/xpconnect/loader/mozJSComponentLoader.cpp:955
11 	libxul.so 	mozJSComponentLoader::LoadModule 	js/xpconnect/loader/mozJSComponentLoader.cpp:517
12 	libxul.so 	nsComponentManagerImpl::KnownModule::Load 	xpcom/components/nsComponentManager.cpp:723
13 	libxul.so 	nsFactoryEntry::GetFactory 	xpcom/components/nsComponentManager.cpp:1738
14 	libxul.so 	nsComponentManagerImpl::CreateInstance 	xpcom/components/nsComponentManager.cpp:974
15 	libxul.so 	nsComponentManagerImpl::GetService 	xpcom/components/nsComponentManager.cpp:1270
16 	libxul.so 	nsJSCID::GetService 	js/xpconnect/src/XPCJSID.cpp:803
17 	libxul.so 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/unix/xptcinvoke_arm.cpp:194
18 	libxul.so 	XPCWrappedNative::CallMethod 	js/xpconnect/src/XPCWrappedNative.cpp:3021
19 	libxul.so 	XPC_WN_CallMethod 	js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1539
20 	libxul.so 	js::Interpret 	js/src/jscntxtinlines.h:314
21 	libxul.so 	js::RunScript 	js/src/jsinterp.cpp:469
22 	libxul.so 	js::InvokeKernel 	js/src/jsinterp.cpp:528
23 	libxul.so 	js_fun_apply 	js/src/jsinterp.h:172
24 	libxul.so 	js::Interpret 	js/src/jscntxtinlines.h:314
25 	libxul.so 	js::RunScript 	js/src/jsinterp.cpp:469
26 	libxul.so 	js::InvokeGetterOrSetter 	js/src/jsinterp.cpp:528
27 	libxul.so 	js::GetPropertyHelper 	js/src/jsscopeinlines.h:287
28 	libxul.so 	js::Interpret 	js/src/jsinterpinlines.h:268
29 	libxul.so 	js::RunScript 	js/src/jsinterp.cpp:469
30 	libxul.so 	js::InvokeKernel 	js/src/jsinterp.cpp:528
31 	libxul.so 	js_fun_apply 	js/src/jsinterp.h:172
32 	libxul.so 	js::Interpret 	js/src/jscntxtinlines.h:314
33 	libxul.so 	js::RunScript 	js/src/jsinterp.cpp:469
34 	libxul.so 	js::InvokeKernel 	js/src/jsinterp.cpp:528
35 	libxul.so 	js_fun_apply 	js/src/jsinterp.h:172
36 	libxul.so 	js::Interpret 	js/src/jscntxtinlines.h:314
37 	libxul.so 	js::RunScript 	js/src/jsinterp.cpp:469
38 	libxul.so 	js::InvokeKernel 	js/src/jsinterp.cpp:528
39 	libxul.so 	js_fun_apply 	js/src/jsinterp.h:172
40 	libxul.so 	js::Interpret 	js/src/jscntxtinlines.h:314
41 	libxul.so 	js::RunScript 	js/src/jsinterp.cpp:469
42 	libxul.so 	js::InvokeKernel 	js/src/jsinterp.cpp:528
43 	libxul.so 	js_fun_apply 	js/src/jsinterp.h:172
44 	libxul.so 	js::Interpret 	js/src/jscntxtinlines.h:314
45 	libxul.so 	js::RunScript 	js/src/jsinterp.cpp:469
46 	libxul.so 	js::Invoke 	js/src/jsinterp.cpp:528
47 	libxul.so 	JS_CallFunctionValue 	js/src/jsapi.cpp:5389
48 	libxul.so 	nsJSContext::CallEventHandler 	dom/base/nsJSEnvironment.cpp:1880
49 	libxul.so 	nsJSEventListener::HandleEvent 	dom/src/events/nsJSEventListener.cpp:239
50 	libxul.so 	nsEventListenerManager::HandleEventSubType 	content/events/src/nsEventListenerManager.cpp:740
51 	libxul.so 	nsEventListenerManager::HandleEventInternal 	content/events/src/nsEventListenerManager.cpp:798
52 	libxul.so 	nsEventTargetChainItem::HandleEvent 	content/events/src/nsEventListenerManager.h:169
53 	libxul.so 	nsEventTargetChainItem::HandleEventTargetChain 	content/events/src/nsEventDispatcher.cpp:348
54 	libxul.so 	nsEventDispatcher::Dispatch 	content/events/src/nsEventDispatcher.cpp:682
55 	libxul.so 	DocumentViewerImpl::LoadComplete 	layout/base/nsDocumentViewer.cpp:1070
56 	libxul.so 	nsDocShell::EndPageLoad 	docshell/base/nsDocShell.cpp:6163
57 	libxul.so 	nsDocShell::OnStateChange 	docshell/base/nsDocShell.cpp:6002
58 	libxul.so 	nsDocLoader::DoFireOnStateChange 	uriloader/base/nsDocLoader.cpp:1383
59 	libxul.so 	nsDocLoader::doStopDocumentLoad 	uriloader/base/nsDocLoader.cpp:963
60 	libxul.so 	nsDocLoader::DocLoaderIsEmpty 	uriloader/base/nsDocLoader.cpp:852
61 	libxul.so 	nsDocLoader::OnStopRequest 	uriloader/base/nsDocLoader.cpp:736
62 	libxul.so 	nsLoadGroup::RemoveRequest 	netwerk/base/src/nsLoadGroup.cpp:731
63 	libxul.so 	nsDocument::DoUnblockOnload 	content/base/src/nsDocument.cpp:7251
64 	libxul.so 	nsDocument::UnblockOnload 	content/base/src/nsDocument.cpp:7193
65 	libxul.so 	nsDocument::DispatchContentLoadedEvents 	content/base/src/nsDocument.cpp:4264
66 	libxul.so 	nsRunnableMethodImpl<void , true>::Run 	nsThreadUtils.h:345
67 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:657
68 	libxul.so 	NS_ProcessNextEvent_P 	obj-firefox/xpcom/build/nsThreadUtils.cpp:245
69 	libxul.so 	mozilla::ipc::MessagePump::Run 	ipc/glue/MessagePump.cpp:110
70 	libxul.so 	MessageLoop::RunInternal 	ipc/chromium/src/base/message_loop.cc:208
71 	libxul.so 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:201
72 	libxul.so 	nsBaseAppShell::Run 	widget/xpwidgets/nsBaseAppShell.cpp:189
73 	libxul.so 	nsAppStartup::Run 	toolkit/components/startup/nsAppStartup.cpp:295
74 	libxul.so 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3703
75 	libxul.so 	GeckoStart 	toolkit/xre/nsAndroidStartup.cpp:109
76 	libmozglue.so 	__res_nsend 	other-licenses/android/res_send.c:1086
77 	dalvik-heap (deleted) 	dalvik-heap @0x5fa696 	
78 	dalvik-heap (deleted) 	dalvik-heap @0x57e466 	
79 	libdvm.so 	libdvm.so@0x11eb6 	
80 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x2088fa 	
81 	dalvik-heap (deleted) 	dalvik-heap @0x5fa696 	
82 	libdvm.so 	libdvm.so@0x43ab5 	
83 	data@[e-mail]@classes.dex 	data@[e-mail]@classes.dex@0xf24af 	
84 	libmozglue.so 	__res_nsend 	other-licenses/android/res_send.c:1071
85 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x2088fa 	
86 	libdvm.so 	libdvm.so@0x43a6b 	
87 	dalvik-heap (deleted) 	dalvik-heap @0x5fa696 	
88 	libdvm.so 	libdvm.so@0x4928b 	
89 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x2088fa 	
90 	data@[e-mail]@classes.dex 	data@[e-mail]@classes.dex@0x86b3c 	
91 	dalvik-heap (deleted) 	dalvik-heap @0x5fa696 	
92 	libdvm.so 	libdvm.so@0x1207e 	
93 	libdvm.so 	libdvm.so@0x170ca 	
94 	libdvm.so 	libdvm.so@0xa0366 	
95 	libdvm.so 	libdvm.so@0x1c29a 	
96 	libdvm.so 	libdvm.so@0x1c20a 	
97 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x1ff5b6 	
98 	libdvm.so 	libdvm.so@0x1b18a 	
99 	core.odex 	core.odex@0xd55d2 	
100 	libutils.so 	_ZN7android6LooperC2Eb 	
104 	libdvm.so 	libdvm.so@0x16daa 	
105 	libdvm.so 	libdvm.so@0x16e22 	
106 	libdvm.so 	libdvm.so@0x16cca 	
107 	libdvm.so 	libdvm.so@0x16cf2 	
108 	libdvm.so 	libdvm.so@0x16d22 	
109 	libdvm.so 	libdvm.so@0x16d46 	
110 	libdvm.so 	libdvm.so@0x79d33 	
111 	framework.odex 	framework.odex@0x206892 	
112 	framework.odex 	framework.odex@0x206898 	
113 	framework.odex 	framework.odex@0x226baa

One crash was at :
about:

Comment 1

[Scoobidiver (away)]

It first appeared in 14.0a1/20120314.

Comment 2

[David Mandelin [:dmandelin]]

Why does this block Fennec? It looks like just an OOM crash.

Comment 3

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I think it was marked because this is a startup crash...

Comment 4

[JP Rosevear [:jpr]]

Re-nomming for review, it doesn't have a lot of reports right now.

Comment 5

[Cristian Nicolae (:xti)]

I was able to reproduce this crash while I was trying to reproduce bug 750272:
https://crash-stats.mozilla.com/report/index/dec62e6c-5fc6-4f52-a90c-a550e2120430

--
Firefox 14.0a2 (2012-04-30)
Device: Samsung Captivate
OS: Android 2.2

Comment 6

[Scoobidiver (away)]

It's #12 top crasher in 14.0a2.

Comment 7

[Scoobidiver (away)]

I added a signature with a similar stack.
With combined signatures, it's #4 top crasher in 14.0a2 over the last week.

More reports at:
https://crash-stats.mozilla.com/report/list?signature=JSC%3A%3AYarr%3A%3AnewlineCreate
https://crash-stats.mozilla.com/report/list?signature=JSC%3A%3AYarr%3A%3AwordcharCreate

Comment 8

[Scoobidiver (away)]

Both crash signatures first appeared for Fennec in 14.0a1/20120314130626, just after the Maple merge.
The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8d1c74566a0b&tochange=a888f210af4e

Comment 9

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Can't seem to repro

Maybe a crash after restarting from a crash?

Comment 10

[Mark Finkle (:mfinkle) (use needinfo?)]

Can we get a birch bisect using old birch-nightlies

Comment 11

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

The start of wordcharCreate has:

   // FIXME: bug 574459 -- no NULL check

which can be hit in the case of OOM; I'd be surprised if that was the case here, but it's the right type of crash, and there isn't much else in wordcharCreate that could do this.

Comment 12

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I tried to reproduce the issue using comment 5 and I could not reproduce.

Comment 13

[Scoobidiver (away)]

It seems to be partially fixed in 15.0a1/20120509 and 14.0a2/20120509, so it's no longer a top crasher. It might also mean that users who hit it gave up (startup crashes).

Comment 14

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

still occurs in 14.0a2 20120515.

Comment 15

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

DROID3 and Droid Bionic... Experia

Comment 16

[Cristian Nicolae (:xti)]

I am still able to reproduce this issue on the latest Nightly and Aurora builds. After Fennec restarts, on Nightly you might wait more until the crash occurs. Also it's easily to reproduce it on a clean profile.

--
Firefox 15.0a1 (2012-05-15)
Device: Samsung Captivate
OS: Android 2.2

Comment 17

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

From Nicolae:

1. Open Fennec
2. Go to https://adblockplus.org/devbuilds/adblockplus/adblockplus-2.0.4a.3399.xpi (http://goo.gl/jOSYR) and install the add-on
3. When install is complete, a popup is triggered. Tap on Restart button
4. After Fennec restarts, wait

Comment 18

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Step 4 for me was to go to : chrome://adblockplus/content/ui/firstRun.xhtml and then I crashed with the following crash :
https://crash-stats.mozilla.com/report/index/bp-a7d982c8-acb6-40a7-a1cf-8792f2120517

Muy report might be different since I'm running a different build of Android OS?

Comment 19

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Oops.  Missed this one.  Will bring this up in crash triage.

Comment 20

[Sheila Mooney]

Dave, can someone in the JS team have a look at this?

Comment 21

[Tony Chung [:tchung]]

removing qawanted, STRs in comment 17

Comment 22

[Wladimir Palant]

Some notes: after the initial restart Adblock Plus will download its filters, this is probably responsible for the "wait" step. Then the filters are added to internal data structures, that's the Array.forEach() call on the stack (luckily the only place this method is used in Adblock Plus code). However, the filters will not be compiled into regular expressions during this step, that happens lazily afterwards. So the crash must be coming from one of the regular expression literals in Adblock Plus code. From what I can tell, there are only three regexps in this code path:

/^(@@)?\/.*\/(?:\$~?[\w\-]+(?:=[^,\s]+)?(?:,~?[\w\-]+(?:=[^,\s]+)?)*)?$/
/\$(~?[\w\-]+(?:=[^,\s]+)?(?:,~?[\w\-]+(?:=[^,\s]+)?)*)$/
/[^a-z0-9%*][a-z0-9%]{3,}(?=[^a-z0-9%*])/g

The first two are used across compartments after CPG landing.

Comment 23

[David Mandelin [:dmandelin]]

(In reply to Sheila Mooney from comment #20)
> Dave, can someone in the JS team have a look at this?

I looked at it myself. I guess comment 2 was far too terse. :-) Expanded version:

There are 2 crashes, which may or may not be the same thing. 

Crash 1 is the one in Socorro, e.g., from comment 0. That crash is clearly an OOM crash in the regexp compiler. It specifically happens in a function wordcharCreate that allocates a new CharacterClass object to represent \w. The code location in the crash report is not actually part of the source tree (it's a generated .h file), so I can't see exactly how it's really crashing. But AIUI, we use infallible malloc on Fennec (and Firefox), which means if we call |new| and we're OOM, we crash immediately. I.e., if we are OOM, crashing is the intended behavior.

If there is a bug, the bug would be that something is allocating too much memory. The question then becomes, "what is allocating too much memory"? I think a CharacterClass object is <256b, so it's extremely unlikely that it's that dynamic allocation site itself. It's possible that way too many CharacterClass objects get created. It could be the rest of the Yarr regexp compiler. It could be something that runs before Yarr.

Questions: How do Fennec developers generally investigate OOM crashes? Are there tools for tracking allocations? Is there way to measure what the memory usage is as the regexp compiler is starting up? Is it true that we're using infallible malloc? Also, I don't see this crash on the top 300 list, so does the Socorro crash really merit attention?

Crash 2 is the reproducible one with STR given in comment 17 and comment 18. The dump from Socorro looks like an OOM crash, but it doesn't have symbols so it's hard to tell what it is. (There aren't even enough symbols to show it has anything to do with wordcharCreate, but I'm assuming that someone somehow verified that.)

I tried to reproduce it just now on my Atrix but it didn't crash. Does it reliably reproduce, or do I have to try it multiple times?

I did notice that after trying a couple times, I somehow ended up with 10 copies of the "AdBlock Plus Installation Complete" window open. I could imagine that OOMing someone. But the Captivate has 512 MB RAM, so that seems unlikely. For fun, I opened Fennec, opened 5 copies of the install complete page, and then opened boingboing, and my Fennec memory usage is 103 MB.

Questions: Is this in fact the same crash? How reliably does it reproduce?

Comment 25

[JP Rosevear [:jpr]]

Mark, is this fixed in the new version of ABP that's coming?

Comment 26

[André Bargull [:anba]]

YARR was removed in bug 976446 - Resolving as Won't fix.