814954 - crash in JSC::Yarr::YarrGenerator<int>::opCompileParenthesesSubpattern

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine, defect)

Description

[Scoobidiver (away)]

It first showed up in 19.0a1/20121104. The regression range might be (discontinuous across builds):
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=2718739a1c83&tochange=5c6b71348e20

It might be an OOM according to System Memory Use Percentage in crash reports.

Signature 	JSC::Yarr::YarrGenerator<int>::opCompileParenthesesSubpattern(JSC::Yarr::PatternTerm*) More Reports Search
UUID	b9cfc62a-c635-4610-aa10-225fb2121125
Date Processed	2012-11-25 11:46:10
Uptime	184
Install Age	14.3 hours since version was first installed.
Install Time	2012-11-24 21:27:45
Product	Firefox
Version	20.0a1
Build ID	20121124030824
Release Channel	nightly
OS	Windows NT
OS Version	6.1.7601 Service Pack 1
Build Architecture	x86
Build Architecture Info	AuthenticAMD family 16 model 4 stepping 2
Crash Reason	EXCEPTION_ACCESS_VIOLATION_WRITE
Crash Address	0x77840004
App Notes 	
AdapterVendorID: 0x10de, AdapterDeviceID: 0x0605, AdapterSubsysID: 00000000, AdapterDriverVersion: 9.18.13.697
D2D? D2D+ DWrite? DWrite+ D3D10 Layers? D3D10 Layers+ 
EMCheckCompatibility	False 
Adapter Vendor ID	0x10de
Adapter Device ID	0x0605
Total Virtual Memory	2147352576
Available Virtual Memory	198828032
System Memory Use Percentage	95
Available Page File	2866634752
Available Physical Memory	170909696

Frame 	Module 	Signature 	Source
0 	mozjs.dll 	JSC::Yarr::YarrGenerator<1>::opCompileParenthesesSubpattern 	js/src/yarr/YarrJIT.cpp:2359
1 	mozjs.dll 	JSC::Yarr::YarrGenerator<1>::opCompileAlternative 	js/src/yarr/YarrJIT.cpp:2440
2 	mozjs.dll 	JSC::Yarr::YarrGenerator<1>::opCompileBody 	js/src/yarr/YarrJIT.cpp:2515
3 	mozjs.dll 	JSC::Yarr::YarrGenerator<1>::compile 	js/src/yarr/YarrJIT.cpp:2639
4 	mozjs.dll 	js::detail::RegExpCode::compile 	js/src/vm/RegExpObject.cpp:201
5 	mozjs.dll 	js::RegExpShared::compile 	js/src/vm/RegExpObject.cpp:453
6 	mozjs.dll 	js::RegExpCompartment::get 	js/src/vm/RegExpObject.cpp:582
7 	mozjs.dll 	js::NewObjectCache::invalidateEntriesForShape 	js/src/jsscope.cpp:1261
8 	mozjs.dll 	js::ExecuteRegExp 	js/src/builtin/RegExp.cpp:578
9 	mozjs.dll 	ExecuteRegExp 	js/src/builtin/RegExp.cpp:646
10 	mozjs.dll 	js::regexp_test 	js/src/builtin/RegExp.cpp:678
11 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:369
12 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2334
13 	mozjs.dll 	js::RunScript 	js/src/jsinterp.cpp:326
14 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:384
15 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:417
16 	mozjs.dll 	JS_CallFunctionValue 	js/src/jsapi.cpp:5793
...

More reports at:
https://crash-stats.mozilla.com/report/list?signature=JSC%3A%3AYarr%3A%3AYarrGenerator%3Cint%3E%3A%3AopCompileParenthesesSubpattern%28JSC%3A%3AYarr%3A%3APatternTerm*%29

Comment 1

[Scoobidiver (away)]

It's not a regression but a crash signature that has morphed.

See the previous one: https://crash-stats.mozilla.com/report/list?signature=JSC%3A%3AYarr%3A%3AYarrGenerator%3A%3AopCompileParenthesesSubpattern%28JSC%3A%3AYarr%3A%3APatternTerm*%29

Comment 2

[Scoobidiver (away)]

It spiked in 23.0a1/20130427 from one crash per build to about eight crashes per build. The regression range for the spike is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=a6104e0e5a2c&tochange=0e45f1b9521f

More reports at:
https://crash-stats.mozilla.com/report/list?signature=JSC%3A%3AYarr%3A%3AYarrGenerator%3Cint%3E%3A%3AopCompileAlternative%28JSC%3A%3AYarr%3A%3APatternAlternative*%29

Comment 3

[Scoobidiver (away)]

It's #52 browser crasher in 23.0, #70 in 24.0b1, #13 in 25.0a2, and #38 in 26.0a1.

It's correlated to Ghostery:
* 23.0:
JSC::Yarr::YarrGenerator<int>::opCompileAlternative(JSC::Yarr::PatternAlternative*)|EXCEPTION_BREAKPOINT (85 crashes)
     60% (51/85) vs.   1% (326/51156) [e-mail] (Ghostery, https://addons.mozilla.org/addon/9609)
          1% (1/85) vs.   0% (2/51156) 2.9.4
         22% (19/85) vs.   0% (185/51156) 2.9.6
         29% (25/85) vs.   0% (116/51156) 5.0.0
          7% (6/85) vs.   0% (13/51156) 5.0.1
* 24.0b1:
JSC::Yarr::YarrGenerator<int>::opCompileAlternative(JSC::Yarr::PatternAlternative*)|EXCEPTION_BREAKPOINT (14 crashes)
     36% (5/14) vs.   0% (68/22597) [e-mail] (Ghostery, https://addons.mozilla.org/addon/9609)
         14% (2/14) vs.   0% (35/22597) 2.9.6
         21% (3/14) vs.   0% (25/22597) 5.0.0

Comment 4

[Scoobidiver (away)]

It's now #20 browser crasher in 23.0, #5 in 24.0b1, #10 in 25.0a2, and #30 in 26.0a1.

Comment 5

[u279076]

Ioana, can you please see if someone on your team can reproduce this?

Comment 6

[Robert Kaiser]

This continues to be high-volume, being #5 in yesterday's 23.0 data.

Naveed, can we get someone who is somewhat familiar with Yarr to look into this? It looks like the crash is in the Yarr code we ship, even though Ghostery is the trigger.

Jorge, can we find out if a Ghostery update became available to public updates this week? If that was the case on Tuesday or Wednesday, that's the trigger for this and it may make sense to work with them to find some way of working around the problem until we can ship a fix.

Comment 7

[Jorge Villalobos [:jorgev] (he/him)]

Version 5.0.1 was released on Tuesday but it was later deleted by the developer. A new version is awaiting review, but I don't know if it addresses this problem. I contacted the developer about it.

Comment 8

[felix.shnir]

5.0.3 release fixes this issue. In 5.0.1 an infinite loop occurs between abp and ghostery.

Comment 9

[felix.shnir]

Err, forgot to mnetion, I am the dev for Ghostery.

Comment 10

[Robert Kaiser]

(In reply to felix from comment #8)
> 5.0.3 release fixes this issue. In 5.0.1 an infinite loop occurs between abp
> and ghostery.

Thanks! So is this an issue on the add-on side or is there an issue in our code behind it and you just triggered it?

Comment 11

[Manuela Muntean [Away]]

Didn't manage to reproduce the crash with Firefox 23 on a Windows 7 32bit machine.

I tried the next scenarios:

- installed the next add-ons: Adblock Plus, AutoCopy 2, BetterPrivacy, Download Helper, Ghostery 5.0.3, Greasemonkey, IE Tab, NoScript, Stylish, TableTools

- tried to send an email from a Yahoo address to a Gmail one

- open >4 tabs: espn.com, facebook (kept on scrolling), punchbowl.com, treegear.com.au, a movie on youtube, drudgereport

- opened and browsed through both asu.edu and http://www.blackboard.com/Platforms/Learn/Products/Blackboard-Learn.aspx

- selected + middle-clicked on a bookmark

- switched tabs

- looked at a book preview on Amazon, then hit the down arrow multiple times quickly

- opened ~10 tabs in 2 windows, containing: cnn.com, facebook, twitter, dealnews, a couple of forums, amazon, instagram, etc.

Comment 12

[Tracy Walker [:tracy]]

this has fallen out of topcrash on all channels. It appears ghostery 5.0.3 has indeed fixed this issue.

Comment 13

[bhavana bajaj [:bajaj]]

Given comment 12, I am removing this from our tracking lists.

Comment 14

[Jeff Walden [:Waldo]]

Uh, regardless how well Ghostery plays with AdBlock Plus, it shouldn't be able to induce a YARR crash.  This is still a valid bug, from the sounds of it, if we can get steps to reproduce it.  (Good that the Ghostery update doesn't seem to trigger this, of course -- but that doesn't mean there's nothing to fix on our side.  After all, other extensions, or web code, could conceivably trigger the exact same issue.)

Comment 15

[felix.shnir]

Jeff, a very valid point. We can provide 5.0.1 for testing, but I can describe how to replicate the issue.

Essentially, we had an issue with removing our content policy when upgrading (using restart-less, so had to clean up), for some reason the nsiCategoryManager.deleteCategoryEntry would not remove our addon during the upgrade. So we opted for a hacky solution (pitched by Mossop) that used enumerateCategories(), copied the ones into array except ours, used deleteCategory() for content policy category and then readded (addCategoryEntry) categories copied earlier.

When coupled with ABP, this procedure would eat up 100% of the CPU. As far as I understand its because ABP actually looks for the event that remove categories, and if it finds that its content policy is being removed, it will simply re-add itself. 

Why this would cause high CPU load is the root of the bug, potentially its because there would be registration for 2 content-policies from ABP, but thats a guess.

Comment 16

[DonB]

I am still seeing this issue.  I am running Ghostery 5.0.4 on Firefox 24.0 (winXP/sp3).

Comment 17

[[e-mail]]

Just crashed with Ghostery 5.0.4 and ABP 2.2.4 on Fx 25.0.1 (win7/32bit)

https://crash-stats.mozilla.com/report/index/1bc30540-2ee9-4dd8-bbea-2d2da2131120

Comment 18

[u279076]

(In reply to Ezra from comment #17)
> Just crashed with Ghostery 5.0.4 and ABP 2.2.4 on Fx 25.0.1 (win7/32bit)
> 
> https://crash-stats.mozilla.com/report/index/1bc30540-2ee9-4dd8-bbea-
> 2d2da2131120

Do you have steps to reproduce? What were you doing before Firefox crashed?

Comment 19

[Benjamin Smedberg]

This crash is now showing up more as a result of fixing the crash reporting in OOM situations (previously it was lumped with all the other unknown OOMs in EMPTY DUMP).

The crash is an EXCEPTION_BREAKPOINT at http://hg.mozilla.org/releases/mozilla-release/annotate/d20d499b219f/js/src/yarr/YarrJIT.cpp#l2480 which is:

m_ops.append(term);

and m_ops is a Vector<YarrOp, 128>

I'm not actually sure *which* Vector this is: mfbt/Vector.h and js/src/yarr/wtfbridge.h and js/public/Vector.h although dxr is linking me to the wtfbridge version so that seems most likely, which points to http://mxr.mozilla.org/mozilla-central/source/js/src/yarr/wtfbridge.h#167 as the MOZ_CRASH we're hitting.

How many ops is reasonable to have in a compiled regex like this? Should this code be using a fallible vector and checking/bailing out on OOM conditions?

Comment 20

[Robert Kaiser]

This is #11 in early 26.0 data

Comment 21

[Robert Kaiser]

This continues to be in the top 20 on 26, module correlations for yesterday have these at the top:
91% (290/317) vs.  35% (13380/37691) dhcpcsvc.dll
82% (261/317) vs.  27% (10204/37691) dhcpcsvc6.DLL
85% (271/317) vs.  32% (12053/37691) mfplat.dll
85% (270/317) vs.  32% (12002/37691) ksuser.dll
70% (222/317) vs.  19% (7095/37691) WindowsCodecs.dll

I think those are all Windows DLLs. There is no significant correlation to add-ons here, but I somewhat feel like there might be a connection to bug 856796 rising as well, both are in JSC:Yarr. Those crashes might have happened all along with EMPTY signatures and missing minidump and just be more visible now that we are able to get minidumps in many scenarios due to reserving memory for crash reporter.

Comment 22

[u279076]

Ranks #8 on Firefox 26 at 1.20%, rising 0 positions in the last week.
Ranks #11 on Beta 27 at 0.78%, rising 17 positions in the last week.
Ranks #20 on Aurora 28 at 0.80%, rising 22 positions in the last week.
Ranks #103 on Nightly 29 at 0.11%, rising 14 positions in the last week.

Comment 23

[Till Schneidereit [:till]]

I looked into this, and it's strange. The vector on which the append fails is a wrapped MFBT Vector. It should just grow when capacity overflows. A simple test case that just contains a lot of grouped elements (as in /(a)(a)..(a)/, with the groups repeated a few dozen times) confirms that this does indeed happen.

However, the reports don't look like it's an OOM crash.

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #19)
> How many ops is reasonable to have in a compiled regex like this? Should
> this code be using a fallible vector and checking/bailing out on OOM
> conditions?

Bailing out instead of crashing would mean serious changes to Yarr. It's not meant to recover from OOM at all, so just crashing in a (real) OOM situation is the expected behavior.

Comment 24

[u279076]

Current Rank
> Firefox 29: #24 @ 0.47%
> Firefox 28: #9 @ 1.25%
> Firefox 27: #8 @ 0.95%
> Firefox 26: #9 @ 1.21%

Comment 25

[Tracy Walker [:tracy]]

Fx 29: #42 @ 0.28%
Fx 28: #12 @ 0.92%
Fx 27: #5  @ 1.45%
Fx 26: #7  @ 1.44%

Comment 26

[Robert Kaiser]

FWIW, I don't see a significant correlation with anything non-Windows there right now, certainly not a Ghostery one.

Comment 27

[u279076]

This is wontfix for Firefox 26 as it's EOL with tomorrow's release of Firefox 27.

Comment 28

[u279076]

The only signature showing up in significant volume now is JSC::Yarr::YarrGenerator<int>::opCompileAlternative(JSC::Yarr::PatternAlternative*).

Top URLs:
138 	https://www.facebook.com/
134 	about:blank
49 	http://ct1.addthis.com/static/r07/sh149.html#
14 	http://templates.buscape.com/dynatemplate/msnplus_800x440.html?type=800x440
12 	http://s7.addthis.com/static/r07/sh149.html#
11 	https://mail.google.com/mail/u/0/?shva=1#inbox
10 	https://www.facebook.com/?ref=tn_tnmn
10 	http://www.youtube.com/

Top DLLs (>= 90%):
100% (796/795) vs.  75% (64627/85820) mswsock.dll
100% (795/795) vs.  76% (64969/85820) wintrust.dll
100% (795/795) vs.  73% (62889/85820) dnsapi.dll
100% (795/795) vs.  66% (56772/85820) nssckbi.dll
100% (795/795) vs.  66% (56868/85820) nssdbm3.dll
100% (795/795) vs.  66% (56870/85820) freebl3.dll
100% (795/795) vs.  66% (57012/85820) softokn3.dll
100% (795/795) vs.  67% (57397/85820) winrnr.dll
100% (794/795) vs.  74% (63184/85820) dbghelp.dll
100% (792/795) vs.  71% (61015/85820) browsercomps.dll
99% (788/795) vs.  66% (56986/85820) rsaenh.dll
99% (790/795) vs.  68% (58347/85820) rasadhlp.dll
96% (766/795) vs.  64% (54936/85820) icm32.dll
96% (764/795) vs.  69% (59487/85820) wbemprox.dll
96% (763/795) vs.  69% (59214/85820) wbemcomn.dll
95% (759/795) vs.  69% (59008/85820) wbemsvc.dll
95% (758/795) vs.  68% (58784/85820) fastprox.dll
95% (755/795) vs.  56% (47859/85820) ntmarta.dll
92% (729/795) vs.  62% (52817/85820) apphelp.dll
91% (722/795) vs.  63% (54182/85820) ntdsapi.dll
90% (717/795) vs.  36% (31171/85820) dhcpcsvc.dll

Top Add-ons:
28% (221/795) vs.  14% (12311/85820) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
6% (51/795) vs.   1% (890/85820) [e-mail]
10% (83/795) vs.   5% (4410/85820) [e-mail]
6% (44/795) vs.   0% (309/85820) [e-mail]

Some key phrases mentioned in comments:
 * HTML5 video on Youtube
 * Printing coupons
 * Uploading photos
 * Reminderfox when tagging an element as "finished" 

Flagging for QAWANTED to see if we can reproduce anything in the next Beta based in the above info.

Comment 29

[Andrei Vaida [:avaida]]

I was unable to reproduce the crash on the latest Beta (Build ID: 20140213172947), Windows 7 32-bit. 

Several attempts were made for a couple of a hours, using the crash associated add-ons and websites, but with no success.

Comment 30

[u279076]

Thanks for trying, Andrei.

Till, does the information in comment 28 provide you any new leads?

Comment 31

[Till Schneidereit [:till]]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #30)
> Till, does the information in comment 28 provide you any new leads?

Not really, sadly.

It gives more info, though: this is pretty obviously triggered by regexps in chrome JS. It might well be triggered by multiple regexps, maybe even arbitrary ones. But about:blank being in the list of URLs, it can't be caused by specific regexps used by some websites.

It'd be really helpful to have at least one of these regexps. Even that might be a dead end, though: maybe the crash is caused by something else and just triggered by whatever regexp is compiled next.

Comment 32

[u279076]

(In reply to Till Schneidereit [:till] from comment #31)
> It'd be really helpful to have at least one of these regexps. Even that
> might be a dead end, though: maybe the crash is caused by something else and
> just triggered by whatever regexp is compiled next.

Is this information we can get from the crash reports, or elsewhere?

Comment 33

[Till Schneidereit [:till]]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #32)
> Is this information we can get from the crash reports, or elsewhere?

It's not readily available anywhere. :(

The only way to get it would be to find a website in the crash reports that only has a small number of different regexps, so that it's likely that one of them is involved.

However, given that about:blank is one of the associated URLs, it might just be the case that all crashes are actually triggered by some internal, chrome regexp.

Comment 34

[u279076]

(In reply to Till Schneidereit [:till] from comment #33)
> However, given that about:blank is one of the associated URLs, it might just
> be the case that all crashes are actually triggered by some internal, chrome
> regexp.

Would it be easier to start here, since it's an internally controlled asset, as opposed to trying to investigate an external website? How would we go about investigating this?

Comment 35

[Till Schneidereit [:till]]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #34)
> Would it be easier to start here, since it's an internally controlled asset,
> as opposed to trying to investigate an external website? How would we go
> about investigating this?

Sadly, that's not how it works in this case. I think it's highly likely that the regexp that triggers this isn't in about:blank at all. It can be anywhere in chrome code at all.

I really don't know any way to investigate this further, as it seems almost impossible to create a reduced test case. Solid STR would be the only really helpful thing, I think.

Comment 36

[Robert Kaiser]

Also note that what we report as the URL is IIRC just the most recently loaded URL. There might even be some tab in the background that runs JS and causes this, or some chrome code running anywhere that triggers the crash. The URL in the crash doesn't necessarily mean that the offending code was run from that URL load, it's just meant to be a helper to make it easier to find likely culprits. This is by far not exact science.

Comment 37

[u279076]

Wontfix for Firefox 28 given we are releasing tomorrow.

Comment 38

[Rob]

(In reply to Till Schneidereit [:till] from comment #35)
> (In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #34)
> > Would it be easier to start here, since it's an internally controlled asset,
> > as opposed to trying to investigate an external website? How would we go
> > about investigating this?
> 
> Sadly, that's not how it works in this case. I think it's highly likely that
> the regexp that triggers this isn't in about:blank at all. It can be
> anywhere in chrome code at all.
> 
> I really don't know any way to investigate this further, as it seems almost
> impossible to create a reduced test case. Solid STR would be the only really
> helpful thing, I think.

The BPs below have no "Related Bugs" and while I was looking for a Dupe I spotted this Thread. Hope this assists with an STR.


When watching Videos on YouTube I had two crashes today and each one popped up the "WinXP Error Reporter" in addition to the MCR (very unusual, not seen in over a year). I do have ABP. My Flash is the newest version.

My 'Yarr BP' is bp-e2187843-0b60-4744-9b6d-0e2c82140331 and my 'non-Yarr BP' (FWIW) is bp-af2ef3a8-25ac-44d6-a36a-3292d2140331 (with both of those entirely different Signatures causing the "WinXP Error Reporter" to appear). I see no connection between the "Modules" and "Source" in those BPs but both caused the WCR to appear.


References:

Firefox 31.0a1 Crash Report [@ JSC::Yarr::Vector<JSC::Yarr::YarrGenerator<int>::YarrOp, int>::append<JSC::Yarr::YarrGenerator<int>::YarrOpCode>(JSC::Yarr::YarrGenerator<int>::YarrOpCode const&) ]
bp-e2187843-0b60-4744-9b6d-0e2c82140331

Firefox 31.0a1 Crash Report [@ nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_CopyWithMemutils>::IncrementLength(unsigned int) | `anonymous namespace''::WorkerJSRuntime::WorkerJSRuntime(JSRuntime*, mozilla::dom::workers::WorkerPrivate*) ]
bp-af2ef3a8-25ac-44d6-a36a-3292d2140331


Socorro BUG Note: In my Socorro "User Comments" I wrote: "Plugin Container.exe keeps crashing (with WinXp Error Popup) at 0(URL removed) (third time in less than 4 hours). Triggers MCR only some of the time." .

The Text "at 0(URL removed)" is where I type the crashing address (as reported by WCR) in the format 0x0ABC:0123 but Socorro thought it was a URL and unhelpfully removed it. :( It is the same address each time and WCR says it is caused by Plugin-Container.exe .


Thanks.

Comment 39

[Chris Peterson [:cpeterson]]

This YARR crash is no longer relevant because bug 976446 replaced YARR with irregexp.

Comment 40

[u279076]

Wont fix as per comment 39.