Closed Bug 559234 Opened 14 years ago Closed 14 years ago

Crash in [@ js_ExecuteRegExp]

Categories

(Core :: JavaScript Engine, defect)

1.9.2 Branch
x86
Windows XP
defect
Not set
critical

Tracking

()

RESOLVED INCOMPLETE

People

(Reporter: marcia, Assigned: cdleary)

Details

(Keywords: crash, Whiteboard: [sg:critical?][no steps to reproduce][critsmash:investigating][stale])

Crash Data

Spinoff of Bug 506586#c28. I have sifted through some of the crash data and the extensions listed in these crash reports don't contain that particular extension.

http://tinyurl.com/y35hp2q is a link to one week's worth of crash data. Many Windows crashes and few Mac/Linux.

When the correlation portion of crash stats is up again we may be able to get some more data.
Shouldn't this be closed? The original bug was.
Group: core-security
checking --- js_ExecuteRegExp 20100411-crashdata.csv
found in: 3.6.3 3.5.9 3.6 3.0.19 3.6.2 3.5.8 3.6.3plugin1
release total-crashes
              js_ExecuteRegExp crashes
                         pct.
all     309235  81      0.000261937
3.6.3   212615  62      0.000291607
3.5.9   28010   6       0.000214209
3.6     16877   5       0.000296261
3.0.19  8938    3       0.000335646
3.6.2   10376   2       0.000192753
3.5.8   3873    2       0.000516396
3.6.3plugin1    1304    1       0.000766871

os breakdown
js_ExecuteRegExpTotal 78
Win5.1  0.87
Win6.0  0.05
Win6.1  0.05
Mac10.4 0.00
Mac10.5 0.00
Mac10.6 0.01
Lin2.4  0.01

and a few test urls
   3 http://www.armagedomfilmes.net/
  1 js_ExecuteRegExp http://www.farmville.com/money.php?ref=add_coins_hud
   1 memcpy | js_ExecuteRegExp http://www.youtube.com/watch?v=xqlFAXBqxAI&feature=topvideos
user comments for this month:

js_ExecuteRegExp It's been crashing even more today than yesterday.  If it weren't for LJlogin, I would have switched everything to Chrome by now.
js_ExecuteRegExp q
js_ExecuteRegExp Stürzt ab immer mal wieder!!!
js_ExecuteRegExp Al volver tras suspender el Mac Firefox se había colgado.
js_ExecuteRegExp por que
js_ExecuteRegExp ta dando kao


that first comment might be a clue. Live Journal login from   
http://ljlogin.e-space.gweep.net/ ?
Without steps to reproduce or an especially useful stack trace, this bug shouldn't be security-sensitive.
Group: core-security
I have been working to try to get a set of STR. I followed the idea chofmann had in Comment 3 by installing the extension and creating multiple livejournal accounts but I have not had any luck yet reproducing the crash.
(In reply to comment #4)
> Without steps to reproduce or an especially useful stack trace, this bug
> shouldn't be security-sensitive.

Disagree.
Group: core-security
I have to side with sayrer here. Please don't open up bugs that reveal a particular area of the JS engine where we have a potentially exploitable bug.
not much useful in the addon data unless its some kind of combination of addons tickling the same bug.

  js_ExecuteRegExp|EXCEPTION_ACCESS_VIOLATION (51 crashes)
     24% (12/51) vs.   0% (52/134912) {a95d8332-e4b4-6e7f-98ac-20b733364387} (LeechBlock, https://addons.mozilla.org/addon/4476)
          6% (3/51) vs.   0% (16/134912) 0.4.4
         18% (9/51) vs.   0% (36/134912) 0.5
     22% (11/51) vs.   3% (3800/134912) {e4a8a97b-f2ed-450b-b12d-ee082ba24781} (Greasemonkey, https://addons.mozilla.org/addon/748)
         22% (11/51) vs.   2% (2530/134912) 0.8.20100408.6
     25% (13/51) vs.  10% (13140/134912) {CAFEEFAC-0016-0000-0016-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/) (6.0.16)
     59% (30/51) vs.  44% (59440/134912) {20a82645-c095-46ed-80e3-08825760534b} (Microsoft .NET Framework Assistant, http://www.windowsclient.net/)
          2% (1/51) vs.   3% (4026/134912) 0.0.0
         57% (29/51) vs.  41% (55357/134912) 1.1
     29% (15/51) vs.  15% (20128/134912) {CAFEEFAC-0016-0000-0019-ABCDEFFEDCBA} (6.0.19)
     35% (18/51) vs.  24% (32067/134912) {CAFEEFAC-0016-0000-0017-ABCDEFFEDCBA} (6.0.17)
     10% (5/51) vs.   4% (5253/134912) avg@igeared
         10% (5/51) vs.   2% (3200/134912) 4.002.023.004
     14% (7/51) vs.   9% (11678/134912) {3f963a5b-e555-4543-90e2-c3908898db71}
          4% (2/51) vs.   2% (3019/134912) 8.5.0.429
Using Mac and  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3, I installed LeechBlock and Greasemonkey. I then visited http://www.armagedomfilmes.net/ and basically doing those operations locks up the browser for some time. I don't crash but it a pretty bad experience. I wouldn't implicate LeechBlock because using another Mac 10.5 machine and loading the site produces the same result - the browser locks up. Loading the site on a Windows 7 and Win XP machine produces no issues. Will continue investigating.
If you can reproduce with a branch nightly, which I believe has symbols, can you get a sample from Activity Monitor?  That would give us a sense of where we're locked up.
When I run that build today using the branch nightly I am not having the issue, and I see I am getting a popup to allow a java applet, so that might be what was causing it to choke.

(In reply to comment #10)
> If you can reproduce with a branch nightly, which I believe has symbols, can
> you get a sample from Activity Monitor?  That would give us a sense of where
> we're locked up.
(In reply to comment #8)

> (Java Console, http://java.sun.com/javase/downloads/) (6.0.16)
>      59% (30/51) vs.  44% (59440/134912) {20a82645-c095-46ed-80e3-08825760534b}

thats a old console/jre version or ?
Have not been able to repro yet, investigating Google earth plugin angle after reviewing a crash report which referenced that site.
Whiteboard: [sg:critical?][critsmash:investigating]
looks like it might take 400,000 adu's to see this in small volume of a few crashes per day, as is the case with the current 3.6.4 release.

checking --- js_ExecuteRegExp 20100510-crashdata.csv
found in: 3.6.3 3.5.9 3.6 3.6b2 3.0.19 3.6b5 3.6b4 3.5.7 3.0.3 3.6.4 3.6.2 3.5 3.0.8 3.0.4 3.0.1
release total-crashes
              js_ExecuteRegExp crashes
                         pct.
all     378690  133     0.000351211
3.6.3   261522  97      0.000370906
3.5.9   33340   11      0.000329934
3.6     14860   5       0.000336474
3.6b2   617     3       0.00486224
3.0.19  11398   3       0.000263204
3.6b5   901     2       0.00221976
3.6b4   934     2       0.00214133
3.5.7   1842    2       0.00108578
3.0.3   936     2       0.00213675
3.6.4   19971   3       0.000150218
3.6.2   4669    4       0.000856714
3.5     1425    1       0.000701754
3.0.8   592     1       0.00168919
3.0.4   680     1       0.00147059
3.0.1   1886    1       0.000530223

not much help with urls here either.  they look like general browsing

domains of sites
  25 http://www.facebook.com
  15 //
  11 \N//
   8 about:blank//
   7 http://www.youtube.com
   5 http://sn104w.snt104.mail.live.com
   3 http://www.google.com
   3 http://apps.facebook.com
   2 about:sessionrestore//
   1 wyciwyg://30
   1 https://stu.edison.sso.vsb.cz
   1 https://mail.google.com
   1 https://ibank.standardchartered.com.sg
   1 https://ava.uninove.br
   1 http://yandex.ua
   1 http://xbox360iso.com
Not able to reproduce using the sites in Comment 14. Also reviewed recent 3.6.4 crashes and have not been able to reproduce on any of the sites listed. I was using  Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.4) Gecko/20100527 Firefox/3.6.4.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → INCOMPLETE
3.6.4 is up over 400k users now and seeing it there.

checking --- js_ExecuteRegExp 20100607-crashdata.csv
found in: 3.6.3 3.6 3.5.9 3.6.4
release total-crashes
              js_ExecuteRegExp crashes
                         pct.
all     378025  102     0.000269823
3.6.3   259578  84      0.000323602
3.6     11340   7       0.000617284
3.5.9   32647   6       0.000183784
3.6.4   31050   5       0.000161031
Resolution: INCOMPLETE → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
One site that came up in crash stats is: http://kp.ru/daily/24505/657769/, but I was not able to crash on that site using the latest nightly. Someone crashed on that site 3x using the 20110611 build. https://crash-stats.mozilla.com/report/index/326861de-283e-4f69-8bd9-146df2100613 is that report.
Assignee: general → cdleary
10 minidumps for this crash delivered to Chris.
Based on my findings in 6b95a84d-1827-4c8f-800e-cf2ae2100620 this is extremely obscure with only minidump data -- the exception occurs within the regexp bytecode interpreter loop/switch with an indeterminate bytecode execution history (not encoded in call stack due to loop/switch), state (arena allocated), regexp source (heap allocated), regexp bytecode (heap allocated), and input string (heap allocated).
Giving up is not an option. What is the next step?
Marking the bug as incomplete? :)
(In reply to comment #20)
> Giving up is not an option. What is the next step?

Finding STR.
Do we have a regexp fuzzer? Might be handy for yarr, too.
Jesse, could you fuzz the older branch code with your regex fuzzer?
Yeah, I'll have time next week.
Several recent comments indicate that users were crashing when they were trying to access their comcast.net email using FF 3.6.6. Anyone have such an account that we could try? Otherwise I can try reaching someone at Comcast to see if they will give us a test account.
adding stephend to the bug for the comcast.net zimbra part in case he can repro using 3.6.6.

Kaspersky Internet Security 2011 is also noted in another comment and is probably worth investigating.
I took a look at this in Windows Vista/7 with Firefox 3.6.6 on Comcast.net's Zimbra app, and couldn't reproduce the problem.  Happy to keep trying with more-specific STR.
Unfortunately the crash comments were not terribly specific, but I will keep monitoring any new comments. I will also contact Comcast and find out if users are reporting this often.

(In reply to comment #28)
> I took a look at this in Windows Vista/7 with Firefox 3.6.6 on Comcast.net's
> Zimbra app, and couldn't reproduce the problem.  Happy to keep trying with
> more-specific STR.
Here is the correlation data we have for 3.6.6 regarding extensions:

19% (10/53) vs.   0% (50/155153) {a95d8332-e4b4-6e7f-98ac-20b733364387} (LeechBlock, https://addons.mozilla.org/addon/4476)
19% (10/53) vs.   9% (13836/155153) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
6% (3/53) vs.   0% (3/155153) {963dc559-de38-7063-5cd8-064258b9695d}
6% (3/53) vs.   0% (68/155153) {CAFEEFAC-0016-0000-0000-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/)
6% (3/53) vs.   0% (532/155153) {E78313ED-E64C-451B-9B5F-8A66A8D08A64}
6% (3/53) vs.   0% (621/155153) quickstores@quickstores.de
6% (3/53) vs.   0% (696/155153) LogMeInClient@logmein.com
11% (6/53) vs.   6% (8887/155153) {b9db16a4-6edc-47ec-a1f4-b86292ed211d} (Video DownloadHelper, https://addons.mozilla.org/addon/3006)
19% (10/53) vs.  14% (21175/155153) {635abd67-4fe9-1b23-4f01-e679fa7484c1} (Yahoo! Toolbar, https://addons.mozilla.org/addon/2032)

I noticed there were a few crashes were users had no extensions installed.
Facebook is implicated in some of the crashes as well - Texas Hold Em came up as one URL, and I have seen several other stacks where individuals were operating in facebook when they crashed.
Some of the most recent crashes on the trunk had the following URLs:

http://search.speedbit.com/ - no extensions
http://zhidao.baidu.com/q?ct=24&cm=18&tn=uiframework&un=%E7%DF%B3%C7%D0%A1%CF%BA&t=1280490040773#info - no extensions
http://blog.livedoor.jp/insidears/archives/52348989.html  - several extensions
I ran my regexp fuzzer on the 1.9.2 branch for a while and didn't find anything.  I tried with and without gczeal.
Whiteboard: [sg:critical?][critsmash:investigating] → [sg:critical?][no steps to reproduce][critsmash:investigating]
Whiteboard: [sg:critical?][no steps to reproduce][critsmash:investigating] → [sg:critical?][no steps to reproduce][critsmash:investigating][stale]
Will investigate the recent crashes I see using 3.6.8 that have comments.
Is this likely to be GC-related?
I took a look at some recent 3.6./3.6.9 crash data.  Among the comments were:

*Facebook Happy Pets
*Someone crashed when logging out of the Bank of the America site
*http://wmr-sports.net/view.php?pg=streampage - "keeps clicking off" - this site requires a login

Some people report that the browser is crashing for them every few minutes. Early 3.6.9 data shows 67 crashes - http://tinyurl.com/2555txw

Correlation show 18% (9/51) vs.   0% (40/179019) {a95d8332-e4b4-6e7f-98ac-20b733364387} (LeechBlock, https://addons.mozilla.org/addon/4476)
Here are the crashes in this stack for the last week, almost all 3.6.x: http://tinyurl.com/32exggb.

We haven't had any luck at all trying to reproduce this, and I look at the new report comments every week.
One recent comment in crash stats indicates: "This is not a site problem - it is a server problem." And one user says he only crashes on this site: http://www.mcmaster.com.
we get some crashes on mcmaster.com, but in checking signatures for the last 3 days on not this signature.

count  mcmaster.com crashes

  21 UserCallWinProcCheckWow
   4 _SEH_prolog
   1 ssl3.dll@0x75
   1 nspr4.dll@0x3f
   1 msvcr80.dll@0x28e88
   1 gfxContext
   1 RtlDeactivateActivationContextUnsafeFast
   1 PK11_InitToken
   1 NavigationServices@0xf278
   1 FlushNativeStackFrame
   1 EmitPropOp
Can we get an update on where this ranks in crashes?
roughly the same as comment 16.  pretty high volume on 3.6.x.  may not be present in 4.0betas, or maybe we just don't have enough users to surface this yet.

checking --- js_ExecuteRegExp 20101018-crashdata.csv
found in: 3.6.10 3.6.8 3.6 3.6.3 3.6.11 3.0.19 3.6.7 3.6.4 3.6.6
release total-crashes
              js_ExecuteRegExp crashes
                         pct.
all     388132  102     0.000262797
3.6.10  246970  78      0.000315828
3.6.8   14390   5       0.000347464
3.6     7058    5       0.000708416
3.6.3   10388   3       0.000288795
3.6.11  7311    3       0.000410341
3.0.19  7838    3       0.000382751
3.6.7   1001    2       0.001998
3.6.4   2693    2       0.000742666
3.6.6   5659    1       0.00017671
(In reply to comment #41)
> may not be present in 4.0betas

It's not, because the entire regular expression engine has been replaced.
This is fixed on trunk by the switch to YARR.

I don't think we're ever likely to figure this out on 1.9.2 branch (and earlier) and shouldn't waste more time on it.  We've tried everything, including some time-consuming options:

* Digging through minidumps (comment 19)
* Fuzzing (comment 33)
* Following comments in crash stats (comment 36)
Status: REOPENED → RESOLVED
Closed: 14 years ago14 years ago
OS: Mac OS X → Windows XP
Resolution: --- → INCOMPLETE
This spiked to #27 total topcrash yesterday with 1232 crashes (#8 on 3.6.15 with 949 crashes on that version alone!) - it has had a quite steady stream of slightly over 100 crashes in the week before.

I guess we'll need to REOPEN and investigate that one.
Crash Signature: [@ js_ExecuteRegExp]
Group: core-security
You need to log in before you can comment on or make changes to this bug.