506586 - Firefox 3.5.1 & 3.6.x crashes [@ js_ExecuteRegExp] with Rewind/FastForward Buttons extension installed

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Robert Krawitz]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.1) Gecko/20090714 SUSE/3.5.1-1.1 Firefox/3.5.1
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.1) Gecko/20090714 SUSE/3.5.1-1.1 Firefox/3.5.1

Firefox 3.5.1 crashes with a seg fault with some reliability when the Rewind/FastForward Button extension is installed.  It appears to be site-specific and I haven't narrowed down the precise site that triggers it (I have about 25 open tabs).  However, I believe the login link from http://wunderground.com may be one of the trigger links.

This happens with both the official Firefox 32-bit package and the 64-bit OpenSUSE 11.1 package.  It did not happen with 3.5.0.

This has been discussed extensively on that extension's page.  See https://addons.mozilla.org/en-US/firefox/addon/1934

Reproducible: Sometimes

Steps to Reproduce:
Start Firefox with a large collection of tabs loaded (not clear which one or ones is the culprit).
Actual Results:  
Firefox crashes with a seg fault in libmozjs.so.

Expected Results:  
Firefox should load the page.

Unfortunately, I have not been able to generate a talkback -- for some reason, talkback fails when I'm at home.

Interestingly enough, even though I have 3.5.1 installed since July 18 (and the current version of Rewind/FF since late June), I did not observe this crash until today, at which point I observed it on two separate machines with different sets of tabs and different extensions.  I have not installed or upgraded any addons in several days.

I'm submitting this as a security bug on the grounds that any browser seg fault, particularly in JavaScript, may potentially be exploitable.  I have no particular knowledge of whether this one is.

I don't want to confuse matters unnecessarily, but in the process of doing a binary chop on the addons, I found I could induce a crash in an empty browser with TamperData installed simply by typing "clear private data firefox" into the search bar and typing <enter>.  Disabling TamperData stopped that crash.  However, Rewind/FF was installed in both cases.  I have not performed a further binary chop to try to narrow that down further.

Comment 1

[Robert Krawitz]

I have a clean reproducible test case, using the mozilla.com tarball.

% firefox -P
  - create new profile named blah
  - install Rewind/FastForward Buttons addon
  - restart
  - Confirm adding new buttons to the toolbar
  - type "clear user data firefox" into the search bar (no quotes)
  - type <enter>
  - Watch Firefox go byebye

Comment 2

[Martijn Wargers (dead)]

This is worksforme, do you have a stacktrace or a breakpad report?
Type about:crashes in the url bar to see the list of breakpad reports.

Comment 3

[Robert Krawitz]

Very curious; I cannot reproduce this at work.  However, at home it reproduces reliably.

A couple of other curiosities:

1) At home, the crash reporter does not work (it says that something went wrong while trying to report it, but there's no real diagnostic).

2) Fairly frequently when I'm at home, when I create a new profile, attempts to access the net fail claiming that the proxy server cannot be found.  I don't use a proxy server and do not know where Firefox is finding it.  When I change the proxy setting to direct connect to the internet, it works.  But crash reporter still fails to send in a report (it's able to do so at work).

3) I have two machines at home: one really is directly connected to the internet, and the other (my laptop, which is what I have at work) is NAT'ed through the first (with a firewall running on the first machine).  The crash happens on both.

I will attach the .dmp and .extra files from the most recent crash.  That, I'm afraid, is the best I will be able to do unless someone can provide me with debug binaries or something.

The stack trace I got from gdb wasn't very useful, because the libraries are stripped.  However, it appeared to be 6 or so levels in to libmozjs.so.

Comment 4

[Robert Krawitz]

Attachment: .dmp file from most recent crash

Comment 5

[Robert Krawitz]

Attachment: .extra file from most recent crash

Comment 6

[Martijn Wargers (dead)]

Robert, those files are not useful, unfortunately. Don't you have a breakpad id, perhaps?

Comment 7

[Robert Krawitz]

Like I said: when it crashes at home, it tries but fails to submit the crash report, so there's no ID.  Is there any other way to get the information?  When it crashes at work it does submit the crash report, but in this instance I'm unable to make it crash at work.

Comment 8

[Martijn Wargers (dead)]

So the breakpad reporter does come up, but for some reason no crash report gets submitted? Maybe that's bug 430469?
I don't know if there is a way to process these dump files, ted?

Comment 9

[(not currently active) Ted Mielczarek]

Not currently, no. I have a bug that is close to landing that will let you submit pending reports directly from about:crashes.

Comment 10

[Robert Krawitz]

The real question, to my mind, is why the proxy issue is coming up at all.  I don't have FF configured for a proxy, and there are no proxy environment variables set.  Is there some hidden file somewhere that would cause FF to think by default that there is a proxy in use where in fact there isn't?

When I select "use system proxy" (which seems to be the default), where does it get proxy information from on Linux?

Comment 11

[Robert Krawitz]

Well, I figured out where it was getting the proxy information from.  I use KDE, but my GNOME configuration still had a proxy file set.  The 32-bit FF I downloaded from mozilla.com was using it (and so getting confused); the 64-bit OpenSUSE FF wasn't using it.

And now I can't seem to reproduce it at all on my laptop.  I don't want to close this yet, because I've experienced it on two separate systems and many other people have reported it (read the reviews on the extension's page, above).  I will make further attempts to reproduce it.

Comment 12

[Robert Krawitz]

At this point I cannot reproduce it at all, on either system.

Given that it happened on two separate builds of Firefox (the official 3.5.1 and the OpenSUSE 64-bit build) and on two separate machines, other people have reported the same thing, it survived a couple of reboots and happened on clean profiles, and at least for a while was very easy to reproduce, I'm a bit reluctant to let this be closed.  On the other hand, I'm also familiar with chasing ghosts, and as long as it can be reopened, it's hard for me to insist otherwise.  I haven't installed any new packages since this happened.  I'd hope that something could eventually be done with the crash data that I have.

I've re-enabled rewind/fast-forward on both of my systems in the hope that it triggers again.  I think I've solved the problem that prevented crash reporting from working on my laptop, so if it does, you should get a crash dump.

I'm not actually using any proxy or shared cache, and I don't believe my ISP does, either.

Comment 13

[Robert Krawitz]

Got one!

http://crash-stats.mozilla.com/report/index/551bdb66-f65e-4ac5-a65b-5e3752090730

Comment 14

[Robert Krawitz]

I have a whole bunch more, all of a sudden.  Needless to say, I definitely do NOT want this closed!

This one is very strange indeed.  It goes for a while without happening at all, then all of a sudden happens consistently for a day or so, with either 32 or 64 bit Firefox.  When it happens, it happens on a clean, brand-new profile -- just so long as I install rewind/fastforward buttons.

The *precise* sequence of operations I did:

firefox -P

- deleted the existing profile named "blah" and removed its files
- created a new profile named "blah"
- opened Tools->Extensions
- searched for rewind/fast forward buttons
- installed it
- restarted firefox
- Clicked Yes to install the rewind/fast forward buttons
- Typed "clear private data firefox" into the search bar
  + This time it came up without crashing
- Clicked the Rewind button.  This time it crashed instantly -- same signature.

I suggest that you look at this soon -- it's entirely possible that this will stop happening in 12-24 hours.  That's very, very, VERY strange!

Some other breakpad ID's:

c989c452-b9e5-49f8-a179-bb7122090730	07/30/2009	09:57 PM
dedf2c2e-0a75-4faf-8a33-56aae2090730	07/30/2009	09:57 PM
0052b0bb-a6d9-475d-a29e-ade342090730	07/30/2009	09:55 PM
1768f217-14c7-4868-9c44-c377d2090730	07/30/2009	09:55 PM
551bdb66-f65e-4ac5-a65b-5e3752090730	07/30/2009	09:49 PM
1a056269-ee1d-42d1-9eb2-a27c32090730	07/30/2009	09:49 PM

Comment 15

[Martijn Wargers (dead)]

Ok, thanks for the breakpad reports, it looks like you're crashing in the js engine.

http://crash-stats.mozilla.com/report/index/0052b0bb-a6d9-475d-a29e-ade342090730?p=1
0  	libmozjs.so  	js_ExecuteRegExp  	 js/src/jsregexp.cpp:3898
1 	libmozjs.so 	regexp_exec_sub 	js/src/jsregexp.cpp:4889
2 	libmozjs.so 	regexp_test 	js/src/jsregexp.cpp:4911
3 	libmozjs.so 	js_Interpret 	js/src/jsinterp.cpp:5147
4 	libmozjs.so 	js_Invoke 	js/src/jsinterp.cpp:1394
5 	libxul.so 	nsXPCWrappedJSClass::CallMethod 	js/src/xpconnect/src/xpcwrappedjsclass.cpp:1697
6 	libxul.so 	nsXPCWrappedJS::CallMethod 	js/src/xpconnect/src/xpcwrappedjs.cpp:561
7 	libxul.so 	PrepareAndDispatch 	xpcom/reflect/xptcall/src/md/unix/xptcstubs_gcc_x86_unix.cpp:95
8 	libxul.so 	nsBrowserStatusFilter::OnLocationChange 	xpfe/browser/src/nsBrowserStatusFilter.cpp:231
9 	libxul.so 	nsDocLoader::FireOnLocationChange 	uriloader/base/nsDocLoader.cpp:1302
10 	libxul.so 	nsDocShell::CreateContentViewer 	docshell/base/nsDocShell.cpp:6457
11 	libxul.so 	nsDSURIContentListener::DoContent 	docshell/base/nsDSURIContentListener.cpp:138
12 	libxul.so 	nsDocumentOpenInfo::TryContentListener 	uriloader/base/nsURILoader.cpp:736
13 	libxul.so 	nsDocumentOpenInfo::DispatchContent 	uriloader/base/nsURILoader.cpp:434
14 	libxul.so 	nsDocumentOpenInfo::OnStartRequest 	uriloader/base/nsURILoader.cpp:280
15 	libxul.so 	nsHttpChannel::CallOnStartRequest 	netwerk/protocol/http/src/nsHttpChannel.cpp:846
16 	libxul.so 	nsHttpChannel::ProcessNormal 	netwerk/protocol/http/src/nsHttpChannel.cpp:1128
17 	libxul.so 	nsHttpChannel::ProcessResponse 	netwerk/protocol/http/src/nsHttpChannel.cpp:1053
18 	libxul.so 	nsHttpChannel::OnStartRequest 	netwerk/protocol/http/src/nsHttpChannel.cpp:4868
19 	libxul.so 	nsInputStreamPump::OnStateStart 	netwerk/base/src/nsInputStreamPump.cpp:439
20 	libxul.so 	nsInputStreamPump::OnInputStreamReady 	netwerk/base/src/nsInputStreamPump.cpp:395
21 	libxul.so 	nsInputStreamReadyEvent::Run 	xpcom/io/nsStreamUtils.cpp:111
22 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:510
23 	libxul.so 	NS_ProcessNextEvent_P 	nsThreadUtils.cpp:227
24 	libxul.so 	nsBaseAppShell::Run 	widget/src/xpwidgets/nsBaseAppShell.cpp:170
25 	libxul.so 	nsAppStartup::Run 	toolkit/components/startup/src/nsAppStartup.cpp:193
26 	libxul.so 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3298
27 	firefox-bin 	main 	browser/app/nsBrowserApp.cpp:156
28 	libc-2.9.so 	libc-2.9.so@0x16704

Comment 16

[Robert Krawitz]

Ayup, and in particular every one I looked at crashed at the same line in the regexp code.

Comment 17

[Robert Krawitz]

Reproduced on a Mac running OS X 10.5.7.  Signature looks similar but not identical.

6c340285-1836-4fba-b54e-cfa482090731	7/31/09	12:02 PM
2dff88b2-e41f-48cb-bb16-5bb962090731	7/31/09	12:01 PM

0  	libmozjs.dylib  	MatchRegExp  	 js/src/jsregexp.cpp:3659
1 	libmozjs.dylib 	js_ExecuteRegExp 	js/src/jsregexp.cpp:4090
2 	libmozjs.dylib 	regexp_exec_sub 	js/src/jsregexp.cpp:4889
3 	libmozjs.dylib 	regexp_test 	js/src/jsregexp.cpp:4911
4 	libmozjs.dylib 	js_Interpret 	js/src/jsinterp.cpp:5147
5 	libmozjs.dylib 	js_Execute 	js/src/jsinterp.cpp:1622
6 	libmozjs.dylib 	JS_EvaluateUCScriptForPrincipals 	js/src/jsapi.cpp:5145
7 	XUL 	nsJSContext::EvaluateString 	dom/src/base/nsJSEnvironment.cpp:1631
8 	XUL 	nsGlobalWindow::RunTimeout 	dom/src/base/nsGlobalWindow.cpp:7799
9 	XUL 	nsGlobalWindow::TimerCallback 	dom/src/base/nsGlobalWindow.cpp:8152
10 	XUL 	nsTimerImpl::Fire 	xpcom/threads/nsTimerImpl.cpp:420
11 	XUL 	nsTimerEvent::Run 	xpcom/threads/nsTimerImpl.cpp:512
12 	XUL 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:510
13 	XUL 	NS_ProcessPendingEvents_P 	nsThreadUtils.cpp:180
14 	XUL 	nsBaseAppShell::NativeEventCallback 	widget/src/xpwidgets/nsBaseAppShell.cpp:121
15 	XUL 	nsAppShell::ProcessGeckoEvents 	widget/src/cocoa/nsAppShell.mm:405
16 	CoreFoundation 	CFRunLoopRunSpecific 	
17 	CoreFoundation 	CFRunLoopRunInMode 	
18 	HIToolbox 	RunCurrentEventLoopInMode 	
19 	HIToolbox 	ReceiveNextEventCommon 	
20 	HIToolbox 	BlockUntilNextEventMatchingListInMode 	
21 	AppKit 	_DPSNextEvent 	
22 	AppKit 	-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] 	
23 	AppKit 	-[NSApplication run] 	
24 	XUL 	nsAppShell::Run 	widget/src/cocoa/nsAppShell.mm:720
25 	XUL 	nsAppStartup::Run 	toolkit/components/startup/src/nsAppStartup.cpp:193
26 	XUL 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3298
27 	firefox-bin 	main 	browser/app/nsBrowserApp.cpp:156
28 	firefox-bin 	firefox-bin@0x1541 	
29 	firefox-bin 	firefox-bin@0x1468 	
30 		@0x2

Comment 18

[David Mandelin [:dmandelin]]

I can duplicate this using the instructions in comment 14. (I'm using TM rev 46ed2bc15e56). It's failing while running a 84358-character non-jitted regexp. I know little about that code. Can someone get us a regression range (in any repo/buildset)?

Comment 19

[Robert Krawitz]

Ref. comment 18: I suggest saving the exact regexp just in case this problem becomes non-reproducible again, like I observed for several days.

Comment 20

[David Mandelin [:dmandelin]]

Attachment: Regexp source text at crash point

Comment 21

[David Mandelin [:dmandelin]]

Attachment: Input string at crash point

The crash point seems to be in a call to .test(). There is a flag set on the regexp of JSREG_NOCOMPILE, which doesn't affect the regular expression semantics and is not user-controllable, but tells the regexp->native compiler not to try to compile the regexp.

Comment 22

[David Mandelin [:dmandelin]]

I tried running the regexp I sampled against the input I sampled in the shell, and it worked fine with JIT on or off.

Comment 23

[Robert Krawitz]

That regexp is the user_pref named
"rewindforward.siteinfo.http%3A%2F%2Fwedata.net%2Fdatabases%2FAutoPagerize%2Fitems.json.cache".  There are a number of other similar preferences.  I wonder if RewindFastForward is downloading an updated list periodically, and sometimes it works and sometimes not.

Comment 24

[Robert Krawitz]

Attachment: prefs.js related to rewind/fastforward

In case it helps (e. g. if it magically stops failing at some point), these are my prefs related to the plugin.

Comment 25

[Robert Krawitz]

Apparently the developer of the extension is aware in some sense of this problem -- note this comment on http://piro.sakura.ne.jp/xul/_rewindforward.html.en :

2.1.2009072701

        * Fixed: Crash caused by too long regular expressions disappeared.

Nonetheless, a crash caused by a long regular expression seems a serious matter.

Comment 26

[David Mandelin [:dmandelin]]

(In reply to comment #25)
> Apparently the developer of the extension is aware in some sense of this
> problem -- note this comment on
> http://piro.sakura.ne.jp/xul/_rewindforward.html.en :
> 
> 2.1.2009072701
> 
>         * Fixed: Crash caused by too long regular expressions disappeared.
> 
> Nonetheless, a crash caused by a long regular expression seems a serious
> matter.

How can I get the pre-fix version of the plugin? That's the only way I know of reproducing the problem.

Also, a regression range from nightlies would be incredibly helpful here.

Comment 27

[Robert Krawitz]

The version on mozilla.org is still the one that triggers this problem (https://addons.mozilla.org/en-US/firefox/addon/1934)

Comment 28

[chris hofmann]

looks like we still get about 70-130 crashes reported per day with the windows signature, and 20-36 crashes per day on the Mac signature and the problem seems a bit worse on fx 3.6

checking --- 20100215-crashdata.csv js_ExecuteRegExp
release total-crashes
              js_ExecuteRegExp crashes
                         pct.
all     254301  132     0.00051907
3.0.15  1205            0
3.0.16  447     1       0.00223714
3.5.5   3205    1       0.000312012
3.5.6   1838    1       0.00054407
3.5.7   117488  38      0.000323437
3.6     82800   80      0.000966184

people seem to also be hitting this on the facebook mafia wars app

domains of sites
  35 \N//
  12 http://www.facebook.com
   9 //
   5 http://mwfb.zynga.com/mwfb
   4 http://apps.facebook.com
   4 about:blank//
   3 http://foto.mail.ru   photo/addphoto 
   3 about:sessionrestore//
   2 http://www.tagged.com  http://www.tagged.com/friends.html#
   2 http://www.gurulib.com

Comment 29

[Marcia Knous [:marcia]]

I am working on trying to find a regression range for this bug. My initial attempts to reproduce the crash have not been successful. Will update this bug when I have more info.

Comment 30

[Marcia Knous [:marcia]]

I filed Bug 559234 to address the crashes that chofmann notes in Comment 28.

Comment 31

[Marcia Knous [:marcia]]

I emailed the extension developer and will report when I hear back from him.

Comment 32

[YUKI "Piro" Hiroshi]

This bug was fixed in the version 2.1.2009072701. But I don't know whetehr the real cause of this problem was fixed by Mozilla or not.

My addon dynamically loads SITEINFOs from the URL: http://wedata.net/databases/AutoPagerize/items.json to detect links to next/previous pages from various services. Because there are too many SITEINFOs, it possibly take much time that I detect which SITEINFO should be use for the current page if I use simple loop (for-in, Array.prototype.forEach/some, etc.) So, the addon generates a long regexp to judge whether there is a SITEINFO for the page or not.

The regexp is getting long everyday, because http://wedata.net/databases/AutoPagerize/items.json is an wiki. On 2009-07-26, added new entry to the website breaks my system. Firefox seemed to be crashed from stack overflow from generated too long regexp.

On the version 2.1.2009072701, I changed the method to generate long regexp. I split operations for each 250 SITEINFOs, so this problem gone.

However, I don't know that the real cause was solved or not. If I re-generate very very long regexp, the problem possibly appears on the latest trunk. The unit "250" can be changed by a secret pref "rewindforward.siteInfo.unit", so I'll try to reproduce this problem on lately Firefox builds.

Comment 33

[Marcia Knous [:marcia]]

Hello Shimodo - Have you had a chance to try to reproduce on nightlies? Thanks.

Comment 34

[Marcia Knous [:marcia]]

Shimodo: Can we get a copy of that old extension and can you isolate the code that might be causing this problem? Thanks in advance for your help!

Comment 35

[chris hofmann]

Once we isolate the code can try to look for the same pattern in other addons and maybe add checks to the addon verfication scanner.

Comment 36

[YUKI "Piro" Hiroshi]

Attachment: Rewind/FastForward Buttons rev.4712

This is the previous revision before I fixed the crash bug.

Comment 37

[YUKI "Piro" Hiroshi]

Attachment: Diff: rev.4712 vs rev.4810

And, I fixed the bug on the rev.4810. This is the diff.

Comment 38

[YUKI "Piro" Hiroshi]

By the way, this revision has a security issue. (Bug 515529) This is the reason why I delete old XPIs from AMO.

Comment 39

[chris hofmann]

ok, so maybe this is a dup of Bug 515529, or the test case/confirmation for that bug.

eval checks are already in the verifier so I think my request in comment 35 has been met.

Comment 40

[Marcia Knous [:marcia]]

Based on Comment 39, can this bug be closed out?

Comment 41

[Marcia Knous [:marcia]]

Do we still need to test this with the old version of the extension, or as I noted in Comment 39 can this bug be closed out?

Comment 42

[chris hofmann]

I can't see any reason to test the old version now that we have a handle on what the exact problem was, and also now that we do verifier scanning for eval.

it's time to close this bug.