643746 - Firefox 4.0 Crash Report [@ JSObject::shrinkSlots(JSContext*, unsigned int) ] mainly with TestPilot 1.1 and below

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Marcia Knous [:marcia]]

Seen while reviewing crash stats. Has moved up to #26 slot. 

Comments: this happened 3 minutes after a zonealarm notification restarted the browser.

https://crash-stats.mozilla.com/report/index/097e6d5d-be4f-416c-8332-f56472110321

Frame 	Module 	Signature [Expand] 	Source
0 	mozjs.dll 	JSObject::shrinkSlots 	js/src/jsobj.cpp:4094
1 	mozjs.dll 	js_TraceObject 	js/src/jsobj.cpp:6536
2 	mozjs.dll 	js::gc::MarkChildren 	js/src/jsgcinlines.h:289
3 	mozjs.dll 	js::gc::MarkKind 	js/src/jsgcinlines.h:579
4 	mozjs.dll 	exn_trace 	js/src/jsexn.cpp:414

Comment 1

[chris hofmann]

Hi Jordy,  still finding more details on this but I wonder if you can check details on your side.  

We noticed an uptick in these crashes just after we pushed out the Firefox 4 final release this morning.

Comment 2

[chris hofmann]

also notice that over 90% of my sample from yesterday has
RadioWMPCore.dll around in the module list.

Comment 3

[Carsten Book [:Tomcat]]

this has jumped up 246 ranks up to a top6 topcrash for firefox 4

Comment 4

[Robert Kaiser]

This spiked very significantly on FF4 release day (yesterday), together with bug 615098 and bug 601102 (all three are in JS and started spiking on 4.0* the day before, so at least the spike cause might be related), which together account for 2800 crashes in a million 4.0* ADU on that day or about 7.4% of all 4.0* crashes on that day, all three are in the top ten top crashers for 4.0* for this day.

Comment 5

[Marcia Knous [:marcia]]

Very high correlation to Test Pilot addon - https://addons.mozilla.org/addon/13661, adding Jono to the bug.

99% (4230/4284) vs.  32% (55359/173285) [e-mail] (Mozilla Labs - Test Pilot, https://addons.mozilla.org/addon/13661)
54% (2314/4284) vs.  12% (21079/173285) [e-mail]
28% (1213/4284) vs.   1% (2110/173285) {0974848a-b5bc-49f2-9778-307742b4a55d}
14% (607/4284) vs.   3% (4371/173285) [e-mail]
10% (443/4284) vs.   1% (944/173285) {9c905b42-976e-43c1-bc30-fc5937017909}
98% (4183/4284) vs.  89% (153526/173285) {972ce4c6-7e08-4474-a285-3208198ce6fd} (Default, https://addons.mozilla.org/addon/8150)
15% (656/4284) vs.   7% (12070/173285) [e-mail]
16% (670/4284) vs.  10% (16968/173285) {20a82645-c095-46ed-80e3-08825760534b} (Microsoft .NET Framework Assistant, http://www.windowsclient.net/)

Comment 6

[Robert Kaiser]

99% of these crashes have testpilot installed, while the average is at 32%, not sure if this correlation is coincidence or not.

Comment 7

[Jakub Klawiter]

(In reply to comment #6)
> 99% of these crashes have testpilot installed, while the average is at 32%, not
> sure if this correlation is coincidence or not.

I'm sure it is because testpilot extension. I've upgraded Fx yesterday from 3.6 stable to 4.0 stable and Fx crashes everytime if test pilot extension is installed and enabled also on new clean profile. 

i've attached console output to bug # 612165

Comment 8

[chris hofmann]

Hi Jakub,

A few questions that might help to sort this out:

Do you remember where and when you might have installed the test pilot addon?

Did you ever run any one of the Firefox 4 beta's?  if so, which ones...

can you link to your crash reports?  they can be found by typing about:crashes in the location bar.

thanks

Comment 9

[Jakub Klawiter]

(In reply to comment #8)

> A few questions that might help to sort this out:
> 
> Do you remember where and when you might have installed the test pilot addon?
Yes i can, i moved to new computer on january because my laptop lost his screen :( So, i've installed Fx-3.6.13 27.01.2011 @ 1:11AM CEWT ;-)  probably few hours later i've installed test pilot on pretty new Fx profile

Than i've upgraded Fx to 3.6.15 on 07.03.2001 and than to 4.0 stable on 23.03.2001 @ 01:42:43AM CEWT ;-)

> Did you ever run any one of the Firefox 4 beta's?  if so, which ones...
Never

> can you link to your crash reports?  they can be found by typing about:crashes
> in the location bar.
Now i'm at work and have only ssh access to my private machine but i'll check and send it later. 

I'm using gentoo linux distribution on amd64 machine so i've compiled Fx myself but because problem with test pilot looks very common it probably does not matter here, anyway i'm running stable gentoo (with some gentoo testing packages) and building Fx from ebuild you can find here: http://gentoo-portage.com/www-client/firefox

Comment 10

[Jakub Klawiter]

I'm sorry for two comments one after one. 
I've configured sshd to start X11 app, and started Fx via ssh (it's not funny with 300kbps upload @ home and Pentium3 900 here but in situation when bad sniper is shooting down test pilots ... ;-) )

Anyway about:crashes says that no crash reports was sent so only what i have is console output attached to bug #612165

Comment 11

[Robert Kaiser]

bp-24990381-635c-43ef-9c94-757f52110323 is a report from Archaeopteryx, an AMO reviewer, and he told me he stopped seeing this crash when he deactivated testpilot. It was not reliably reproducible with it as well, though.

As this has js::gc::* stuff in its stack, the reproducibility problem may just come from the connection with GC.

Comment 12

[David Mandelin [:dmandelin]]

Bill, is this a dup of 643746, or do we not know that yet?

Comment 13

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

(In reply to comment #12)
> Bill, is this a dup of 643746, or do we not know that yet?

You mean bug 601102? Yes, it's almost certainly a dupe.

Comment 14

[Jono Xia]

Hi Jakub (and everybody).  Test Pilot developer here.  Sorry my add-on seems to be causing crashes!

Marcia, CHoffman - I'd like to help debug it but I can't glean anything from the crash stats or the stack trace.  What can I do to help?  Any suggestions on how I could at least narrow down what section of the JS code might be causing jsObject::shrinkSlots to trip up?

Comment 15

[chris hofmann]

so it sounds like the fix for 601102 might have resolved this, but a patch to test pilot might be able to be pushed out to users faster if there is anything that can be done there.

Comment 16

[Jono Xia]

I read some of the C++ code here and here:

http://mxr.mozilla.org/mozilla-central/source/js/src/jsobj.cpp#4086

http://mxr.mozilla.org/mozilla-central/source/js/src/jsarray.cpp#642

it looks like shrinkSlots happens when the garbage collector tries to free up memory by compacting an array that has had elements removed from it.

So I suspect it's not something that Test Pilot is calling into directly, but rather that Test Pilot is creating some kind of unusual data structure that can trip some kind of edge case in the garbage collector.  Maybe I can look through the Test Pilot code for where elements might be getting removed from arrays.  If I see anything that looks like we're creating an unusual array structure I can try to turn it into some kind of minimal test case.

Comment 17

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Here's some info about the crash that might help to work around the problem in Test Pilot. The problem seems to happen when Exception objects are long-lived. Based on the stack trace, it seems like Test Pilot might be storing some Exception objects in an array and keeping them around for a while. Shortening the lifetime of Exceptions will make the crash less likely to occur.

Luke, when you added the compartment assertion, do you remember where it triggered?

Comment 18

[Jono Xia]

Boom, that's it. Yes, to make remote debugging easier I started storing exception objects raised during the study so that I can upload the exception strings along with the data submission.

There's no reason I need to save the exception objects themselves, though.  I can just copy out the relevant strings and drop the reference to the exception object itself.

Thanks a lot Bill!

Comment 19

[Jono Xia]

Started a new bug for this in the Test Pilot component: https://bugzilla.mozilla.org/show_bug.cgi?id=646122

Comment 20

[Luke Wagner [:luke]]

Bill: in InitExnPrivate, exactly where now we have the compartment check (on TM tip).

Comment 21

[Jakub Klawiter]

I'm not sure if it can help but as I understand you have some problems with reproducing this bug. So probably I have different problem also connected to test Pilot because:

- My Fx-4.0 hangs every time when Test Pilot is installed and enabled also on fresh profile where TestPilot-1.1 
- My Fx just hangs with some magic ;-) output to console but without crash report sent to Mozilla 

BTW isn't good idea to upgrade test pilot to 1.2 with only difference that it should be set as NOT compatible with Fx-4.x?

Comment 22

[Jono Xia]

OK, fixed in Test Pilot in http://hg.mozilla.org/labs/testpilot/rev/1a4ac5185e48
Test Pilot no longer holds on to exception objects.

Comment 23

[Jono Xia]

Jakub: Could you tell me what the "magic" output to the console is?  Rather than marking Test Pilot 1.2 incompatible with Fx 4, I would rather *make it compatible* with Fx 4!!

Comment 24

[Jakub Klawiter]

Attachment: Console output, clean profile, test pilot + gentoo x86

I'm attaching console output from work (here where I am now). It's also gentoo linux on x86 machine. Fx is hanging after few seconds every time if Test Pilot is installed and enabled.

As i said before same output from my private (amd64) machine is attached to bug #612165 

About compatibility of Test pilot. Of course it is better to make it working with Fx4 but in case it is not trivial maybe it is good idea to turn it off for some time (time to find and kill the bug). Many ppl may think that they lost their profiles after upgrade to Fx-4.0.

Comment 25

[Jono Xia]

Jakub:  Oh, I didn't realize this was the same as bug #612165.

I think your problem is unrelated to the shrinkSlots crash.  I believe the shrinkSlots crash is now fixed, so let's close this bug and take discussion of your crash over to #612165.