561289 - Firefox 3.6.4 and other release crash [@ js_DeepBail(JSContext*) ]

Status: RESOLVED INCOMPLETE

(Core :: JavaScript Engine, defect)

Description

[chris hofmann]

Spotted while poking around in 3.6.4 crash data, but it appear to have been around awhile.   Its possible that this is a crash volume regression in 3.6.4

checking --- js_DeepBail 20100421-crashdata.csv
found in: 3.6.3 3.6.3plugin1 3.6.4 3.5.9 3.6.2
release total-crashes
              js_DeepBail crashes
                         pct.
all     383147  51      0.000133108
3.6.3   257898  29      0.000112448
3.6.3plugin397  12      0.0302267
3.6.4   16781   7       0.000417138
3.5.9   34290   2       5.8326e-05
3.6.2   7133    1       0.000140193

some changes were landed in jstracer.cpp in the last few days for bug 547299  so we should just confirm they haven't  tickled this area of the code induced some more crashes here.  we can also watch to see if the volume levels out as the beta population grows for 3.6.4.   security closed since bug 547299 is closed right now.

stack looks like

http://crash-stats.mozilla.com/report/index/6637a2ad-11fa-4156-a93a-5319a2100418

0  	js3250.dll  	js_DeepBail  	 js/src/jstracer.cpp:7565
1 	xul.dll 	nsXPCWrappedJSClass::CallMethod 	
2 	xul.dll 	nsXPCWrappedJS::CallMethod 	js/src/xpconnect/src/xpcwrappedjs.cpp:570
3 	xul.dll 	PrepareAndDispatch 	xpcom/reflect/xptcall/src/md/win32/xptcstubs.cpp:114
4 	xul.dll 	SharedStub 	xpcom/reflect/xptcall/src/md/win32/xptcstubs.cpp:141
5 	xul.dll 	nsComponentManagerImpl::AutoRegisterComponent 	xpcom/components/nsComponentManager.cpp:3147
6 	xul.dll 	nsLocalFile::QueryInterface 	xpcom/io/nsLocalFileWin.cpp:790
7 	xul.dll 	nsLocalFile::IsDirectory 	xpcom/io/nsLocalFileWin.cpp:2457
8 		@0x937e1f

more reports at
http://crash-stats.mozilla.com/report/list?range_value=2&range_unit=weeks&signature=js_DeepBail%28JSContext*%29&version=Firefox%3A3.6.4

os breakdown shows its tilted more toward windows 7
js_DeepBailTotal 51
Win5.1  0.35
Win6.0  0.12
Win6.1  0.53

Comment 1

[Marcia Knous [:marcia]]

Seen on the trunk as well: http://tinyurl.com/2boy62z. All reports that day were Win XP. No URLs available for that group.

Comment 2

[chris hofmann]

I'll grab correlation data.

Comment 3

[chris hofmann]

only slight higher frequency of several common addons.

  js_DeepBail(JSContext*)|EXCEPTION_ACCESS_VIOLATION (13 crashes)
     23% (3/13) vs.   9% (19056/218672) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
          0% (0/13) vs.   0% (1/218672) 0.7.5.5
          0% (0/13) vs.   0% (2/218672) 1.0.1
          0% (0/13) vs.   0% (5/218672) 1.0.2
          0% (0/13) vs.   0% (333/218672) 1.1.1
          0% (0/13) vs.   0% (264/218672) 1.1.2
         23% (3/13) vs.   8% (18450/218672) 1.1.3
          0% (0/13) vs.   0% (1/218672) 1.1.3+.2010041702
     15% (2/13) vs.   4% (8313/218672) avg@igeared
          0% (0/13) vs.   0% (1/218672) 2.709.018.001
          0% (0/13) vs.   0% (3/218672) 2.710.016.005
          8% (1/13) vs.   2% (3583/218672) 3.011.025.005
          8% (1/13) vs.   2% (4726/218672) 4.002.023.004
     23% (3/13) vs.  12% (26588/218672) {CAFEEFAC-0016-0000-0015-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/) (6.0.15)
     15% (2/13) vs.   5% (11339/218672) {CAFEEFAC-0016-0000-0020-ABCDEFFEDCBA} (6.0.20)
     54% (7/13) vs.  44% (95520/218672) {20a82645-c095-46ed-80e3-08825760534b} (Microsoft .NET Framework Assistant, http://www.windowsclient.net/

Comment 4

[chris hofmann]

only 1/3 of these appear near startup and/or repeat crashes.

Correlation to startup or time of session
18 total crashes for js_DeepBail on 20100425-crashdata.csv
2 start up crashes inside 30 seconds of startup
4 start up crashes inside 3 minutes of startup
5 start up crashes inside 3 minutes of last crash

Comment 5

[Marcia Knous [:marcia]]

marcia will try to repro based on URL information.

Comment 6

[Damon Sicore (:damons)]

Any progress here?

Comment 7

[Marcia Knous [:marcia]]

Have explored some URLs on Win 7 and have not yet reproduced a crash. There is a comment related to hulu and playing videos but no luck on that site. http://www.windfinder.com/ was another site that involves webcams, so that may be worth pursuing. Will follow up with more information.

Comment 9

[David Anderson [:dvander] - inactive, e-mail if emergency]

According to the crash reports we're dying in DeepBail():
    LeaveTree(tm, *tracecx->tracerState, tracecx->bailExit);

All of the addresses are in [0x204, 0x210] or so. Sounds like tracecx is NULL. I can't see how this could be exploitable. That's not good enough to open the bug though, DeepBail()'s sole two callers do a NULL check on tracecx.

Comment 10

[David Anderson [:dvander] - inactive, e-mail if emergency]

How can I get access to minidumps for these crashes?

Comment 11

[Damon Sicore (:damons)]

David, did you get your minidumps yet?  If not, walk over and ask JST.  He'll get them for you.

Comment 12

[Damon Sicore (:damons)]

Please tell me you have minidumps now (and analyzed them)?  :)

Comment 13

[Jesse Ruderman]

I noticed DeepBail in the summary. Is this  the kind of bug that could be found by fuzzing with a patch for bug 569451?

Comment 14

[David Anderson [:dvander] - inactive, e-mail if emergency]

critsmash update: I did get the minidumps. Sorry, I have not looked into them yet. I will do so today.

Jesse: I suspect there is no GC foul play here.

Comment 15

[Damon Sicore (:damons)]

(In reply to comment #14)
> critsmash update: I did get the minidumps. Sorry, I have not looked into them
> yet. I will do so today.
> 
> Jesse: I suspect there is no GC foul play here.

dvander rules.

Comment 16

[David Anderson [:dvander] - inactive, e-mail if emergency]

Of the minidumps (thanks jst!)

1. The value of |tracecx| is 0x00002000.

2. The stack is hopelessly corrupted... it looks like somehow we jumped into JS_FrameIterator and died. |tracecx| is garbage.

3. |tracecx| is NULL.

For the rest, I don't get symbols for js3250.dll. (Does our symbol server *still* not download binaries?) More minidumps would help here, if any are available.

Brief analysis: The fact that |tracecx| in the first is non-NULL is troubling, but the fact that it _is_ NULL in other cases is even more troubling. There is no path such that this is possible, unless the compiler has so inlined and optimized that there is a mistake somewhere. I don't have enough data to deduce this yet.

If there are no compiler problems, the only way this could happen (barring actual hardware bugs) is if the memory has been touched in another thread in between the js_LeaveTrace check and the js_DeepBail check.

Comment 17

[Johnny Stenback (:jst)]

More minidumps handed off to dvander.

Comment 18

[Damon Sicore (:damons)]

David, any ETA on this?

Comment 19

[David Anderson [:dvander] - inactive, e-mail if emergency]

Sorry, this fell off the radar. I will post an update later today.

Comment 20

[David Anderson [:dvander] - inactive, e-mail if emergency]

... wherein "today" occurs a week later. Brief analysis of some of the crash dumps.

1b7eb6b7-a6d7-4d98-9f47-affad2100618:
This one seems to be happening from ShutdownXPCom -> NPAPIPluginInstance::Stop. The crash is:

    tm->tracecx = NULL;
0036CE38  mov         dword ptr [eax],0

This sheds new light, as JSTraceMonitor is corrupt, which means either the JSThreadData or JSContext are also corrupt.

2a187cf8-72f0-4841-a90b-1ebc12100623: This one has some weird DLL loaded - FFExternalAlert.dll.

5f227f98-d1ed-4139-a189-278c82100618: DeepBail crash, but NPSWF32 is in the callstack. Maybe another case of Flash jumping into random memory. TraceMonitor is garbage.

9e268257-fc61-45ae-ac3e-6f3352100624: Another one in nsNPAPIPluginInstance::Stop(). NPMYWEBS.DLL, M3PLUGIN.DLL present. This one crashes on a bad tracecx.

70afd277-3ccb-4c09-ba5c-da6092100623: Another one in nsNPAPIPluginInstance::Stop() -> JS_EndRequest. IBitCometExtension.dll and a few other weird DLLs present. tracecx is 0x00100000.

Comment 21

[Damon Sicore (:damons)]

Can we see if this crash correlates to an extension?  We seem to have enough crashes.

Comment 22

[chris hofmann]

there is some addon correlation data in comment 3, and no current correlation data for addons in recent days.   not too much interesting there.

 but there are some interesting module correlations

100% (116/116) vs.  78% (119420/153178) firefox.exe
100% (116/116) vs.  78% (119877/153178) dbghelp.dll
100% (116/116) vs.  87% (133066/153178) mswsock.dll
100% (116/116) vs.  88% (134415/153178) setupapi.dll

user comments also talk a lot about just finishing up an upgrade and/or just trying to start up.

why would firefox.exe be around 78% of the time in other crashes and 100% here?

is this a possible indication of multiple instances of firefox running?

msdn docs on setupapi.dll and dbghelp.dll indidcate that the wrong version of dbghelp.dll ending up on a system can result in instability.

http://msdn.microsoft.com/en-us/library/ms679294%28VS.85%29.aspx

and setupapi.dll is involved in installation programs getting things installed.

could corruption introduced by firefox or OS updates be involved in this crash?

Comment 23

[Marcia Knous [:marcia]]

A few crashes in this stack have showed up in Beta 3 - http://tinyurl.com/28k9dpd. I tried to reproduce with the URLs but was not able to do so.

Comment 24

[Marcia Knous [:marcia]]

I asked the question in the triage meeting - wondering if the crash in js::DeepBail(JSContext*) which I see on the trunk is the same as  [@ js_DeepBail(JSContext*) ]? I see the first stack in B3 crash results with ~75 crashes.

Comment 25

[Damon Sicore (:damons)]

dvander, can you address the question in Comment 24?