511758 - SkipList Out of memory crashes with "new" are shown as [@ RaiseException | _CxxThrowException] - should skip RaiseException, _CxxThrowException, append "operator new" to signature

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[[:Cww]]

Number 6 Firefox topcrash

http://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=exact&query=&date=&range_value=1&range_unit=weeks&do_query=1&signature=RaiseException

(filing because no bugs were linked from crash-stats and I couldn't find any in a quick search)

Comment 1

[chris hofmann]

0  	kernel32.dll  	RaiseException  	
1 	mozcrt19.dll 	_CxxThrowException 	throw.cpp:159
2 	mozcrt19.dll 	operator new 	new.cpp:57
3 	xul.dll 	LiteralImpl::Create 	mozilla/rdf/base/src/nsRDFService.cpp:484
4 	xul.dll 	RDFServiceImpl::GetLiteral 	mozilla/rdf/base/src/nsRDFService.cpp:1131
5 	xul.dll 	xul.dll@0x860ee3 	
6 	xul.dll 	RDFContentSinkImpl::OpenObject 	mozilla/rdf/base/src/nsRDFContentSink.cpp:1192
7 	xul.dll 	RDFContentSinkImpl::HandleStartElement 	mozilla/rdf/base/src/nsRDFContentSink.cpp:470
8 	xul.dll 	Driver_HandleStartElement 	mozilla/parser/htmlparser/src/nsExpatDriver.cpp:97
9 	xul.dll 	doContent 	mozilla/parser/expat/lib/xmlparse.c:2464
10 	xul.dll 	contentProcessor 	mozilla/parser/expat/lib/xmlparse.c:2095
11 	xul.dll 	MOZ_XML_ParseBuffer 	mozilla/parser/expat/lib/xmlparse.c:1618
12 	xul.dll 	MOZ_XML_Parse 	mozilla/parser/expat/lib/xmlparse.c:1589
13 	xul.dll 	nsExpatDriver::ConsumeToken 	mozilla/parser/htmlparser/src/nsExpatDriver.cpp:1125
14 	xul.dll 	nsParser::Tokenize 	mozilla/parser/htmlparser/src/nsParser.cpp:2435
15 	xul.dll 	nsExpatDriver::WillResumeParse 	mozilla/parser/htmlparser/src/CNavDTD.cpp:3177
16 	xul.dll 	nsParser::OnDataAvailable 	mozilla/parser/htmlparser/src/nsParser.cpp:2305
17 	nspr4.dll 	FileAvailable 	mozilla/nsprpub/pr/src/io/prfile.c:158

Comment 2

[Wayne Mery (:wsmwk)]

#8 topcrash for thunderbird 3.0b3

example bp-4cba5244-a200-44e4-9388-8a4702090822
sending a message using Vista SP2 32-bit with network.auth-use-sspi=true while joined to Active Directory

Comment 3

[Mark Banner (:standard8)]

(In reply to comment #2)
> #8 topcrash for thunderbird 3.0b3
> 
> example bp-4cba5244-a200-44e4-9388-8a4702090822
> sending a message using Vista SP2 32-bit with network.auth-use-sspi=true while
> joined to Active Directory

Please compare more than just the top of stack - RaiseException looks to be a generic error in windows code (kernel32) and in this case it is probably more likely that something stupid is being done.

Also I've seen reports of a bug with sspi and vista, so may be worth looking there, I think bienvenu has been looking at it.

Comment 4

[(not currently active) Ted Mielczarek]

"RaiseException" is just a Windows API to...raise an exception. In the stack in comment 1, it looks like an OOM crash. We should get that function name (as well as CxxThrowException and probably "operator new") added to the list of "frames to skip or append to the signature" in order to get more unique signatures.

Comment 5

[Jesse Ruderman]

Getting more unique signatures for OOM is useless, even counterproductive, without a fix for bug 493779 (breakpad detecting OOM) or bug 441324 (xmalloc).

Comment 6

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

I don't actually believe that some of these are oom -- some could also just be bogus too-large allocations.  I just crashed at http://crash-stats.mozilla.com/report/index/8a037afd-678d-46c8-8b80-11ca02090830 and the browser/computer was fine just before the crash.

Comment 7

[(not currently active) Ted Mielczarek]

This is one case where I really wish we had argument values on the stack. :-/

Comment 8

[Jesse Ruderman]

Vlad has a good point.  Quite a few of these even happen at startup!

Since we don't use exceptions ourselves, let's skip "RaiseException" and "_CxxThrowException", and append "operator new".  

For example, bp-d238e1b5-12b5-47a9-b54d-d02242091013 should have the signature [operator new | nsFastLoadFileReader::ReadFooter].

Comment 9

[Jesse Ruderman]

Together, these crashes are topcrash #31.  Splitting them up would make them easier to tackle.  Many of the "too large allocation" crashes will be easy to fix, and getting correlations for the crashes we suspect are truly "out of memory" could tell us what extensions cause severe leaks.

Comment 11

[chris hofmann]

re: breakdown of startup crashes and breakdown of other RaiseExpection signatures

report for the stack signature RaiseException 

2081 total crashes for RaiseException on 20091019-crashdata.csv
695 start up crashes inside 3 minutes
328.935 (days) total uptime for 1959 of these crashes where user crashed within the last year 
241.79 (minutes) avg time since last crash 
111  number of bogus time-since-last-crash reported for this signature 

RaiseException signature breakdown
signature distribution
     101
signature list
1313 RaiseException | _CxxThrowException
  90 RaiseException | mspbwkr.dll@0x25a22
  79 RaiseException | Shell32.dll@0x28b3
  62 RaiseException | wmp.dll@0x22933f
  40 RaiseException | AcroRd32.dll@0xf20df
  36 RaiseException | GoogleDesktopMozilla.dll@0x1555b
  35 RaiseException | npietab.dll@0x20192
  31 RaiseException | AcroRd32.dll@0xf2009
  27 RaiseException | _raise_exc_ex
  25 RaiseException | radhslib.dll@0x254f
  25 RaiseException | _hfirewall.dll@0x2a1b
  22 RaiseException | DeezRipDll.dll@0x3a56
  22 RaiseException | AcroRd32.dll@0xf1749
  21 RaiseException | MySpaceFFoxTB.dll@0x4354b
  17 RaiseException | AcroRd32.dll@0xf20d0
  16 RaiseException | wmp.dll@0x228c2b
  14 RaiseException | wmp.dll@0x1b138a
  14 RaiseException | AcroRd32.dll@0xf181f
  12 RaiseException | tmagent.dll@0x8d870
  11 RaiseException | wmp.dll@0x228c62
   9 RaiseException | AcroForm.api@0x4ce5ec
   8 RaiseException | nw.dll@0x153e1
   8 RaiseException | coreclr.dll@0x17c24d
   6 RaiseException | xwsg.dll@0x433c
   6 RaiseException | crescendoCatcher.dll@0x43ec9
   5 RaiseException | AcroRd32.dll@0xf1ffa
   5 RaiseException | AcroRd32.dll@0xef2be
   5 RaiseException | AcroRd32.dll@0xef1e8
   4 RaiseException | wmp.dll@0x228d3d
   4 RaiseException | mspbwkr.dll@0x295d2
   4 RaiseException | _hfirewall.dll@0x269f
   3 RaiseException | wmp.dll@0x2293fe
   3 RaiseException | wmp.dll@0x1b13ac
   3 RaiseException | sdata.dll@0x252f
   3 RaiseException | mscscc.dll@0x28ab
   3 RaiseException | gsl.dll@0x7bb0
   3 RaiseException | Shell32.dll@0x3f96
   3 RaiseException | SbieDll.dll@0x87ce
   3 RaiseException | GoogleDesktopMozilla.dll@0x154dd
   3 RaiseException | CoolType.dll@0x17200d
   2 RaiseException | wmp.dll@0x228c0b
   2 RaiseException | stormchaser.dll@0x494d7
   2 RaiseException | sis.dll@0x192c1
   2 RaiseException | radhslib.dll@0x252f
   2 RaiseException | nw.dll@0x6aa5
   2 RaiseException | npietab.dll@0x1fe89
   2 RaiseException | Shell32.dll@0x288b
   2 RaiseException | Scxpx86.dll@0x524cf
   2 RaiseException | SSS.dll@0x859c3
   2 RaiseException | GoogleDesktopMozilla.dll@0x16982
   2 RaiseException | CoolType.dll@0x1720fd
   2 RaiseException | CMediaViewHandler::Shutdown()
   2 RaiseException | Acrobat.dll@0x13cccc
   2 RaiseException | Acrobat.dll@0x13cbf6
   2 RaiseException | Acrobat.dll@0x13b519
   2 RaiseException | AcroRd32.dll@0xf0d3f
   2 RaiseException | AcroForm.api@0x4cde7c
   1 RaiseException | zgtips.dll@0xbfc88
   1 RaiseException | xwsg.dll@0x43d0
   1 RaiseException | xwsg.dll@0x4378
   1 RaiseException | xshared.dll@0x2e530
   1 RaiseException | wmp.dll@0x7c422c
   1 RaiseException | wmp.dll@0x284f1e
   1 RaiseException | wmp.dll@0x277a98
   1 RaiseException | wmp.dll@0x2774e8
   1 RaiseException | wmp.dll@0x2752f1
   1 RaiseException | wmp.dll@0x22aa7e
   1 RaiseException | wmp.dll@0x228cea
   1 RaiseException | wmp.dll@0x228c36
   1 RaiseException | wmp.dll@0x1b2e52
   1 RaiseException | sysspc.dll@0x28b3
   1 RaiseException | spxpclp32.dll@0x28b3
   1 RaiseException | spcffwl.dll@0x28b3
   1 RaiseException | sQusi20Blk.dll@0xa772
   1 RaiseException | nw.dll@0x265f
   1 RaiseException | npwinext.dll@0x10a82
   1 RaiseException | npctrl.1.0.30716.0.dll@0x987fa
   1 RaiseException | npGarmin.dll@0x200cb0
   1 RaiseException | npCCTVplayer.dll@0x11bc2
   1 RaiseException | nexFilter.dll@0x2a47
   1 RaiseException | mscorwks.dll@0x5e773
   1 RaiseException | mscoree.dll@0x2f99c
   1 RaiseException | lsp.dll@0x14bb6
   1 RaiseException | lsp.dll@0x135c5
   1 RaiseException | kikin.dll@0x1cb59
   1 RaiseException | hfirewall.dll@0x269f
   1 RaiseException | googletoolbar-ff3.dll@0x2b32f
   1 RaiseException | controlkids2.dll@0x1e134
   1 RaiseException | TMSender.XPCom.dll@0x4b8b1
   1 RaiseException | NPComponent.dll@0xa23b
   1 RaiseException | LsNdis.dll@0x273f
   1 RaiseException | IconixFF3_5.dll@0x4c3d8
   1 RaiseException | GoogleDesktopMozilla.dll@0x1639c
   1 RaiseException | GoogleDesktopMozilla.dll@0x161f4
   1 RaiseException | CoolType.dll@0x17213d
   1 RaiseException | CTabWindow::MarkTabAsNotHung(unsigned long)
   1 RaiseException | Acrobat.dll@0x13cbb5
   1 RaiseException | Acrobat.dll@0x13cadf
   1 RaiseException | Acrobat.dll@0x13b5ef
   1 RaiseException | AcroForm.api@0x4b6c2c
   1 RaiseException

Comment 12

[chris hofmann]

Attachment: RaiseException-crashes

something that might help organize the various crashes into buckets.   all the raiseException crashes sorted by "time since last crash".

Comment 13

[chris hofmann]

In the set of close to start up crashes there seem to be a bunch of fast load related stacks that look something like this.

0  	kernel32.dll  	RaiseException  	
1 	mozcrt19.dll 	_CxxThrowException 	throw.cpp:159
2 	mozcrt19.dll 	operator new 	new.cpp:57
3 	xul.dll 	nsFastLoadFileReader::ReadFooter 	mozilla/xpcom/io/nsFastLoadFile.cpp:722
4 	xul.dll 	nsFastLoadFileReader::Open 	mozilla/xpcom/io/nsFastLoadFile.cpp:978
5 	xul.dll 	NS_NewFastLoadFileReader 	mozilla/xpcom/io/nsFastLoadFile.cpp:1225
6 	xul.dll 	nsFastLoadService::NewInputStream 	mozilla/xpcom/io/nsFastLoadService.cpp:172
7 	xul.dll 	mozJSComponentLoader::StartFastLoad 	mozilla/js/src/xpconnect/loader/mozJSComponentLoader.cpp:880
8 	xul.dll 	CallGetService 	nsComponentManagerUtils.cpp:94
9 	xul.dll 	nsSSLSocketProvider::AddRef 	mozilla/netwerk/base/src/nsFileStreams.cpp:364
10 	xul.dll 	mozJSComponentLoader::GlobalForLocation 	mozilla/js/src/xpconnect/loader/mozJSComponentLoader.cpp:1127
11 	xul.dll 	nsLocalFile::ResolveAndStat 	mozilla/xpcom/io/nsLocalFileWin.cpp:847
12 	xul.dll 	nsComponentManagerImpl::AutoRegisterComponent 	mozilla/xpcom/components/nsComponentManager.cpp:2988
13 	xul.dll 	nsLocalFile::IsDirectory 	mozilla/xpcom/io/nsLocalFileWin.cpp:2399
14 	xul.dll 	nsComponentManagerImpl::AutoRegisterImpl 	mozilla/xpcom/components/nsComponentManager.cpp:2862

Is this the first one we could spin out from this bug?

Comment 14

[Jesse Ruderman]

chofmann, comment 11 is offtopic.  PLEASE stop spamming bugs with long, offtopic comments.

Regarding comment 13, yes.

Comment 15

[(not currently active) Ted Mielczarek]

Lars: can you add RaiseException and _CxxThrowException to irrelevantSignatureRegEx, and "operator new" to prefixSignatureRegEx? (It looks like RaiseException is already in prefixSignatureRegEx, I suppose we should remove it from there.)

Comment 16

[Frank Griswold [:griswolf] [:fgriswold]]

See also bug 514505: Fix at same time

Comment 17

[Frank Griswold [:griswolf] [:fgriswold]]

Resolved per bug 529076 2009-11-18 16:07:48 PST

Comment 18

[Jesse Ruderman]

That didn't quite work.  

bp-fcb1828b-e1d2-419c-9424-4e9ae2091126
[@operator new(unsigned int) ]
should be:
[@operator new(unsigned int) | nsFastLoadFileReader::ReadFooter(nsFastLoadFileReader::nsFastLoadFooter*) ]

Comment 19

[K Lars Lohn [:lars] [:klohn]]

it just looks like "operator new" was not added to the prefixSignatureRegEx config parameter.  Griswolf, would you do the honors and submit another IT request for this change?

Comment 20

[Frank Griswold [:griswolf] [:fgriswold]]

in bp-fcb1828b-e1d2-419c-9424-4e9ae2091126 we need operator new with an argument. In comment #1 and comment #13, we appear to need the no-argument version. Further code spelunking: We need to handle a single parameter. Do not want to match a placement new.

Comment 21

[Jesse Ruderman]

Fixed half an hour ago in bug 531895.  Seems to be working great for new crashes:

bp-fd9c67f9-f0bf-4cf9-b751-b31362091201
bp-7d15fbc0-b6fe-4c31-b56b-795882091201