432284 - Firefox 3.0pre Crash Reports [@ 0x0 - secmod_argParseModuleSpec] - nss3.dll & xul.dll on the stack

Status: RESOLVED WORKSFORME

(NSS :: Libraries, defect)

Description

[chris hofmann]

while breakpad crashes have a variety of problems under the #6 topcrash @x0x0 stack signature,  several of the @0x0 crash bugs appear to be happening at start up (or within seconds of firefox launch), and stack traces look something like

0  	 	@0x0  	
1 	nss3.dll 	secmod_argParseModuleSpec 	pk11pars.h:254
2 	nss3.dll 	SECMOD_LoadModule 	mozilla/security/nss/lib/pk11wrap/pk11pars.c:305
3 	nss3.dll 	nss_Init 	mozilla/security/nss/lib/nss/nssinit.c:541
4 	nss3.dll 	NSS_InitReadWrite 	mozilla/security/nss/lib/nss/nssinit.c:606
5 	xul.dll 	nsNSSComponent::InitializeNSS 	mozilla/security/manager/ssl/src/nsNSSComponent.cpp:1548
6 	xul.dll 	nsNSSComponent::Init 	mozilla/security/manager/ssl/src/nsNSSComponent.cpp:1746
7 	xul.dll 	nsNSSComponentConstructor 	mozilla/security/manager/ssl/src/nsNSSModule.cpp:166
8 	xul.dll 	nsGenericFactory::CreateInstance 	nsGenericFactory.cpp:80
9 	xul.dll 	nsComponentManagerImpl::CreateInstanceByContractID 	mozilla/xpcom/components/nsComponentManager.cpp:1756
10 	xul.dll 	nsComponentManagerImpl::GetServiceByContractID 	mozilla/xpcom/components/nsComponentManager.cpp:2189
11 	xul.dll 	CallGetService 	nsComponentManagerUtils.cpp:94
12 	xul.dll 	nsJAR::GetCertificatePrincipal 	mozilla/modules/libjar/nsJAR.cpp:377
13 	xul.dll 	nsJARChannel::GetOwner 	mozilla/modules/libjar/nsJARChannel.cpp:494
14 	xul.dll 	nsJARChannel::Open 	mozilla/modules/libjar/nsJARChannel.cpp:665
15 	xul.dll 	nsScriptSecurityManager::GetChannelPrincipal 	mozilla/caps/src/nsScriptSecurityManager.cpp:403
16 	xul.dll 	CSSLoaderImpl::LoadSheet 	mozilla/layout/style/nsCSSLoader.cpp:1314
17 	xul.dll 	CSSLoaderImpl::InternalLoadNonDocumentSheet 	mozilla/layout/style/nsCSSLoader.cpp:2044

most of the code around the top of the stack hasn't changed in a long time, but I did spot this one check-in recently

mozilla/ security/ nss/ lib/ nss/ nssinit.c

1.94	alexei.volkov.bugs%sun.com	2008-03-26 11:49	 	397832 - libpkix leaks memory if a macro calls a function that returns an error. Object leak test. r=nelson
1.93	alexei.volkov.bugs%sun.com	2008-03-11 13:48	 	412468 - modify certutil, vfychain and vfyserv utilities to use CERT_PKIXVerifyCert function. Patches: suply trustlist to CERT_PKIXVerifyCert; use double "p" argument to use CERT_PKIXVerifyCert for validation. r=nelson.

It would be good to try and figure out if there is a regression here, or it may just be a long standing bug that is starting to surface as other top crash bugs are fixed...

see these for more details.
http://crash-stats.mozilla.com/report/index/bf5207e3-1955-11dd-a348-001cc45a2c28
http://crash-stats.mozilla.com/report/index/c251db7a-1955-11dd-856d-001cc45a2c28
http://crash-stats.mozilla.com/report/index/c17bb985-1955-11dd-9828-001cc45a2c28

no user comments or test cases to go on right now, so this will just need to be code inspection.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

The list of top crashers for Firefox 3.0pre that Chris mentioned can be
seen at http://crash-stats.mozilla.com/topcrasher/byversion/Firefox/3.0pre

The graph at http://crash-stats.mozilla.com/report/list?range_unit=weeks&version=Firefox%3A3.0pre&range_value=2&signature=%400x0 shows a big spike in 
frequency on 04/20.  I'm guessing that Mozilla updated the tag by which it 
pulls NSS sources on that day.  Chris, is there a way to look at the stacks
with that signature from that day (4/20)?

If any change to nssinit.c is suspect, I'd say it's rev 1.92, which was a big
change to add a DB Merge form of NSS initialization.  But I'm not sure there's
any bug there, yet.  

Looking at the "raw" dumps in the URLs cited in comment 0, I see that in 
every one of them there are already an SSL thread and a cert verification 
thread running.  Would those threads be launched before NSS is initialized?
Is NSS normally initialized by XUL code?  
Could this be an add-on/plugin/extension that is initializing its own copy 
of NSS?  

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

Hmm, when I go and look at that graph for the last 4 weeks, instead of the
last 2, it appears that the frequency of crashes with that signature has 
been pretty constant over the whole period. 

http://crash-stats.mozilla.com/report/list?range_unit=weeks&version=Firefox%3A3.0pre&range_value=4&signature=%400x0

In that graph, the activity on 04/20 doesn't appear to be a big increase, 
and in fact the activity on and after 4/20 is dwarfed by the activity on 
3/27, 4/10 and 4/15.  I'd really like to get an idea of when this started,
but I don't know how to look at the crashes from those earlier dates.

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

I found the crashes for the earlier dates in another tab on the page cited
in comment 2.

The NSS_CO_TAG was changed to 
- NSS_3_12_RC3 in rev 1.380 of client.mk on 05/03, and before that, to
- NSS_3_12_RC2 in rev 1.378 of client.mk on 04/08

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

The 3 crashes Christ cited in comment zero all happened on 05/03 within
a few minutes of each other, all within a single-digit amount of time 
since startup (don't know the relevant units: seconds?).  Looking back over
the signature 0x0 crashes as far back as 04/08, I don't see any other 
crashes with very short lifetimes and similar stack signatures.  However
I see a fair number of crashes with that signature on windows that no
stack showing, just address 0.  Makes me wonder if maybe those 3 crashes
come from someone's debug build.

Comment 5

[chris hofmann]

I found a way to go back and look at a longer history of @0x0 crashes, but I'm
not sure how accurate or valuable it is.

This basically gives all @0x0 "trunk" and beta relaase crashes for the last
serveral months...  
http://crash-stats.mozilla.com/report/list?range_unit=weeks&query_search=signature&query_type=contains&product=Firefox&branch=1.9&signature=%400x0&query=%400x&range_value=20

Interesting to note the request is for 20 weeks, but the graph shows 13 weeks
in the title, and the x-axis streaches back over the entire last year.  Looks
like some bugs in the reporting system there will need to be sorted out.

The other problem as mentioned in comment 0 is that the @0x0 stack signature
most likely has several bugs associated with it with many different stack
traces.  For example
http://crash-stats.mozilla.com/report/index/1a7779c7-0fc5-11dd-a8d0-001cc4e2bf68
shows an 0x0 crash related to a docshell/docloader/loadgroup bug with nothing
on the stack related to nss code.

I think ted is working on fixes that will get the more precision in the stack
signature so we can get a bit better classification of related bugs.  We might
have to wait for those changes to make any more progress on analysis for this
bug.

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

Chris,
Other than the 3 that occurred on 5/3, do you see any other occurrences
that seem to include NSS in initialization?

(oh, sorry, didn't mean to deify you in comment 4 :)

Comment 7

[Nelson Bolyard (seldom reads bugmail)]

Bob, rev 1.92 of nssinit.c is potentially suspect.  
It was the addition of a DB Merge form of NSS initialization.

Comment 8

[Robert Relyea]

It would be good to see what modulespec looks like.

It would also be good to know what kind of input could cause secmod_argParseModuleSpec to crash (it should unless the string isn't null terminated...

Comment 9

[chris hofmann]

re: Nelson's questions and comments in commment 6

I have a few queries ready to go to try and tease more reports out of the database but there are some perf problems with the reporting system right now.  Fixes should be on the way next that will allow us to see if there are more reports like this, and if the crashes are bounded in time or have been there awhile.

I know your just trying to make up for all the bad names you have called ( or thought about ) me in the past. ;-)

Comment 10

[(not currently active) Ted Mielczarek]

(need to test and land the patch in bug 411349, to make the reporting better for crashes like this)

Comment 11

[Nelson Bolyard (seldom reads bugmail)]

It might be helpful if we could get a copy of the file secmod.db from 
users who encounter this crash.  In all but one uses of this code, the
strings parsed by the function that is crashing are derived (at least,
in part) from the contents of the secmod.db file. 

Comment 12

[Wayne Mery (:wsmwk)]

(In reply to Nelson Bolyard (seldom reads bugmail) from comment #11)
> It might be helpful if we could get a copy of the file secmod.db from 
> users who encounter this crash.  In all but one uses of this code, the
> strings parsed by the function that is crashing are derived (at least,
> in part) from the contents of the secmod.db file.

This will likely be difficult to impossible, because the crash is rare. I examined about 20 crashes and came up with one address in bp-269705c2-ae72-462d-a4b7-c20b22130331, which I've just tried to contact.

Comment 13

[Wayne Mery (:wsmwk)]

on crash-stats only 3 crashes in 4 months and none are newer than version 17