486811 - pk12util memory corruption in optimized Windows builds

Status: RESOLVED INACTIVE

(NSS :: Tools, defect, P5)

Description

[Julien Pierre]

This is a new crash I am seeing only after using the debug memory allocator. This showed up in my optimized builds only, 32 and 64 bits.

The command that was run was :

tools.sh: Tools Tests with ECC ===============================
tools.sh: Exporting Alice's email cert & key - default ciphers
tools.sh: Exporting Alice's key & cert with [default:default] (pk12util -o)
pk12util -o Alice.p12 -n "Alice" -d ../alicedir \
         -k ../tests.pw -w ../tests.pw

Here are the 2 (very similar) stacks :

>	msvcr90d.dll!_free_dbg_nolock(void * pUserData=0x00000000028e68f0, int nBlockUse=1)  Line 1317 + 0x39 bytes	C++
 	msvcr90d.dll!_free_dbg(void * pUserData=0x00000000028e68f0, int nBlockUse=1)  Line 1258 + 0xe bytes	C++
 	msvcr90d.dll!free(void * pUserData=0x00000000028e68f0)  Line 50	C++
 	pk12util.exe!P12U_ExportPKCS12Object(char * nn=0x0000000002581000, char * outfile=0x0000000000000000, PK11SlotInfoStr * inSlot=0x00000000026c8ef0, SECOidTag cipher=SEC_OID_PKCS12_V2_PBE_WITH_SHA1_AND_3KEY_TRIPLE_DES_CBC, SECOidTag certCipher=SEC_OID_PKCS12_V2_PBE_WITH_SHA1_AND_40_BIT_RC2_CBC, secuPWData * slotPw=0x000000000012fec0, secuPWData * p12FilePw=0x000000000012fed0)  Line 714 + 0x1d bytes	C
 	pk12util.exe!main(int argc=0, char * * argv=0x0000000000000000)  Line 1103 + 0x2d bytes	C
 	pk12util.exe!__tmainCRTStartup()  Line 586 + 0x19 bytes	C
 	kernel32.dll!0000000077ae495d() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	
 	ntdll.dll!0000000077ce8791() 	


>	msvcr90d.dll!_free_dbg_nolock(void * pUserData=0x004727d0, int nBlockUse=1)  Line 1317 + 0x30 bytes	C++
 	msvcr90d.dll!_free_dbg(void * pUserData=0x004727d0, int nBlockUse=1)  Line 1258 + 0xd bytes	C++
 	msvcr90d.dll!free(void * pUserData=0x004727d0)  Line 49 + 0xb bytes	C++
 	pk12util.exe!p12u_DestroyContext(p12uContextStr * * ppCtx=0x00000000, int removeFile=0)  Line 115 + 0x7 bytes	C
 	pk12util.exe!P12U_ExportPKCS12Object(char * nn=0x0047148a, char * outfile=0x00db2658, PK11SlotInfoStr * inSlot=0x025971f0, SECOidTag cipher=SEC_OID_PKCS12_V2_PBE_WITH_SHA1_AND_3KEY_TRIPLE_DES_CBC, SECOidTag certCipher=SEC_OID_PKCS12_V2_PBE_WITH_SHA1_AND_40_BIT_RC2_CBC, secuPWData * slotPw=0x0017ff28, secuPWData * p12FilePw=0x0017ff30)  Line 715	C
 	pk12util.exe!main(int argc=11, char * * argv=0x00471410)  Line 1103 + 0x1e bytes	C
 	pk12util.exe!__tmainCRTStartup()  Line 586 + 0x17 bytes	C
 	kernel32.dll!7670e3f3() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	
 	ntdll.dll!77edcfed() 	
 	ntdll.dll!77edd1ff() 	


After I closed these two and let the QA script proceed, nearly every invocation of pk12util.exe had the same problem as well .

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

Julien, what compiler are you using for this?

Comment 2

[[e-mail]]

I believe msvcr90d.dll is part of Visual C++ 2008.

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

Indeed.  I hadn't noticed that in the stack.
I wonder if this is a compiler bug. I remember all the many problems we 
had with MSVC 2005 until they released SP1.
I wonder how this behaves when built with MSVC2005SP1