Closed Bug 706042 Opened 13 years ago Closed 12 years ago

Crash in SECOID_Shutdown @ je_free | plds4.dll@0x1...

Categories

(Core :: Security, defect)

9 Branch
x86
Windows 7
defect
Not set
critical

Tracking

()

RESOLVED DUPLICATE of bug 675260
Tracking Status
firefox8 - unaffected
firefox9 + affected
firefox10 --- affected
firefox11 --- affected

People

(Reporter: scoobidiver, Unassigned)

References

Details

(Keywords: crash, regression, Whiteboard: [sg:moderate])

Crash Data

It's #47 top crasher in 9.0b3 and first appeared in 9.0a1/20110916.
The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f2a2adaaacba&tochange=39b192706927
The likely culprit is bug 683441.

Signature	je_free | plds4.dll@0x168f
UUID	604d7732-0676-46ff-8441-afcca2111129
Date Processed	2011-11-29 02:34:43.608093
Uptime	903
Last Crash	3.2 weeks before submission
Install Age	3.1 days since version was first installed.
Install Time	2011-11-26 06:58:51
Product	Firefox
Version	9.0
Build ID	20111122192043
Release Channel	beta
OS	Windows NT
OS Version	5.1.2600 Service Pack 3
Build Architecture	x86
Build Architecture Info	GenuineIntel family 15 model 1 stepping 2
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x1000
App Notes 	AdapterVendorID: 8086, AdapterDeviceID: 2562, AdapterSubsysID: 71481462, AdapterDriverVersion: 6.14.10.4020
D3D10 Layers? D3D10 Layers-
D3D9 Layers? D3D9 Layers-
EMCheckCompatibility	True

Frame 	Module 	Signature [Expand] 	Source
0 	mozutils.dll 	je_free 	memory/jemalloc/jemalloc.c:6260
1 	plds4.dll 	plds4.dll@0x168f 	
2 	plds4.dll 	DefaultFreeEntry 	nsprpub/lib/ds/plhash.c:87
3 	plds4.dll 	PL_HashTableDestroy 	nsprpub/lib/ds/plhash.c:147
4 	nssutil3.dll 	SECOID_Shutdown 	security/nss/lib/util/secoid.c:2161
5 	nss3.dll 	nss_Shutdown 	security/nss/lib/nss/nssinit.c:1101
6 	nss3.dll 	NSS_Shutdown 	security/nss/lib/nss/nssinit.c:1156
7 	xul.dll 	nsNSSComponent::ShutdownNSS 	security/manager/ssl/src/nsNSSComponent.cpp:1918
8 	xul.dll 	nsNSSComponent::DoProfileBeforeChange 	security/manager/ssl/src/nsNSSComponent.cpp:2597
9 	xul.dll 	nsNSSComponent::Observe 	security/manager/ssl/src/nsNSSComponent.cpp:2231
10 	xul.dll 	nsObserverList::NotifyObservers 	xpcom/ds/nsObserverList.cpp:130
11 	xul.dll 	nsObserverService::NotifyObservers 	xpcom/ds/nsObserverService.cpp:182
12 	xul.dll 	xul.dll@0xbc1e43 	
13 	xul.dll 	ScopedXPCOMStartup::~ScopedXPCOMStartup 	toolkit/xre/nsAppRunner.cpp:1080
14 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3587
15 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:107
16 	firefox.exe 	firefox.exe@0x4033 	
17 	firefox.exe 	__tmainCRTStartup 	crtexe.c:594
18 	xul.dll 	nsRefreshDriver::~nsRefreshDriver 	layout/base/nsRefreshDriver.cpp:122
19 	firefox.exe 	_SEH_epilog4 	

More reports at:
https://crash-stats.mozilla.com/report/list?signature=je_free%20|%20plds4.dll%400x168f
The crash address is an interestingly round number, and the list of crashes shows that pretty consistently (0x1000, 0x2000, 0xb000, etc). Page boundaries? Don't know if accessing memory that low could be exploited or not. Also appears to be a shutdown crash which would be tricky to convince users to do as part of an attack. Guessing at sg:moderate as an appropriate rating.
Assignee: nobody → mh+mozilla
Whiteboard: [sg:moderate]
(In reply to Scoobidiver from comment #0)
> It's #47 top crasher in 9.0b3 and first appeared in 9.0a1/20110916.
> The regression range is:
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=f2a2adaaacba&tochange=39b192706927
> The likely culprit is bug 683441.

The likely culprit is bug 677501 or bug 680440.
Blocks: 677501, 680440
No longer blocks: 683441
I'm not able to reproduce these crashes, but having glanced at the code the only way I can think of that could explain the problem is if somehow the dynOidHash values are allocated from the crt and freed with jemalloc. I see that nspr still imports the _malloc_crt symbol, though I haven't looked where it is being used from. If that is the real problem, the corresponding culprit would actually be new style jemalloc (bug 515492) and bug 678195 which switches to it everywhere.

Ted, Kyle, what do you think?
Kyle Huey@KYLEHUEY-PC /c/dev/mozilla-beta/obj-i686-pc-mingw32/nsprpub
$ grep 'malloc_crt' -r ./
Binary file ./lib/ds/plds4.dll matches
Binary file ./lib/ds/plds4.ilk matches
./lib/ds/plds4.map: 0001:000019d6       __malloc_crt               100029d6 f
mozcrt:MSVCR100.dll
./lib/ds/plds4.map: 0004:00000278       __imp___malloc_crt         10008278
mozcrt:MSVCR100.dll
Binary file ./lib/ds/plds4.pdb matches
Binary file ./lib/libc/src/plc4.dll matches
Binary file ./lib/libc/src/plc4.ilk matches
./lib/libc/src/plc4.map: 0001:00002888       __malloc_crt               10003888
 f   mozcrt:MSVCR100.dll
./lib/libc/src/plc4.map: 0004:00000298       __imp___malloc_crt         10008298
     mozcrt:MSVCR100.dll
Binary file ./lib/libc/src/plc4.pdb matches
Binary file ./pr/src/nspr4.dll matches
Binary file ./pr/src/nspr4.ilk matches
./pr/src/nspr4.map: 0001:00032e50       __malloc_crt               10033e50 f
mozcrt:MSVCR100.dll
./pr/src/nspr4.map: 0004:00000774       __imp___malloc_crt         10041774
mozcrt:MSVCR100.dll
Binary file ./pr/src/nspr4.pdb matches

That's consistent with malloc_crt only being used in the CRT glue and not in our actual code.  This looks like an allocator mismatch at first glance (as do some other bugs that I've seen lately) but I haven't been able to actually find any mismatch.
Kyle Huey@KYLEHUEY-PC /c/dev/mozilla-beta/obj-i686-pc-mingw32/nsprpub
$ dumpbin.exe -IMPORTS ./lib/ds/plds4.dll
Microsoft (R) COFF/PE Dumper Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file ./lib/ds/plds4.dll

File Type: DLL

  Section contains the following imports:

    nspr4.dll
              100082F4 Import Address Table
              10008184 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                   52 PR_DestroyLock
                   85 PR_Free
                  166 PR_Unlock
                   F5 PR_NewLock
                   27 PR_CallOnce
                   E7 PR_Lock
                   ED PR_Malloc
                   2D PR_CeilingLog2

    MSVCR100.dll
              1000824C Import Address Table
              100080DC Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                  2B1 _initterm_e
                  131 __CppXcptFilter
                  153 __clean_type_info_names_internal
                  48D _unlock
                  15B __dllonexit
                  323 _lock
                  3C9 _onexit
                  221 _except_handler4_common
                  2B0 _initterm
                  1FB _crt_debugger_hook
                  219 _encoded_null
                  334 _malloc_crt
                  5F8 strcmp
                  5D3 memset
                  5CF memcpy
                  1C5 _amsg_exit

    mozutils.dll
              100082C4 Import Address Table
              10008154 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                    5 frex

    KERNEL32.dll
              100081D4 Import Address Table
              10008064 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                  300 IsDebuggerPresent
                  4A5 SetUnhandledExceptionFilter
                  4D3 UnhandledExceptionFilter
                  1C0 GetCurrentProcess
                  4C0 TerminateProcess
                  279 GetSystemTimeAsFileTime
                  1C1 GetCurrentProcessId
                  1C5 GetCurrentThreadId
                  2E9 InterlockedCompareExchange
                  4B2 Sleep
                  2EC InterlockedExchange
                   CA DecodePointer
                  293 GetTickCount
                   EA EncodePointer
                  3A7 QueryPerformanceCounter
                   DE DisableThreadLibraryCalls

Kyle Huey@KYLEHUEY-PC /c/dev/mozilla-beta/obj-i686-pc-mingw32/nsprpub
$ dumpbin.exe -IMPORTS ./pr/src/nspr4.dll
Microsoft (R) COFF/PE Dumper Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file ./pr/src/nspr4.dll

File Type: DLL

  Section contains the following imports:

    ADVAPI32.dll
              10041534 Import Address Table
              1004108C Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                   20 AllocateAndInitializeSid
                   76 CopySid
                  136 GetLengthSid
                  15A GetTokenInformation
                  1F7 OpenProcessToken
                  120 FreeSid
                  2B6 SetSecurityDescriptorDacl
                   10 AddAccessAllowedAce
                  176 InitializeAcl
                  2B7 SetSecurityDescriptorGroup
                  2B8 SetSecurityDescriptorOwner
                  177 InitializeSecurityDescriptor

    WSOCK32.dll
              100418D8 Import Address Table
              10041430 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                      Ordinal    52
                      Ordinal    54
                      Ordinal     9
                      Ordinal     8
                      Ordinal    53
                      Ordinal    51
                      Ordinal    18
                      Ordinal   151
                      Ordinal     5
                      Ordinal     6
                      Ordinal    22
                      Ordinal    17
                      Ordinal    20
                      Ordinal    19
                      Ordinal    16
                      Ordinal    13
                      Ordinal     2
                      Ordinal     4
                      Ordinal     1
                      Ordinal    23
                      Ordinal    12
                      Ordinal     3
                      Ordinal    21
                      Ordinal   115
                      Ordinal   116
                      Ordinal    57
                      Ordinal     7
                      Ordinal   111
                      Ordinal    14
                      Ordinal    15

    WINMM.dll
              100418A8 Import Address Table
              10041400 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                   94 timeGetTime

    MSVCR100.dll
              1004175C Import Address Table
              100412B4 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                  219 _encoded_null
                  2B0 _initterm
                  2B1 _initterm_e
                  1C5 _amsg_exit
                  131 __CppXcptFilter
                  1FB _crt_debugger_hook
                  334 _malloc_crt
                  153 __clean_type_info_names_internal
                  48D _unlock
                  15B __dllonexit
                  323 _lock
                  3C9 _onexit
                  221 _except_handler4_common
                  471 _time64
                  1BA _access
                  39E _mbspbrk
                  35E _mbsdec
                  437 _stat64i32
                  364 _mbsinc
                  22A _exit
                  1D4 _beginthreadex
                  441 _stricmp
                  607 strpbrk
                  21C _environ
                  5DE qsort
                  5FF strftime
                  3C6 _mktime64
                  5A3 isalpha
                  321 _localtime64
                  603 strncmp
                  5FD strerror
                  5F5 strcat
                  608 strrchr
                  60A strstr
                  59B getenv
                  61B tolower
                  60E strtol
                  60F strtoul
                  5F7 strchr
                  5CD memchr
                  5AB isspace
                  5A5 isdigit
                  5EF sprintf
                  583 fprintf
                  55C abort
                  596 fwrite
                  579 fflush
                  5F8 strcmp
                  581 fopen
                  5EB setvbuf
                  576 fclose
                  5F3 sscanf
                  165 __iob_func
                  600 strlen
                  5FA strcpy
                  5CF memcpy
                  566 atoi
                  5D3 memset
                  21E _errno
                  3DB _putenv

    mozutils.dll
              10041994 Import Address Table
              100414EC Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                    C strdup
                    7 malloc
                    3 calloc
                    B realloc
                    5 frex
                    4 free

    KERNEL32.dll
              10041598 Import Address Table
              100410F0 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                  279 GetSystemTimeAsFileTime
                  1C1 GetCurrentProcessId
                  1C5 GetCurrentThreadId
                  304 IsProcessorFeaturePresent
                  300 IsDebuggerPresent
                  4A5 SetUnhandledExceptionFilter
                  4D3 UnhandledExceptionFilter
                  2E9 InterlockedCompareExchange
                   CA DecodePointer
                   EA EncodePointer
                  378 OpenFileMappingA
                  3A7 QueryPerformanceCounter
                  293 GetTickCount
                  383 OpenSemaphoreA
                  33F LoadLibraryW
                  339 LeaveCriticalSection
                  4D4 UnlockFile
                  352 LockFile
                  400 RemoveDirectoryA
                   7C CreateDirectoryA
                  35E MoveFileA
                  1FF GetHandleInformation
                  470 SetHandleInformation
                  1EC GetFileInformationByHandle
                  1D2 GetDriveTypeA
                  473 SetLastError
                  1F8 GetFullPathNameA
                   D3 DeleteFileA
                  143 FindNextFileA
                  132 FindFirstFileA
                  12E FindClose
                  157 FlushFileBuffers
                  413 ResumeThread
                  1C4 GetCurrentThread
                   E8 DuplicateHandle
                  4C6 TlsFree
                  4C5 TlsAlloc
                  286 GetThreadContext
                  3FE ReleaseSemaphore
                  466 SetFilePointer
                   AB CreateSemaphoreA
                  525 WriteFile
                  1C0 GetCurrentProcess
                  4D6 UnmapViewOfFile
                  357 MapViewOfFile
                  15D FormatMessageA
                   89 CreateFileMappingA
                  2A3 GetVersionExA
                  4C0 TerminateProcess
                  4F9 WaitForSingleObject
                  1DF GetExitCodeProcess
                  1D8 GetEnvironmentStrings
                  160 FreeEnvironmentStringsA
                   A4 CreateProcessA
                  277 GetSystemTime
                  4BD SystemTimeToFileTime
                  4C8 TlsSetValue
                   D1 DeleteCriticalSection
                  2E2 InitializeCriticalSection
                  4C7 TlsGetValue
                  3C0 ReadFile
                   88 CreateFileA
                  4BA SuspendThread
                  4B2 Sleep
                  264 GetStdHandle
                  499 SetThreadPriority
                   A1 CreatePipe
                   52 CloseHandle
                  2BF GlobalMemoryStatus
                  2EC InterlockedExchange
                  273 GetSystemInfo
                  218 GetModuleHandleW
                  214 GetModuleFileNameW
                  245 GetProcAddress
                  162 FreeLibrary
                  367 MultiByteToWideChar
                  202 GetLastError
                  511 WideCharToMultiByte
                  33E LoadLibraryExW
                  215 GetModuleHandleA
                   C7 DebugBreak
                   EE EnterCriticalSection
                  389 OutputDebugStringA
                  490 SetThreadAffinityMask
                  246 GetProcessAffinityMask

Everything looks in order to me ...
It's currently #37 top crasher in 9.0b4.
Since it doesn't happen on my machine, and since it doesn't happen for all users (in which case it would be top #1), i suppose something else (AV software?) is triggering it. Question is: how can we narrow it down?
Assignee: mh+mozilla → khuey
I don't think there's anything actionable here that I can do.  Maybe chofman can try to glean something from the crash data?
Assignee: khuey → nobody
Crash Signature: [@ je_free | plds4.dll@0x168f ] → [@ je_free | plds4.dll@0x168f] [@ je_free | plds4.dll@0x147f]
Summary: Crash in je_free | plds4.dll@0x168f → Crash in SECOID_Shutdown @ je_free | plds4.dll@0x1...
Group: core-security
Keywords: testcase-wanted
Crash Signature: [@ je_free | plds4.dll@0x168f] [@ je_free | plds4.dll@0x147f] → [@ je_free | plds4.dll@0x168f] [@ je_free | plds4.dll@0x147f] [@ je_free | plds4.dll@0x148f]
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.