Open Bug 785940 Opened 7 years ago Updated 2 years ago

Firefox startup crash in SECMOD_GetModuleSpecList @ __hash_open with SProtector.dll (Search Assistant SProtector)

Categories

(Core :: Security, defect, critical)

20 Branch
x86
Windows XP
defect
Not set
critical

Tracking

()

Tracking Status
firefox16 - ---

People

(Reporter: marcia, Unassigned)

References

Details

(Keywords: crash, Whiteboard: [startupcrash])

Crash Data

This bug was filed from the Socorro interface and is 
report bp-0b798181-5ea4-411f-92e3-c83102120827 .
============================================================= 

Seen while looking at crash stats. https://crash-stats.mozilla.com/report/list?signature=_open. Startup crash which affects Windows 7 and is seen across all versions.

Frame 	Module 	Signature 	Source
0 		@0xc0857489 	
1 	msvcr100.dll 	_open 	f:\dd\vctools\crt_bld\self_x86\crt\src\open.c:119
2 	nssdbm3.dll 	__hash_open 	dbm/src/hash.c:186
3 	nssdbm3.dll 	dbopen 	dbm/src/db.c:106
4 	nssdbm3.dll 	secmod_OpenDB 	security/nss/lib/softoken/legacydb/pk11db.c:561
5 	softokn3.dll 	sftkdb_LoadLibrary 	security/nss/lib/softoken/lgglue.c:194
6 	softokn3.dll 	sftkdb_ReadSecmodDB 	security/nss/lib/softoken/sftkmod.c:251
7 	softokn3.dll 	NSC_ModuleDBFunc 	security/nss/lib/softoken/pkcs11.c:2663
8 	nss3.dll 	SECMOD_GetModuleSpecList 	security/nss/lib/pk11wrap/pk11pars.c:1026
9 	nss3.dll 	SECMOD_LoadModule 	security/nss/lib/pk11wrap/pk11pars.c:1139
10 	nss3.dll 	nss_InitModules 	security/nss/lib/nss/nssinit.c:469
11 	nss3.dll 	NSS_Initialize 	security/nss/lib/nss/nssinit.c:850
12 	xul.dll 	nsNSSComponent::InitializeNSS 	security/manager/ssl/src/nsNSSComponent.cpp:1747
13 	xul.dll 	nsNSSComponent::Init 	security/manager/ssl/src/nsNSSComponent.cpp:1984
14 	xul.dll 	nsNSSComponentConstructor 	security/manager/ssl/src/nsNSSModule.cpp:176
15 	xul.dll 	mozilla::GenericFactory::CreateInstance 	obj-firefox/xpcom/build/GenericFactory.cpp:16
16 	xul.dll 	nsComponentManagerImpl::CreateInstanceByContractID 	xpcom/components/nsComponentManager.cpp:1032
17 	xul.dll 	nsComponentManagerImpl::GetServiceByContractID 	xpcom/components/nsComponentManager.cpp:1434
18 	xul.dll 	nsCOMPtr_base::assign_from_gs_contractid 	obj-firefox/xpcom/build/nsCOMPtr.cpp:99
19 	xul.dll 	nsCOMPtr<nsINSSComponent>::nsCOMPtr<nsINSSComponent> 	obj-firefox/dist/include/nsCOMPtr.h:581
20 	xul.dll 	EnsureNSSInitialized 	security/manager/ssl/src/nsNSSComponent.cpp:303
21 	xul.dll 	`anonymous namespace'::nsRandomGeneratorConstructor 	security/manager/ssl/src/nsNSSModule.cpp:218
22 	xul.dll 	mozilla::GenericFactory::CreateInstance 	obj-firefox/xpcom/build/GenericFactory.cpp:16
23 	xul.dll 	nsComponentManagerImpl::CreateInstance 	xpcom/components/nsComponentManager.cpp:945
24 	xul.dll 	nsJSCID::CreateInstance 	js/xpconnect/src/XPCJSID.cpp:736
25 	xul.dll 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/win32/xptcinvoke.cpp:70
26 	xul.dll 	XPCWrappedNative::CallMethod 	js/xpconnect/src/XPCWrappedNative.cpp:2442
27 	xul.dll 	XPC_WN_CallMethod 	js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1500
28 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:313
29 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2515
30 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:329
31 	mozjs.dll 	array_readonlyCommon<ArrayForEachBehavior> 	js/src/jsarray.cpp:3231
32 	mozjs.dll 	js::GetPropertyOperation 	js/src/jsinterpinlines.h:227
33 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2515
34 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:329
35 	mozjs.dll 	js::Invoke 	js/src/jsinterp.h:125
36 	mozjs.dll 	js_fun_apply 	js/src/jsfun.cpp:735
37 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:313
38 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:361
39 	mozjs.dll 	js::CrossCompartmentWrapper::call 	js/src/jswrapper.cpp:651
40 	mozjs.dll 	proxy_Call 	js/src/jsproxy.cpp:1649
41 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:306
42 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2515
43 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:329
44 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:361
45 	mozjs.dll 	js::CrossCompartmentWrapper::call 	js/src/jswrapper.cpp:651
46 	mozjs.dll 	proxy_Call 	js/src/jsproxy.cpp:1649
47 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:306
48 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2515
49 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:329
50 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:361
51 	mozjs.dll 	JS_CallFunctionValue 	js/src/jsapi.cpp:5549
52 	xul.dll 	nsXPCWrappedJSClass::CallMethod 	js/xpconnect/src/XPCWrappedJSClass.cpp:1474
53 	mozglue.dll 	arena_dalloc_small 	memory/jemalloc/jemalloc.c:4537
54 	xul.dll 	PrepareAndDispatch 	xpcom/reflect/xptcall/src/md/win32/xptcstubs.cpp:85
55 	xul.dll 	nsXREDirProvider::DoStartup 	toolkit/xre/nsXREDirProvider.cpp:763
56 	mozalloc.dll 	mozalloc.dll@0x109f 	
57 		@0x2e1455f 	
58 	msvcr100.dll 	_mbsnbicoll_l 	f:\dd\vctools\crt_bld\self_x86\crt\src\mbsnbico.c:66
59 	msvcr100.dll 	_mbsnbicoll_l 	
60 	msvcr100.dll 	_unlock 	f:\dd\vctools\crt_bld\self_x86\crt\src\mlock.c:374
61 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:100
62 	msvcr100.dll 	_initterm 	f:\dd\vctools\crt_bld\self_x86\crt\src\crt0dat.c:872
63 	firefox.exe 	_SEH_epilog4 	
64 	ntdll.dll 	wcstombs 	
65 	ntdll.dll 	RtlInvalidHandlerDetected 	
66 	firefox.exe 	pre_c_init 	crtexe.c:261
67 		@0xfedfdfff
QA Contact: mozillamarcia.knous
This crash doesn't seem to have any manual correlations, but I see sprotector.dll in most of the crashes.  [@ CreateFileA ] is another Windows 8 signature where the same dll appears - https://crash-stats.mozilla.com/report/list?signature=%20CreateFileA and that is also a startup crash.

http://community.sophos.com/t5/Sophos-SafeGuard-products/bd-p/SGprods - I think that dll is from Sophos.
Whiteboard: [Win8] → [Win8],startupcrash
Benjamin, this is a win8 top crasher and the comments on crash-stats are not actionable.Any recommendation for who could take a look at the stack ? Thanks !
All crash reports I've checked contain an unversioned DLL, SProtector.dll, that seems to belong to Safend Protector (see http://www.safend.com/65-en/safend%20protector.aspx).
Summary: Firefox startup crash in _open → Firefox startup crash in _open with SProtector.dll (Safend Protector application?)
Whiteboard: [Win8],startupcrash → [Win8][startupcrash]
Crash Signature: [@ _open] → [@ _open] [@ CreateFileA]
Summary: Firefox startup crash in _open with SProtector.dll (Safend Protector application?) → Firefox startup crash in _open with SProtector.dll (Safend Protector or Sophos SafeGuard?)
WinDBG actually does a worse job with the stack than Breakpad in this case:
ChildEBP RetAddr  
WARNING: Frame IP not in any known module. Following frames may be wrong.
00caadec 68417289 0xc0857489
00caae4c 684177c8 msvcr100!_tsopen_nolock(int * punlock_flag = 0x00caae7c, int * pfh = 0x00caae80, char * path = 0x05a0d7f0 "--- memory read error at address 0x05a0d7f0 ---", int oflag = 0n32768, int shflag = 0n64, int pmode = 0n384)+0x245 [f:\dd\vctools\crt_bld\self_x86\crt\src\open.c @ 399]
00caae9c 6ed1f9dc msvcr100!_open(char * path = 0x05a0d7f0 "--- memory read error at address 0x05a0d7f0 ---", int oflag = 0n32768)+0x50 [f:\dd\vctools\crt_bld\self_x86\crt\src\open.c @ 119]
00caafd4 773af5ea nssdbm3!__hash_open(char * file = 0x773b10f2 "???", int flags = 0n0, int mode = 0n0, struct HASHINFO * info = 0x0000002e, int dflags = 0n2)+0x12c [e:\builds\moz2_slave\rel-m-beta-w32-bld\build\dbm\src\hash.c @ 186]
00cab0c4 773ae5f0 ntdll!RtlWow64EnableFsRedirectionEx+0x70
00cab0d8 773ae5ba ntdll!LdrpReleaseModuleDatatableLock+0x13
00cab280 773b5240 ntdll!_SEH_epilog4_GS+0xa
773b5584 90909090 ntdll!LdrLoadDll+0xc6
773b5598 00000000 0x90909090
I don't think there's a lot left we can do with this in terms of engineering without STR. Can somebody from QA spend a little time trying safend/safeguard and figure out which of these has this DLL and whether any obvious crashes result?

It seems odd that we'd be seeing large crash volumes from either of those products, though...
Marcia, can you please give a second look at this , based on comment 6.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #6)
> It seems odd that we'd be seeing large crash volumes from either of those
> products, though...
115 crashes per week in 15.0 is a large crash volume for an OS that accounts for 0.2% of ADU. But as it's a startup crash, it just means that that product is incompatible with Firefox on Windows 8.
I installed Sophos on the machine in the lab but I don't see that dll in the directory. If it is malware, I have to wait until we have a Win 8 VM to try to test some more.
Marcia, any luck testing this on the Win 8 VM ?
In order to get http://www.safend.com/65-en/safend%20protector.aspx you have to have a salesperson contact you for an evaluation. 

Meanwhile, the correlation report also shows a 38% (6/16) vs.   2% (2977/142540) browsemngr.dll, which is part of Babylon. Bug 782706 is the bug tracking that crash. The version implicated is the same one in Bug 782706, 2.2.630.40.
Marcia, on our side , one more thing worth trying would be to have this reproduced having Babylon installed since the correlation seems pretty high . On the other hand we are reaching out to Safend about the dll .
I've heard back from Safend: "SProtcetor.dll existed in our legacy versions that will not work on Windows 8"

Can we re-run the correlation report to see if browsemngr.dll (now fixed in bug 782706) and Sprotector.dll are the only two significant correlations? If so, I think we should block Sprotector.dll on Win8 for FF16 beta 5.
(In reply to Alex Keybl [:akeybl] from comment #13)
> I've heard back from Safend
What about Sophos SafeGuard?

Here are the latest correlations per module version:
  _open|EXCEPTION_ACCESS_VIOLATION_EXEC (16 crashes)
    100% (16/16) vs.   0% (297/162255) sprotector.dll (unversioned)
     38% (6/16) vs.   3% (4218/162255) browsemngr.dll
          0% (0/16) vs.   0% (57/162255) 2.2.565.25
          0% (0/16) vs.   0% (1/162255) 2.2.623.36
         19% (3/16) vs.   1% (1314/162255) 2.2.630.40
         19% (3/16) vs.   2% (2846/162255) 2.2.643.41

  CreateFileA|EXCEPTION_ACCESS_VIOLATION_READ (17 crashes)
    100% (17/17) vs.   0% (297/162255) sprotector.dll (unversioned)
     53% (9/17) vs.   3% (4218/162255) browsemngr.dll
          0% (0/17) vs.   0% (57/162255) 2.2.565.25
          0% (0/17) vs.   0% (1/162255) 2.2.623.36
         12% (2/17) vs.   1% (1314/162255) 2.2.630.40
         41% (7/17) vs.   2% (2846/162255) 2.2.643.41
(In reply to Scoobidiver from comment #14)
> (In reply to Alex Keybl [:akeybl] from comment #13)
> > I've heard back from Safend
> What about Sophos SafeGuard?

Do you have a link to suggest that Sophos also has an sprotector.dll? We haven't been able to find related info.

Also wanted to say that this has been brought up on SUMO: https://support.mozilla.org/ne-NP/questions/937251

> Here are the latest correlations per module version:
>   _open|EXCEPTION_ACCESS_VIOLATION_EXEC (16 crashes)
>          19% (3/16) vs.   1% (1314/162255) 2.2.630.40
>          19% (3/16) vs.   2% (2846/162255) 2.2.643.41
> 
>   CreateFileA|EXCEPTION_ACCESS_VIOLATION_READ (17 crashes)
>          12% (2/17) vs.   1% (1314/162255) 2.2.630.40
>          41% (7/17) vs.   2% (2846/162255) 2.2.643.41

Looks like the the latest version doesn't fix this issue. I'll do outreach to Babylon.
(In reply to Alex Keybl [:akeybl] from comment #15)
> Looks like the the latest version doesn't fix this issue. I'll do outreach
> to Babylon.

Actually, we'll wait to see if blocklisting sprotector.dll on Win8 does the trick before investigating the Babylon lead further.
I've filed bug 792541.
Depends on: 792541
And indeed https://crash-stats.mozilla.com/report/index/022ab031-151a-484c-9694-365352120930 which has a buildid of 20120929042011 still has sprotector.dll in the module list.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #19)
> And indeed
> https://crash-stats.mozilla.com/report/index/022ab031-151a-484c-9694-
> 365352120930 which has a buildid of 20120929042011 still has sprotector.dll
> in the module list.

They may be using DLL injection - glad we tried to fix here, but we'll have to see if this is actually a top crasher post-Win8 release.
Crash Signature: [@ _open] [@ CreateFileA] → [@ _open] [@ @0x0 | _open] [@ CreateFileA]
It may be a trojan: https://www.microsoft.com/security/portal/Threat/Encyclopedia/Entry.aspx?Name=TrojanDownloader%3aHTML%2fSprotector.A
Summary: Firefox startup crash in _open with SProtector.dll (Safend Protector or Sophos SafeGuard?) → Firefox startup crash in SECMOD_GetModuleSpecList @ __hash_open with SProtector.dll (Safend Protector, Sophos SafeGuard or TrojanDownloader:HTML/Sprotector.A?)
A top crasher on Windows 8 doesn't need the topcrash keyword.
Keywords: topcrash
It started spiking across all versions on April 8th at 6H UTC.

Here are correlations:
  _tsopen_nolock|EXCEPTION_ACCESS_VIOLATION_EXEC (36 crashes)
    100% (36/36) vs.   1% (300/48661) lmrn.dll (Perion's software seen in bug 825600)
    100% (36/36) vs.   1% (306/48661) sqlite3.dll
        100% (36/36) vs.   1% (305/48661) 
          0% (0/36) vs.   0% (1/48661) 3.6.22.0
    100% (36/36) vs.   3% (1600/48661) sprotector.dll (adware - see http://www.shouldiremoveit.com/Search-Assistant-AppsAreFun-11665-program.aspx)
          0% (0/36) vs.   3% (1396/48661) 
        100% (36/36) vs.   0% (204/48661) 1.66.1133.0

The first frames of the stack trace on Windows XP are:
Frame 	Module 	Signature 	Source
0 		@0xd5e9fcff 	
1 	msvcr100.dll 	_tsopen_nolock 	f:\dd\vctools\crt_bld\self_x86\crt\src\open.c:399
2 	msvcr100.dll 	_open 	f:\dd\vctools\crt_bld\self_x86\crt\src\open.c:119
3 	nssdbm3.dll 	__hash_open 	dbm/src/hash.c:186
4 	nssdbm3.dll 	dbopen 	dbm/src/db.c:106
5 	nssdbm3.dll 	lgdb_OpenDB 	security/nss/lib/softoken/legacydb/pk11db.c:530
6 	nssdbm3.dll 	legacy_ReadSecmodDB 	security/nss/lib/softoken/legacydb/pk11db.c:572
7 	softokn3.dll 	NSC_ModuleDBFunc 	security/nss/lib/softoken/pkcs11.c:2723
8 	nss3.dll 	SECMOD_GetModuleSpecList 	security/nss/lib/pk11wrap/pk11pars.c:915
9 	nss3.dll 	SECMOD_LoadModule 	security/nss/lib/pk11wrap/pk11pars.c:1028
10 	nss3.dll 	nss_InitModules 	security/nss/lib/nss/nssinit.c:438
11 	nss3.dll 	NSS_Initialize 	security/nss/lib/nss/nssinit.c:826
12 	xul.dll 	nsNSSComponent::InitializeNSS 	security/manager/ssl/src/nsNSSComponent.cpp:1688

More reports at:
https://crash-stats.mozilla.com/report/list?signature=_tsopen_nolock
Crash Signature: [@ _open] [@ @0x0 | _open] [@ CreateFileA] → [@ _open] [@ @0x0 | _open] [@ CreateFileA] [@ _tsopen_nolock ]
OS: Windows 8 → Windows XP
Summary: Firefox startup crash in SECMOD_GetModuleSpecList @ __hash_open with SProtector.dll (Safend Protector, Sophos SafeGuard or TrojanDownloader:HTML/Sprotector.A?) → Firefox startup crash in SECMOD_GetModuleSpecList @ __hash_open with SProtector.dll (Search Assistant SProtector)
Whiteboard: [Win8][startupcrash] → [startupcrash]
Version: 15 Branch → 20 Branch
You need to log in before you can comment on or make changes to this bug.