531337 - Firefox 3.6b Start up Crash [@ timeGetTime] HOOK.dll is an interesting bystander

Status: RESOLVED FIXED

(Toolkit :: Blocklist Policy Requests, defect)

Description

[chris hofmann]

in early beta4 data this is the #9 top crash

http://crash-stats.mozilla.com/report/index/d5e35290-8d3d-4b62-9060-f2cbf2091126

Frame  	Module  	Signature [Expand]  	Source
0 	winmm.dll 	timeGetTime 	
1 	xul.dll 	nsTimerImpl::Fire 	xpcom/threads/nsTimerImpl.cpp:382
2 	nspr4.dll 	_PR_MD_UNLOCK 	nsprpub/pr/src/md/windows/w95cv.c:344
3 	xul.dll 	nsTimerEvent::Run 	xpcom/threads/nsTimerImpl.cpp:519
4 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:527
5 	xul.dll 	nsBaseAppShell::Run 	widget/src/xpwidgets/nsBaseAppShell.cpp:170
6 	xul.dll 	nsAppStartup::Run 	toolkit/components/startup/src/nsAppStartup.cpp:182
7 	nspr4.dll 	PR_GetEnv 	
8 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:120
9 	firefox.exe 	__tmainCRTStartup 	obj-firefox/memory/jemalloc/crtsrc/crtexe.c:591
10 	kernel32.dll 	BaseProcessStart 

http://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox%3A3.6b4&query_search=signature&query_type=exact&query=&date=&range_value=1&range_unit=weeks&do_query=1&signature=timeGetTime

all those indicate crash within 12 seconds of start up, with most almost immediately.

I suspect it will go down in ranking if it follow similar pattern for new releases.  We seem to have a flurry of around each new beta release.

grep  timeGetTime  20091* | awk -F\t '$8 ~ /3.6/ {print $1}' | sort | uniq -c

[no reports oct 1 - oct 25 ]

   1 20091026-crashdata.csv:timeGetTime
   1 20091029-crashdata.csv:timeGetTime
   1 20091030-crashdata.csv:timeGetTime
   1 20091104-crashdata.csv:timeGetTime
   1 20091105-crashdata.csv:timeGetTime
   2 20091106-crashdata.csv:timeGetTime
  10 20091107-crashdata.csv:timeGetTime
  12 20091108-crashdata.csv:timeGetTime
  12 20091109-crashdata.csv:timeGetTime
  23 20091111-crashdata.csv:timeGetTime
  13 20091112-crashdata.csv:timeGetTime
   4 20091113-crashdata.csv:timeGetTime
   3 20091114-crashdata.csv:timeGetTime
   7 20091115-crashdata.csv:timeGetTime
   6 20091116-crashdata.csv:timeGetTime
   6 20091117-crashdata.csv:timeGetTime
   2 20091118-crashdata.csv:timeGetTime
   6 20091119-crashdata.csv:timeGetTime
  15 20091120-crashdata.csv:timeGetTime
   6 20091121-crashdata.csv:timeGetTime
  10 20091122-crashdata.csv:timeGetTime
  16 20091123-crashdata.csv:timeGetTime
   3 20091124-crashdata.csv:timeGetTime
  22 20091125-crashdata.csv:timeGetTime


This signature doesn't seem to appear in 3.5.x, though I might be messing up on that query.  More research needed to figure out if this is a regression.

Also more research needed into what might be causing the crash, and if a wide number of users might hit it as an upgrade crash when 3.6 gets million+ user distribution.

Comment 1

[chris hofmann]

ok, this does appear to exist in 3.5.5 and ramps in volume as more users get on that release.  on 11/7, for example, 3.5.5 had 22milion users and 42 of these crashes.  now 3.5.5 has ~50m users and around 80-90 of these crashes per day.

crashdata chofmann$ grep  timeGetTime  20091* | awk -F\t '$8 ~ /3.5.5/ {print $1,$8}' | sort | uniq -c | grep -v arena

[no reports oct1 -nov5]

   1 20091105-crashdata.csv:timeGetTime 3.5.5
  18 20091106-crashdata.csv:timeGetTime 3.5.5
  43 20091107-crashdata.csv:timeGetTime 3.5.5
  50 20091108-crashdata.csv:timeGetTime 3.5.5
  57 20091109-crashdata.csv:timeGetTime 3.5.5
  55 20091110-crashdata.csv:timeGetTime 3.5.5
  45 20091111-crashdata.csv:timeGetTime 3.5.5
  59 20091112-crashdata.csv:timeGetTime 3.5.5
  64 20091113-crashdata.csv:timeGetTime 3.5.5
  75 20091114-crashdata.csv:timeGetTime 3.5.5
  66 20091115-crashdata.csv:timeGetTime 3.5.5
  44 20091116-crashdata.csv:timeGetTime 3.5.5
   1 20091116-crashdata.csv:timeGetTime 3.5.5pre
  49 20091117-crashdata.csv:timeGetTime 3.5.5
  57 20091118-crashdata.csv:timeGetTime 3.5.5
  69 20091119-crashdata.csv:timeGetTime 3.5.5
  60 20091120-crashdata.csv:timeGetTime 3.5.5
  58 20091121-crashdata.csv:timeGetTime 3.5.5
  94 20091122-crashdata.csv:timeGetTime 3.5.5
  91 20091123-crashdata.csv:timeGetTime 3.5.5
  55 20091124-crashdata.csv:timeGetTime 3.5.5
  83 20091125-crashdata.csv:timeGetTime 3.5.5

Comment 2

[chris hofmann]

this gets more strange by the minute

module correlations in
20091126_Firefox_3.5.5-interesting-modules-with-versions.txt
say HOOK.dll is strongly correlated

  timeGetTime|EXCEPTION_ACCESS_VIOLATION (82 crashes)
     80% (66/82) vs.   0% (93/99159) HOOK.DLL

http://www.processlibrary.com/directory/files/hook/20068 says HOOK.dll is evil.

belongs to the Backdoor.Spymon Trojan. This Trojan allows attackers to access your computer from remote locations, stealing passwords, Internet banking and personal data. This process is a security risk and should be removed from your system. 

common path is

 %system%\hook.dll

I'm guessing it may be hard to block from there, and may, or may not have much impact on this particular crash if we did try and block.

Another question might be why would winmm.dll (Windows Multimedia and top frame on the stack) be running so close to startup?

Comment 3

[chris hofmann]

http://www.symantec.com/security_response/writeup.jsp?docid=2005-112514-4016-99&tabid=2  has info in the "Technical Details" section about how hook.dll gets launched on windows start up and Opens a back door on a TCP port.

Comment 4

[timeless]

winmm is the windows multimedia subsystem (roughly)

http://msdn.microsoft.com/en-us/library/dd757629(VS.85).aspx

since timeGetTime is defined in winmm, it isn't an interesting bystander.

http://mxr.mozilla.org/mozilla-central/source/nsprpub/pr/src/md/windows/ntinrval.c#65

timeGetTime is the way nspr implement PR_GetIntervalNow which is used by the timer to figure out scheduling. None of this is interesting.

Comment 5

[Benjamin Smedberg]

Is there any reason we shouldn't just blacklist hook.dll?

Comment 6

[timeless]

nope. please blocklist it.

in the long run, it doesn't scale, i'd suggest we only allow modules which are either:
signed (with a valid signature)
or
in places where we specifically try to load them (extensions, plugins)
-- anything else should just get a valid signature if it wants us to load it, or stay out of our process space :).

Comment 7

[Johnathan Nightingale [:johnath]]

Attachment: Blocklist hook.dll, all versions

Process library:

 http://www.processlibrary.com/directory/files/hook/

and McAfee threatscan

 http://vil.nai.com/vil/content/v_142258.htm

and chofmann's Symantec link all agree that this is malware, and I confirm it doesn't show up in a default Vista install.

Here's a patch that blocks all versions, since it doesn't have a reported version in crash reports.

To avoid giving every blocklist review to vlad as punishment for writing this code, I'm gonna deputize a couple other people to do blocklist entry reviews - they are barely code change, anyhow. bsmedberg and sicking feel like they can handle "3 line edits to an array."

WHO DARES OPPOSE?!

Comment 8

[Benjamin Smedberg]

Comment on attachment 415623 [details] [diff] [review]
Blocklist hook.dll, all versions

I dare not!

Comment 9

[Johnathan Nightingale [:johnath]]

http://hg.mozilla.org/mozilla-central/rev/9d964e429ec4

Comment 10

[Johnathan Nightingale [:johnath]]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/2afd48548a13