467167 - Mozilla crashes on startup [@ _PR_MD_SEND ] [@ VirtualAllocEx ] [@ send ] because of various buggy LSPs

Status: RESOLVED INCOMPLETE

(Core :: Networking, defect)

Description

[Peter Hollin]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.4) Gecko/2008102920 Firefox/3.0.4
Build Identifier: firefox 3.0.4 on vista SP1, latest updates

Owner reports this started happening three weeks ago with no drastic changes to computer.  Owner has tried reinstallation.  I've removed google toolbar, removal of profile directory, reinstallation, deletion of prefetch profile, and manual deletion of directories after uninstallation.  Possible I missed some directory, as owner is using vista and I'm not familiar with all the filesystem layout changes.  Deleted information from %profile%\appdata\local %profile%\appdata\roaming did not find global application data for FF in vista directory for that (can't remember location, but I verified it was the right one).

Also Based on #403466, installed thunderbird to see if problem was common to all mozilla apps.  Thunderbird ran just fine.

Based on crash data, tried disabling virus scanner (avg) and windows firewall.  No change.  I do see evidence of a spyware infection attempt (suspicious entries in startup file pointing to executables in appdata) but windows has not allowed those to run.  Possible winsock bust by malware?  Dates on files match date problem started.  Tried netsh winsock reset to no avail.  Also other network apps working fine (safari, ie).  User correlates ff crashes starting after viewing a youtube video, and virus scanner reporting a problem of some kind at that time.

Reproducible: Always

Steps to Reproduce:
1.  Start firefox in regular or safe mode
2.  ???
3.  Profit!
Actual Results:  
crash before profile manager runs

Expected Results:  
profit!

User is running vanilla copy of FF - nothing special.  Only running adblock extensions.

Comment 1

[Matthias Versen [:Matti]]

Install Firefox in a clean directory, (remove the old application directory first), create a new profile http://support.mozilla.com/en-US/kb/Managing+profiles.

If you still crash read :
http://developer.mozilla.org/En/How_to_get_a_stacktrace_for_a_bug_report
If the crash reporter doesn't come up, use windbg. To get the crash ID look in the profile directory for the crashdata subdirectory and get the crash-ID from there (example id: 0777843b-d2e9-427d-9ac3-479bf2081127).
As this is a single issue on a single system you should have go to http://support.mozilla.com instead of bugzilla. I bet this is a spyware infection on the system.

Comment 2

[Peter Hollin]

Hi, Matthias

Thanks for responding.

Step #1 already performed - new installation crashes before profile can be created.

crash id is referenced in the URL field http://crash-stats.mozilla.com/report/index/7667839e-8756-4221-bee2-10e5d2081129?p=1

I've checked at support.mozilla.com and haven't found any similar issues.  Looking at the stack trace it appears as though it's crashing on the network calls somehow but I don't understand it well enough to be sure on that.  I could simply wipe the system and start fresh, but then I have to backup the user's system and so forth and we'd never really know what happened.  But I think there's something more going on here and that there may be a bug underlying it.  Especially since it doesn't happen with other applications, and neither the virus scanner nor the antispyware report current problems.  I'd like for someone more familiar with the codebase than I am to look at the stack and give me some hints for where to look.  I've been a network engineer since the mid 90's and am willing and able to troubleshoot things, but I am not a systems-level programmer and stack traces aren't really my thing.

I agree there's an infection (aborted or successful).  But my instincts tell me there's more to this than just malware - if other applications aren't crashing and firefox is then users will assume the problem is firefox.  I'll have the user run a rootkit detector as well, but my suspicion is that the malware attempt at installation is the problem and that it did not actually install.

Thanks for all the work you put in,
Peter

Comment 3

[Matthias Versen [:Matti]]

locate icx.dll on the system and remove for it a try (create a backup for later analysis or if it belongs to some software that stops working) if you can't assign it to a specific application.
If that doesn't help uninstall the google desktop.

Comment 4

[Matthias Versen [:Matti]]

>if other applications aren't crashing and firefox is then users will assume the >problem is firefox.

Sure but there is nothing we could do against that.

BTW: I suggest that you look that the user upgraded for example flash to flash10 and you have to upgrade both flash version: ActiveX(IE) and All other browsers (NPAPI) flash player.There are known security holes in 9.0.124. I tend myself to install Secunia PSI for such users.

Comment 5

[timeless]

Signature @0xdf8ce5 
UUID 7667839e-8756-4221-bee2-10e5d2081129 
Time 2008-11-29 09:55:40-08 
Uptime 2 
Last Crash 540 seconds before submission 
Product Firefox 
Version 3.0.4 
Build ID 2008102920 
Branch 1.9 
OS Windows NT 
OS Version 6.0.6001 Service Pack 1 
CPU x86 
CPU Info GenuineIntel family 6 model 15 stepping 13 
Crash Reason EXCEPTION_ACCESS_VIOLATION 
Crash Address 0xdf8ce5 
Comments  

Crashing Thread
Frame Module Signature [Expand] Source 
0  @0xdf8ce5  
1 nspr4.dll _PR_MD_SEND mozilla/nsprpub/pr/src/md/windows/w95sock.c:357  
2 nspr4.dll SocketSend mozilla/nsprpub/pr/src/io/prsocket.c:694  
3 nspr4.dll SocketWrite mozilla/nsprpub/pr/src/io/prsocket.c:714  
4 nspr4.dll PR_SetPollableEvent mozilla/nsprpub/pr/src/io/prpolevt.c:498  
5 xul.dll nsSocketTransportService::Shutdown mozilla/netwerk/base/src/nsSocketTransportService2.cpp:440  
6 xul.dll xul.dll@0x7c534b  

icx.dll    
GoogleDesktopNetwork3.dll 5.7.801.7324 5F2503C82DC540218B078020C36D705B4 GoogleDesktopNetwork3.pdb

Comment 6

[Samuel Sidler (old account; do not CC)]

This is currently the #5 topcrash for Firefox 3.0.6.

Jason, anything in the signature point to any obvious problems or should we assume this is from an evil add-on of some kind?

Comment 7

[Peter Hollin]

I haven't been in contact with owner of this system recently.  I'll check for other bugs referencing this crash and mark as resolved if active troubleshooting is happening elsewhere.

Comment 8

[timeless]

ss: (not sure who jason is), crashes under _PR_MD_SEND are generally LSPs

http://en.wikipedia.org/wiki/Layered_Service_Provider

IME, LSPs are buggy but getting a vendor to fix there's is....

Comment 9

[Kevin Brosnan]

This looks to be the same as the Number one crash on 1.9.1
http://crash-stats.mozilla.com/query/query?do_query=1&branch=1.9.1&date=&range_value=4&range_unit=days&query_search=signature&query_type=exact&query=

Comment 10

[Jason Duell]

This is happening inside the NSPR socket code for Windows.  It's not code I'm at all familiar with;  you may have more luck with the NSPR maintainers.   I'm moving the component for the bug to NSPR.

Comment 11

[Nelson Bolyard (seldom reads bugmail)]

The URL in comment 9 doesn't work for me.  Try:
http://crash-stats.mozilla.com/report/list?product=Firefox&branch=1.9.1&query_search=signature&query_type=exact&query=&date=&range_value=4&range_unit=days&do_query=1&signature=_PR_MD_SEND

Comment 12

[timeless]

so the vast majority of those are [@ 0x10a51e39 - _PR_MD_SEND]
http://crash-stats.mozilla.com/report/index/cbda6bcf-7ae2-4dd8-bbd3-63b7b2090406
iS3lsp.dll is among the libraries....

conveniently, there's actually a list of LSPs:
http://www.spywaredata.com/spyware/spyware-adware/lsp/1/results.php

http://crash-stats.mozilla.com/report/index/abdb5612-1481-40f1-b11e-298142090406 has voymoy.dll which is tagged as a trojan (along w/ a bunch of other dlls w/ essentially 0 hits) - that user or class of users repeats.

http://crash-stats.mozilla.com/report/index/8e6d66d9-6ab8-443b-89fc-4ed302090406 has ipdll.dll whichc is tagged as a trojan

note that it's possible for libraries to load at fixed addresses (this generally happens) and to unload (we're not used to this, but it can happen). if things don't unload 'properly', it's easy for them to appear as fixed addresses w/o a library name.

wrt line numbers... some crashes are at 357 and some are at 375
357         while ((rv = send( osfd, buf, amount, 0 )) == -1) 
373         }
374         bytesSent += rv;
375         if (fd->secret->nonblocking)
376         {
377             break;
378         }
379         if (bytesSent < amount) 
380         {
381             rv = socket_io_wait(osfd, WRITE_FD, timeout);

357 contains send which would be the WinSock entrypoint to the LSPs.
My guess is that 375 is the optimizer's rewrite of 357 since 373 is the while loop and 374 is probably resequenced with 377 (as 381 and 357 do not need it.

Comment 14

[Wan-Teh Chang]

It's crashing here:

http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/nsprpub/pr/src/md/windows/w95sock.c&rev=3.15&mark=357#347

It is best to investigate this bug as a Core:Networking bug.
A crash in an NSPR function doesn't mean it's an NSPR bug.

Comment 15

[Matthew Middleton (:zzxc)]

This crash has been reported a lot on SUMO as well.

Two users have reported that this crash with address 0x10001e39 (eg. bp-1e65916d-a58e-4972-bbeb-d30442090415 and bp-542f9450-937c-47f5-8bce-c9d772090415) was fixed by removing the mdnsNSP.dll LSP (From Apple's Bonjour) using the lspfix utility, but the same crash has also happened without mdnsNSP.dll.

The user reporting bp-542f9450-937c-47f5-8bce-c9d772090415 confirmed that no LSP's other than the ones included with Windows were enabled.  This user also scanned with Rootkit Revealer and found no rookits.

Some other users have tried to debug the crash with windbg (using the instructions at https://developer.mozilla.org/En/How_to_get_a_stacktrace_with_WinDbg ), but this crash hasn't occurred with windbg enabled.  An example of a windbg report from someone experiencing this crash is https://support.mozilla.com/tiki-view_forum_thread.php?forumId=1&comments_parentId=324459

Comment 16

[timeless]

so, comments_parentId=324459 seems to be flash9, just tell the user to upgrade to flash10.

given that you've confirmed removing some lsp's have fixed the problem for some users, then i claim i'm right and we should talk to MS and the LSP providers and figure out who's at fault. I'm fairly confident it isn't us.

Comment 17

[Matthew Middleton (:zzxc)]

Another user on SUMO confirmed that the crash still happens after disabling the mdnsNSP LSP.  The user was able to reproduce the problem by searching Google for: exception access

This search immediately crashed with report bp-dc2f473f-c844-413e-bf24-5c8b52090416

Comment 18

[timeless]

fslsp.dll 	2.1.580.0 	AE9628E8125943BE9906CB89DB3D58531 	fslsp.pdb

http://www.spyany.com/files/fslsp_dll.html

fslsp.dll is a Winsock Layered Service Provider module for F-Secure anti-virus software. This is a required module for F-Secure anti-virus software properly and should not be removed.

Offhand, I'd blame fsecure in this instance. It's hard to tell though w/o a list of unloaded libraries.

Ask this user to try disabling fsecure (this might involve setting up a real hardware firewall and then uninstalling fsecure). if it's fsecure's fault, then someone should complain to them.

Comment 19

[Ricardo Maçãs [:Ricmacas]]

Hello, I'm the live chat supporter for comments_parentId=324459 and i debugged Happygal's crash. 
@timeless:
After the debugging session in Live Chat, I immediately told her to post the results. I saw the debugging report and, still in Live Chat, i told her to disable Flash through the traditional Tools-Addons-Plugins method. She didnt experience any crash the next few hours. But then, as she posts, "FF worked great last evening. I woke up and went to the computer only to find a crash report."
So Flash was not the problem after all.
Also, I can't find any instance of the F-Secure's dll on HappyGal's crash reports. 
I will just ask her if she has ever had F-Secure installed on her PC...
Posting futher information here as soon as I have it.
Regards,
Ricmacas

Comment 20

[Ricardo Maçãs [:Ricmacas]]

Also, i would like to stress that she says: "I've been using malware and spyware detectors, and although they have cleaned up a lot of files (mostly cookies), FF continues to crash. ", "despite trying several malware programs and avast pre-boot software".

If its malware, its not recognized by most software vendors.
Also, her computer was highly unstable: "The computer basically isn't working any more", "I also have Skype randomly opening a window on me, and IE is also crashing quite frequently but not as much as mozilla.", "I'm going to end up wiping out the hard drive and reinstalling all my software. I know that'll clean up the bug. "

Ricmacas

Comment 21

[Matthew Middleton (:zzxc)]

(In reply to comment #18)
> fslsp.dll     2.1.580.0     AE9628E8125943BE9906CB89DB3D58531     fslsp.pdb
> 
> http://www.spyany.com/files/fslsp_dll.html
> 
> fslsp.dll is a Winsock Layered Service Provider module for F-Secure anti-virus
> software. This is a required module for F-Secure anti-virus software properly
> and should not be removed.
> 
> Offhand, I'd blame fsecure in this instance. It's hard to tell though w/o a
> list of unloaded libraries.
> 
> Ask this user to try disabling fsecure (this might involve setting up a real
> hardware firewall and then uninstalling fsecure). if it's fsecure's fault, then
> someone should complain to them.

This user tested by disabling the f-secure LSP using lspfix, but the crash still occurred.  (Several people have crashed without this or any other non-default LSP installed.)

A number of people with this stack have also mentioned that Safari and/or IE crash as well.(In reply to comment #18)
> fslsp.dll     2.1.580.0     AE9628E8125943BE9906CB89DB3D58531     fslsp.pdb
> 
> http://www.spyany.com/files/fslsp_dll.html
> 
> fslsp.dll is a Winsock Layered Service Provider module for F-Secure anti-virus
> software. This is a required module for F-Secure anti-virus software properly
> and should not be removed.
> 
> Offhand, I'd blame fsecure in this instance. It's hard to tell though w/o a
> list of unloaded libraries.
> 
> Ask this user to try disabling fsecure (this might involve setting up a real
> hardware firewall and then uninstalling fsecure). if it's fsecure's fault, then
> someone should complain to them.

This user tested by disabling the f-secure LSP using lspfix, but the crash
still occurred.  (Several people have crashed without this or any other
non-default LSP installed.)

A number of people with this stack have also mentioned that Safari and/or IE
crash as well.

Comment 22

[Tyler Downer [He/Him]]

(In reply to comment #20)
This users sounds like she has other issues. if not malware, it could be a legit looking program. FF would not make the computer act like that, so while it does not rule out FF, it does make other variables more possible, and makes it difficult to detect the issue. A reinstall of the OS would help. If it still crashs when she has nothing else installed, it narrows it down.

Comment 23

[Ricardo Maçãs [:Ricmacas]]

Tyler: If you look at the forum thread, she says exactly that:

"I'm going to end up wiping out the hard drive and reinstalling all my software. I know that'll clean up the bug."

There must be an external cause, though.
Ricmacas

Comment 24

[Tyler Downer [He/Him]]

(In reply to comment #23)
uh, yeah, that's what I said. 
It could be security...

Comment 25

[Mike Beltzner [:beltzner, not reading bugmail]]

ss suspects a Bonjour update; can we try to reproduce this with iTunes and Bonjour installed on a computer?

Comment 26

[Tyler Downer [He/Him]]

(In reply to comment #25)
I have bonjour and itunes, with Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2a1pre) Gecko/20090421 Minefield/3.6a1pre (.NET CLR 3.5.30729) ID:20090421041537. no crash

Comment 27

[Samuel Sidler (old account; do not CC)]

I'm doing some crash analysis on reports from the last week. Taking for investigation...

Comment 28

[Samuel Sidler (old account; do not CC)]

Attachment: list of modules

This is a list of modules, sorted by frequency, for 11,919 crash reports over the last week or so. I've omitted version information and am digging through that list now.

Comment 29

[Joseph Jacob]

This may or not be relevant, but we encountered similar types of problems (although not the exact same types of crashes) in Bug 434403 (https://bugzilla.mozilla.org/show_bug.cgi?id=434403). Turned out Specter Pro was the problem.

Comment 30

[Samuel Sidler (old account; do not CC)]

None of the DLLs in that bug are in the list of DLLs I attached.

Comment 31

[timeless]

Attachment: analysis of attachment 373952

thanks, that's a start. note that http://www.spywaredata.com/spyware/spyware-adware/lsp/1/results.php indicates some LSPs do not have LSP in their name.

I did a quick analysis of the items w/ LSP in their name. Note that my "untagged" item mostly meant that processlibrary.com didn't say "this is an LSP". Things that have LSP in that column are either listed as such in that service or are listed by a debugging tool on the web as being "in the LSP stack".

Items i've marked with EVILBIT are EVIL. Anyone with those modules should seek anti-whichever.

Comment 32

[Noah (NoahSUMO/SolidSnake) [:Noah]]

http://pastebin.mozilla.org/644093

Some LSPs timeless asked me to google; contains threads mentioning these LSPs, potential fixes and diagnoses of what they are.

Comment 33

[chris hofmann]

from crash stats looks like about 4112 3.0.9 users hit this problem in the last week, and similar pct. are hitting on 3.5b4.  

djst, we should consider trying to draft an article.  any suggestions from anyone on what a suppport article might look like?

Comment 34

[Johnny Stenback (:jst)]

blocking- as this is unlikely something we can do anything about in the code.

Comment 35

[David Tenser [:djst]]

Cheng, is this crash id appearing in our forum and live chat? See also comment 33.

Comment 36

[David Tenser [:djst]]

Please ignore my previous comment -- I should have read up on the comments in the bug first.

Definitely sounds like we need an article once we know what's causing this, or at least know how to work around it (formatting the whole hard drive doesn't really sound like the solution we want to propose in a KB article).

Comment 37

[timeless]

djst/jst: wtf. we know what's causing it. it's a bunch of (presumably poorly written) LSPs.

do you really not have the time to read a bug with only 32 comments, most of them technical?

Comment 38

[David Tenser [:djst]]

Sorry, but I don't know what the solution in a KB article would be -- removing "a bunch of" LSPs? Which ones? How? 

If this it entirely clear to you, please help rather than being rude. Thanks.

Comment 39

[timeless]

we don't have a complete solution, if we had one, we'd have provided it.

http://www.cexx.org/lspfix.htm is the starting point, mostly it means trying to disable all the LSPs it lists, and ideally that'd result in the problem "going away". it'd also break whatever "feature" the various LSPs provide (among them are antivirus applications and virii).

Comment 40

[Ricardo Maçãs [:Ricmacas]]

I did some research on the internet, and found this:
http://www.online-tech-tips.com/computer-tips/winsock-fix/
We can try some of the suggestions in that website, not only lspfix :D .

Comment 41

[chris hofmann]

I saw a couple of interesting 3.5b4 comments where people pasted in some system info into their crash reports

_PR_MD_SEND     0xffffffff942c7a14      200905102115    24      46

        Add-ons: {972ce4c6-7e08-4474-a285-3208198ce6fd}:3.5b4\r\nBuildID: 20090423204732\r\nCrashTime: 1242015342\r\nEmail: [e-mail]\r\nInstallTime: 1242014711\r\nProductName: Firefox\r\nSecondsSinceLastCrash: 46\r\nStartupTime: 1242015318\r\nTheme: classic/1.0\r\nThrottleable: 1\r\nURL: \r\nVendor: Mozilla\r\nVersion: 3.5b4\r\n
        http://crash-stats.mozilla.com/report/index/395ef08c-dd6f-4625-b295-d51002090510


_PR_MD_SEND     0xffffffff94c97a14      200905102114    26      54

        Add-ons: {972ce4c6-7e08-4474-a285-3208198ce6fd}:3.5b4\r\nBuildID: 20090423204732\r\nCrashTime: 1242015296\r\nEmail: [e-mail]\r\nInstallTime: 1242014711\r\nProductName: Firefox\r\nSecondsSinceLastCrash: 54\r\nStartupTime: 1242015270\r\nTheme: classic/1.0\r\nThrottleable: 1\r\nURL: \r\nVendor: Mozilla\r\nVersion: 3.5b4\r\n\r\nThis report also contains technical information about
        http://crash-stats.mozilla.com/report/index/b8be7193-2d7c-47ad-a1bb-3b98c2090510


I also ran a search on that comment about Add-ons: {972ce4c6-7e08-4474-a285-3208198ce6fd}

This guy says he had not installed addons but somehow ended up with that folder.

http://www.raymond.cc/blog/archives/2008/06/27/my-temporary-solution-to-fix-mozilla-firefox-3-random-crashing-problem/

Removing the addon folder solved his crashing problem.  

This sumo forum thread also indicates that work for some, and not others.

Do we have a list of addon id's posted somewhere?

Comment 42

[Samuel Sidler (old account; do not CC)]

(In reply to comment #41)
> I also ran a search on that comment about Add-ons:
> {972ce4c6-7e08-4474-a285-3208198ce6fd}

Google says that's the default theme add-on.

Comment 43

[timeless]

chris...

ID: 395ef08c-dd6f-4625-b295-d51002090510
nmdfgds0.dll <- trojan: http://www.greatis.com/appdata/d/n/nmdfgds0.dll.htm
PVYSQV.DLL <- malware: http://www.prevx.com/filenames/496427370254512110-X1/PVYSQV.DLL.html
PROTO.DLL <- adware: http://www.prevx.com/filenames/X1262011632948320212-X1/PROTO.DLL.html
khfDutTM.dll <- no hits  	 	 	
xelodbad.dll <- no hits relevant

The other one has the same libraries, I have to assume it's the same user.

Since he left an email address, contact him and suggest his computer be cleaned (or nuked).

Comment 44

[Samuel Sidler (old account; do not CC)]

No longer investigating this since it's slowly disappearing from crash-stats...

Comment 45

[[:Cww]]

#3 crash still... maybe we should investigate again?

Comment 46

[chris hofmann]

1706 total crashes for _PR_MD_SEND on 20090826-crashdata.csv
779 start up crashes inside 3 minutes

_PR_MD_SEND related signature breakdown
signature distribution
       5
signature list
1694 _PR_MD_SEND
   9 strstr | _PR_MD_SEND
   1 @0x0 | @0x2c123d1 | _PR_MD_SEND
   1 @0x0 | @0x20223d1 | _PR_MD_SEND
   1 @0x0 | @0x10414dda | @0x10414f36 | _PR_MD_SEND

I think we really need to have the tools outlined in https://bugzilla.mozilla.org/show_bug.cgi?id=439679 to make headway on this bug.

for the shorter term I'll try and play around with something more primitive to see if I can identify some of the more serious LSP contributors to this problem.

Comment 48

[Jim Mathies [:jimm]]

I'm not convinced this specifically involves bad lsps, seems like it has more to do with crappy/virus/trojan/spyware modules loading into our address space. Maybe we should be working on a compiling a list of these and figure out a way to reject them from our process space?

Doing a survey of about 30 from today's reports -


http://crash-stats.mozilla.com/report/index/c1a4e0fd-950d-4a3a-8206-1bbf72091014

(long stack)
MWSOESTB.DLL - My Web Search Bar

http://crash-stats.mozilla.com/report/index/d27e7830-ff28-4267-84ef-42a622091014

(long stack)
Novel client - lots of modules loaded

http://crash-stats.mozilla.com/report/index/d4ba8d9b-09a6-4bb0-ac4d-1188b2091014

SPhoneParser.dll - Skype phone number parser
mgAdaptersProxy.dll - SweetIM
PNRComponent.dll - generic toolbar dev platform dll

http://crash-stats.mozilla.com/report/index/07af8f12-7ce5-458a-8e49-1af0e2091014

Google Desktop Network
cmcfg3232.dll - trojan?

http://crash-stats.mozilla.com/report/index/f006ea86-1f86-41a6-97a3-10f8f2091014

Google Desktop Network
SPhoneParser.dll - Skype phone number parser
ahivotuketox.dll - suspect
PNRComponent.dll - generic toolbar dev platform dll

http://crash-stats.mozilla.com/report/index/48060f33-39eb-46ea-be7c-f96642091014

calc.dll - spyware/virus
gasfkyuaqsrrwp.dll
cooliris19.dll

http://crash-stats.mozilla.com/report/index/9ca9e609-a99d-45e8-bfbe-a88702091014

(long stack)
BkavHook.dll - spyware/virus
UKHook35.dll - spyware/virus

http://crash-stats.mozilla.com/report/index/4302a870-ab1e-4569-8b0b-13e442091014

(long stack)
windows media player
SmileyCore.dll - trojan

http://crash-stats.mozilla.com/report/index/4e96a762-9700-4931-b1d8-cd2152091014

calc.dll - spyware/virus

http://crash-stats.mozilla.com/report/index/6ddee1a3-fb7e-4ba0-ae78-ba41a2091014

BkavHook.dll - spyware/virus
UKHook40.dll - spyware/virus

http://crash-stats.mozilla.com/report/index/451e1baf-4fa0-4e25-9d24-f0a422091014

Uptime, 1 second, no bad modules that I could spot

http://crash-stats.mozilla.com/report/index/df288203-3009-452a-9149-051fa2091014

(long stack)
Groove networks
fsecure lsp

http://crash-stats.mozilla.com/report/index/214bb1eb-08fe-4134-b2fd-c0f842091014

MSNChatHook.dll - virus

http://crash-stats.mozilla.com/report/index/c73bed6a-dd6d-4247-8370-40de52091014

google toolbar
smum32.dll - spyware doctor

http://crash-stats.mozilla.com/report/index/4674be89-7b25-4d8a-9279-877a32091014

MSNChatHook.dll - virus

http://crash-stats.mozilla.com/report/index/690e5fa4-d03d-46b7-846c-bfe382091014

(long stack)
e086e193-c3df-7874-b7b9-bc82a276591f.dll - can't be good

http://crash-stats.mozilla.com/report/index/7e4791b2-a5a0-4501-a240-f99692091014

(long stack)
TempIadHide3.dll - spyware
PNRComponent.dll - generic toolbar dev platform dll
SPhoneParser.dll - Skype phone number parser
google talk

http://crash-stats.mozilla.com/report/index/e051699c-6989-4c3c-aacc-3f2da2091014

gamevancelib32.dll - trojan

http://crash-stats.mozilla.com/report/index/586a9ed4-a74f-4c7d-883a-4da3c2091014

idmmkb.dll - some download manager
rpchromebrowserrecordhelper.dll - real player

http://crash-stats.mozilla.com/report/index/60739601-b736-4fe7-a490-949622091014

(long stack)
avgrsstx.dll - avg
nothing sinister looking here.

http://crash-stats.mozilla.com/report/index/daaf2514-a8c0-4842-93f2-4ad572091014

calc.dll - spyware/virus

Comment 49

[Jim Mathies [:jimm]]

From the pre-beta of 3.6:

_PR_MD_SEND

http://crash-stats.mozilla.com/report/index/c455482c-8c49-4667-ba76-0bdb42091015

nothing suspect

http://crash-stats.mozilla.com/report/index/f0f79789-2d75-4351-a48b-085432091008

PackAssist.dll - worm
GameLink.dll - possible malicious lsp?

_PR_MD_CONNECT

http://crash-stats.mozilla.com/report/index/491bee1d-3880-483d-b65f-4c8c92091014

sockspy.dll - known for crashing browsers, BitDefender, anti-virus

http://crash-stats.mozilla.com/report/index/6eeb6e5b-393f-4a52-813f-cc84d2091013

vksaver.dll - malicious

3.5.3, top five from today:

_PR_MD_SEND

http://crash-stats.mozilla.com/report/index/b973e398-8c9d-40bb-8760-847f32091016

gamevancelib32.dll - trojan/virus
SpSubLSP.dll - anti-spam product

http://crash-stats.mozilla.com/report/index/7742be69-a0cd-49a1-8367-a0cf82091015

gamevancelib32.dll - trojan/virus

http://crash-stats.mozilla.com/report/index/912cf7cf-e44f-44ac-8c19-2e0ea2091015

392414xxx.dll - questionable
PrnTrack.dll - Pharos - google blocks this in chrome

http://crash-stats.mozilla.com/report/index/c114161d-9d83-47e0-b2e0-59a0e2091016

vksaver.dll

http://crash-stats.mozilla.com/report/index/d2e755df-d45d-4def-a568-d126a2091016

autochk.dll - trojan

I'd say 70-80% of these crashes clearly indicate some form of malicious infection.

It's sort of sad, we have people submitting crash reports lamenting that they don't know what's going wrong, and right there in the module list is some sort of worm, trojan, virus, or other malady. Couldn't we use crashreporter and a back end database (bug 439679?) to identify these so that after a crash report is submitted, cr could throw up a dialog warning the user?

Comment 50

[Johnny Stenback (:jst)]

(In reply to comment #49)
[...]
> It's sort of sad, we have people submitting crash reports lamenting that they
> don't know what's going wrong, and right there in the module list is some sort
> of worm, trojan, virus, or other malady. Couldn't we use crashreporter and a
> back end database (bug 439679?) to identify these so that after a crash report
> is submitted, cr could throw up a dialog warning the user?

Yes, sad indeed. I think the best thing we could do here is to detect this error before we crash. IOW, run thought the list of loaded DLLs on startup and check for known bad ones, and warn the user before we get far enough to touch the network, which may crash them again, etc. Now that's just my two cents, but it seems like a better solution than relying on the crash reporter. The question is, can we enumerate loaded libraries at a good point in time when starting up, w/o taking a big startup perf penalty hit, and check a local db that we install with the app and update periodically etc?

Thoughts?

Comment 51

[Ricardo Maçãs [:Ricmacas]]

Jim, I see that you have been investigating bug reports, but notice this:

1) The DLLs you found arent, let's say, "consistent", because you found DLLs from various types of malware and from various different threads that probably attack in totally different ways;
2) It still doesnt explain the crashes from people without malware.

My conclusion here is that the probability of having this kind of crash is much higher when you have malware installed, maybe because they mess with networking and LSPs. People experiencing the crash with no malware could have had malware previously installed or they simply have their configurations messed up.

And yeah, we could add some kind of "Web of trust" / "Norton Insight" / "Phishtank" -like service that:
1) Listens for actual user feedback on infected DLLs
2) Checks statistics for suspicious DLLs (how many people use it, how long has it been on the internet, trustworthiness, how many crashes result from that DLL, etc.)
3) Checks externally for known bad DDLs and compiles a list of them.

Which is basically what we're MANUALLY DOING on this bug.

Bug 439679 seems to be more about manually creating a "reports that give us trends in the existence of adware/malware to watch for attacks or outbreaks of malware affecting firefox" so I could suggest moving this discussion to a new bug that clearly proposes a cloud-based alerting system that would effectively tell people that the crash they are experiencing could be a result of malware.

Negative aspects of this would be, of course, people thinking that Mozilla Firefox is an antivirus or some kind of virus detector because of those alerts, specially the less net-savvy ones. It would have to be something to consider. These negative aspects mean that directly alerting users might not be the best idea. "Suggesting a virus scan" could be better, for example.

Nevertheless, my suggestion is to open a new bug and move it to there.
Ceasing conversation about this feature here would be, IMO, a good idea.

Those are my thoughts.

Comment 52

[Jim Mathies [:jimm]]

(In reply to comment #50)
> (In reply to comment #49)
> [...]
> > It's sort of sad, we have people submitting crash reports lamenting that they
> > don't know what's going wrong, and right there in the module list is some sort
> > of worm, trojan, virus, or other malady. Couldn't we use crashreporter and a
> > back end database (bug 439679?) to identify these so that after a crash report
> > is submitted, cr could throw up a dialog warning the user?
> 
> Yes, sad indeed. I think the best thing we could do here is to detect this
> error before we crash. IOW, run thought the list of loaded DLLs on startup and
> check for known bad ones, and warn the user before we get far enough to touch
> the network, which may crash them again, etc. Now that's just my two cents, but
> it seems like a better solution than relying on the crash reporter. The
> question is, can we enumerate loaded libraries at a good point in time when
> starting up, w/o taking a big startup perf penalty hit, and check a local db
> that we install with the app and update periodically etc?
> 
> Thoughts?

Just alerting the user doesn't protect us from crashes unless we find a way to reject / trap the entry points of these libraries. A local list is more proactive though. The negative side of a local list is that the list will become stale. The advantages of the server side/crash reporter approach are we have more control over the active list (a longer list, dealing with false positives, adding entries, removing them, etc.) Crash reporter is also a separate, simpler process that can run in the background to handle the job. To do this right and protect against false positives, we would need checksums of suspect files, that's not something we can do in startup on the fly without dragging the system down. Personally I would lean toward changes in crash reporter / server side for these reasons. 

We might look at how google is handling this in their renderer process, they have a single static list embedded in their code of plugin modules they reject. The list is pretty small -

http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/sandbox_policy.cc?view=markup&pathrev=27445

(In reply to comment #51)
> Negative aspects of this would be, of course, people thinking that Mozilla
> Firefox is an antivirus or some kind of virus detector because of those alerts,
> specially the less net-savvy ones. It would have to be something to consider.
> These negative aspects mean that directly alerting users might not be the best
> idea. "Suggesting a virus scan" could be better, for example.

This is something we should definitely think about, currently we are a passive player in all this. Becoming proactive means we become a target.

Comment 53

[Jesse Ruderman]

Will bug 524904, "Add support for generic DLL blocklist", help here?

Comment 54

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Seems to be correlated with:
kbdnet.dll
BkavHook.dll
calc.dll
in that each of these is responsible for 4-10% of the crashes with this signature.  That's not nearly all of them, though.

Comment 55

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Also:
rundll32.dll

and there could also be a bunch of correlations falling under the 5% threshold to show up in the reports.

Comment 56

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

And:
msxm192z.dll
which showed up in the past few days only.

Comment 57

[chris hofmann]

at least a few sources say calc.dll is evil
http://www.greatis.com/appdata/d/SysDir/c/calc.dll.htm
http://www.prevx.com/filenames/263801209745593736-X1/CALC.DLL.html

Comment 58

[timeless]

bug 523410 talks about blocking LSPs in general, that's the better approach than trying to manually add all of the LSPs in the world to a blocklist.

Comment 59

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Though I think it's unclear that it's only LSPs that are messing with us here, see comment 48.

Comment 60

[timeless]

jonas: based on where the crash is, and how the code works, the place where we are is basically an entrypoint to LSP land. i'd like to start by locking out LSPs. We can re-review the domain after we've limited the range :). It could be that the evil creatures hack LSP land without registering as proper LSPs, I'm not quite sure how much we could do about that, but let's find out first....

Comment 61

[Scoobidiver (away)]

It is #35 top crasher in b7pre build for the last week.

More reports at:
http://crash-stats.mozilla.com/report/list?range_value=2&range_unit=weeks&signature=_PR_MD_SEND&version=Firefox%3A4.0b7pre

Comment 62

[Sheila Mooney]

So what's the deal with this bug. Is it a blocker for 2.0?

Comment 63

[Johnny Stenback (:jst)]

Not blocking on this particular bug (there's other work being done wrt LSPs which I believe is blocking, or already landed).

Comment 64

[chris hofmann]

we would probably need an exception to the feature freeze to get one of the proposals about LSP blocking/whitelisting in to 4.0 at this point.   we might have missed the firefox 4 boat for an idea on that.

it does seems like thats the approach we are heading to solve at least some of the crashes under this particular signature.

we could spin of a bug to block calc.dll to get a bit of improvement, but thats not tied to firefox 4.    

This is a good and high impact bug to track and make progress on outside of the firefox 4 ship schedule.

Comment 65

[Radek 'sysKin' Czyz]

Agter bug 523410, this is essentially win 2000 and XP only, isn't it?
Or do you still expect multiple buggy LSPs to pass bug 523410's categories check.

Comment 66

[Scoobidiver (away)]

> Agter bug 523410, this is essentially win 2000 and XP only, isn't it?
It happens only on Windows XP.
It is now #14 top crasher in b8pre build for the last week.
I am requesting a blocking status for Gecko 2.0.

More reports at:
http://crash-stats.mozilla.com/report/list?range_value=4&range_unit=weeks&signature=_PR_MD_SEND&version=Firefox%3A4.0b8pre

Comment 67

[Sheila Mooney]

jst, this was renominated. Can you look at it and make a call.

Comment 68

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Has this been an issue since we started filtering LSPs on Vista+?  AFAICT these are exclusively Windows XP crashes now.  I'd suggest that we resolve this, there's probably not much we can do for users on the older versions.

Comment 69

[Sheila Mooney]

So do people agree we should resolve this? Just trying to clean out the nom queue.

Comment 70

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Yes per comment 68.

Comment 71

[Scoobidiver (away)]

Comment 66 is still applicable : #14 top crasher in 4.0b8pre.
If you don't want to fix it, a WONTFIX status would be better.

Comment 72

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

We've been at under 15 crashes a day for two weeks except for a major spike yesterday that I can't really explain ... we can reopen this if it persists.

Comment 73

[Scoobidiver (away)]

> We've been at under 15 crashes a day for two weeks except for a major spike
> yesterday that I can't really explain
The spike persists in recent 4.0b8pre builds (100 crashes/buildday).
It is now #1 top crasher in 4.0b8pre for the last 3 days.
See comment 66 for crash reports link.
So I reopen the bug.

Comment 74

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Sigh.

OK, so the crashes are still almost all on Windows XP.  Also of interest, the correlation scripts say:

81% (35/43) vs.  27% (791/2954) [e-mail] (Java Quick Starter, http://java.sun.com/javase/downloads/)

Comment 75

[Scoobidiver (away)]

> 81% (35/43) vs.  27% (791/2954) [e-mail] (Java Quick Starter,
> http://java.sun.com/javase/downloads/)
There is no update of Java on 11/03/2010. Version 1.6.22 was released before.
If it was due to an external update, it would be visible in the crash correlation by version (modules, addons). I don't see anything.

Here is the regression range for the crash spike:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=45d6a73ef138&tochange=0c4fb639c49e

Comment 76

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to comment #75)
> Here is the regression range for the crash spike:
> http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=45d6a73ef138&tochange=0c4fb639c49e

Almost everything in that log is NPOTB.  It started on a Wednesday, maybe MS released a patch on the Tuesday that's relevant?  I don't think the additional crash volume is anything we did.

Comment 77

[Scoobidiver (away)]

> It started on a Wednesday, maybe MS released a patch on the Tuesday that's
> relevant?
There is no release of MS patches for Windows XP on this date:
http://www.microsoft.com/downloads/en/results.aspx?freetext=windows+xp&displaylang=en&stype=s_basic

Comment 78

[[e-mail]]

I'm pretty sure we can add bug 601089 to the crash count here. For the past week, we had a total of 534 crashes with that signature, across 1.9.2 and 2.0; c.f. 11272 here. So, not a big addition.

The curious thing is that the sig in bug 601089 only shows up on 2.0, not 1.9.2; unclear right now why that is.

Comment 79

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Bug 611930 could provide us a lot more information here.

Comment 80

[[e-mail]]

As noted above, during 11/2 - 11/3 there was a major uptick in crashes; from:

117 (11/2) -> 1903 (11/3) on 2.0, and -> 3638 on 11/4
576 (11/2) -> 1007 (11/3) on 1.9.2

Both figures were roughly stable before and after the jump. So this affected 1.9.2 and 2.0 simultaneously (though much less on 1.9.2!?), which pretty much wraps up that it's a result of fresh malware.

Comment 82

[Johnny Stenback (:jst)]

Given the nature of this bug, I don't think we can block on this.

Comment 83

[chris hofmann]

If this is malware it might be isolated to particular regions like Brazil.

looking at crash urls for 4.0b8pre and 4.0b7 an unusually high number seem to be from Brazilian domains.  For example for yesterday on 4.0b7 of the 221 crash urls between 1/3rd to half are from .br.  looks like about the same for 4.0b8pre

  45 www.orkut.com.br
   3 letras.terra.com.br
   2 ziggi.uol.com.br
   2 www.uol.com.br
   2 www.terra.com.br
   2 www.lojavirus.com.br
   2 www.baixaki.com.br
   2 espnbrasil.terra.com.br
   1 www.yahoo.com.br
   1 www.parana-online.com.br
   1 www.naosalvo.com.br
   1 www.mtv.com.br
   1 www.lancenet.com.br
   1 www.habbo.com.br
   1 www.google.com.br
   1 www.correiodoestado.com.br
   1 www.cielo.com.br
   1 www.brazil-tv.rg3.net
   1 webmail.itelefonica.com.br
   1 produto.mercadolivre.com.br
   1 portal.aridesa.com.br
   1 noticiasaovivo.terra.com.br
   1 mundocanibal.uol.com.br
   1 mail.uol.com.br
   1 gostei.abril.com.br
   1 games.levelupgames.uol.com.br
   1 carrinho.casasbahia.com.br
   1 capricho.abril.com.br
   1 br.yahoo.com
   1 alfa-industria-moveis-hospitalares-lt.br.telelistas.net

same pattern seems to go back to early in nov.

Comment 85

[Scoobidiver (away)]

Now there are enough crash stats with LSP in App notes.
My first thought is that buggy LSPs are exclusively English, meanwhile domains are mainly Brasilian.

Chris,
Is is possible to have correlations by LSP and module version (same as dbaron's script) for both crash signatures and the following LSPs:
* MSAFD Tcpip [TCP/IP]
* MSAFD Tcpip [UDP/IP]
* MSAFD Tcpip [RAW/IP]
* RSVP UDP Service Provider
* RSVP TCP Service Provider
* MSAFD Irda [IrDA]
* MSAFD Pgm (RDM)
* MSAFD Pgm (Stream)
* MSAFD RfComm [Bluetooth]
* MSAFD NetBIOS [\Device...] SEQPACKET .
* MSAFD NetBIOS [\Device...] DATAGRAM .
* MSAFD nwlnkspx [...] ...
* MSAFD nwlnkipx [...]

Comment 86

[timeless]

scoobi: there's no point, we need to fix our lsp enum stuff so it skips the ms lsp's :(

Comment 87

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Well, we really just need to be able to get them all, but we need Socorro to help us out on that.

Comment 88

[chris hofmann]

Attachment: simple inventory of pr_send tcpip lsp's

its pretty easy for anyone to get samples of this data since app data has been added to the .csv crash analysis files.  get a crash data file like 20101214-pub-crashdata.csv.gz file from a location like http://people.mozilla.com/crash_analysis/20101214/

then run something like

awk -F\t '$1 ~ /_PR_MD_SEND/ && $8 ~ /4.0/ {print $26}' 20101213* | awk -F'|' '{for ( i = 1 ;  i <= NF ; i++ ) print $i }'  | sort | uniq -c | sort -nr | grep -v AdapterVendor > pr_md_send-wsock.txt

this filters on the signature and Firefox 4.0x reports then, gets counts and  strips out the graphics card/driver info.

Is this kind of inventory what we are after?

Comment 89

[chris hofmann]

Attachment: VirtualAllocEx wsock inventory

Comment 90

[chris hofmann]

but I guess https://bugzilla.mozilla.org/show_bug.cgi?id=611930#c15 is indicating that this might be and incomplete set of information.

Comment 91

[Scoobidiver (away)]

For the _PR_MD_SEND crash signature, it is #9 top crasher in 4.0.1.
Almost all comments are in Russian and it is correlated to VKsaver 2.2.2.0:
     67% (30/45) vs.   3% (333/10127) vksaver.dll
          2% (1/45) vs.   1% (108/10127) 
         64% (29/45) vs.   2% (210/10127) 2.2.2.0
For VKsaver crashes, see bug 614966.

Comment 92

[Sheila Mooney]

This is still around but really feel in the ranking. I don't think it qualifies as a top crash anymore.

Comment 93

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 94

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 95

[Randell Jesup [:jesup] (needinfo me)]

Clear wildptr crashes; ongoing at a low rate. Perhaps older OSes? Clearing old/automatic priority

Comment 96

[Andrew McCreight [:mccr8]]

Could you please file a new bug for this? This signature looks very generic and presumably the crashes here are unrelated to the STR in comment 1.

Also, if you find crashes that look like they are bad, please link them, so other people don't have to try and figure out which crashes you might be talking about.

Comment 97

[Randell Jesup [:jesup] (needinfo me)]

Filed 2 bugs to replace this, one a sec bug

Comment 98

[BugBot (nomail) [:suhaib / :marco/ :calixte]]

Since the bug is closed, the stalled keyword is now meaningless.
For more information, please visit auto_nag documentation.