Closed Bug 801394 Opened 12 years ago Closed 11 years ago

crash in nsInputStreamPump::OnInputStreamReady with randomly named DLL (malware)

Categories

(Core :: Networking, defect)

18 Branch
x86
Windows 7
defect
Not set
critical

Tracking

()

RESOLVED INCOMPLETE
Tracking Status
firefox18 + affected

People

(Reporter: scoobidiver, Assigned: benjamin)

References

(Blocks 1 open bug)

Details

(Keywords: crash, regression, Whiteboard: [startupcrash])

Crash Data

It's #2 top crasher in 18.0a2 with many dupes. It first appeared in 18.0a1/20121005. The regression range might be (discontinuous across builds):
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=4cb8f88213f5&tochange=fd724f194a1f

It's correlated to various modules:
*Oct 13:
  nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*)|EXCEPTION_ACCESS_VIOLATION_EXEC (19 crashes)
     58% (11/19) vs.   6% (11/170) rpchrome150browserrecordhelper.dll (RealPlayer)
     42% (8/19) vs.   5% (8/170) DockShellHook.dll (ObjectDock)
*Oct 14:
  nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*)|EXCEPTION_ACCESS_VIOLATION_EXEC (74 crashes)
     61% (45/74) vs.   5% (60/1135) GoogleDesktopNetwork3.dll (Google Desktop - discontinued)
     28% (21/74) vs.   4% (48/1135) datamngr.dll (MediaBar - spyware)
     28% (21/74) vs.  10% (109/1135) rpchrome150browserrecordhelper.dll (RealPlayer)

Signature 	nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) More Reports Search
UUID	4c66ed34-acd5-4a1e-a09d-726c72121012
Date Processed	2012-10-12 04:41:45
Uptime	2
Last Crash	12 seconds before submission
Install Age	1.1 days since version was first installed.
Install Time	2012-10-11 02:42:39
Product	Firefox
Version	19.0a1
Build ID	20121010030605
Release Channel	nightly
OS	Windows NT
OS Version	6.1.7600
Build Architecture	x86
Build Architecture Info	AuthenticAMD family 16 model 4 stepping 3
Crash Reason	EXCEPTION_ACCESS_VIOLATION_EXEC
Crash Address	0x2ece5c
App Notes 	
AdapterVendorID: 0x10de, AdapterDeviceID: 0x1081, AdapterSubsysID: 15723842, AdapterDriverVersion: 9.18.13.623
D2D? D2D+ DWrite? DWrite+ 
EMCheckCompatibility	True
Adapter Vendor ID	0x10de
Adapter Device ID	0x1081
Total Virtual Memory	4294836224
Available Virtual Memory	3954167808
System Memory Use Percentage	49
Available Page File	5718089728
Available Physical Memory	2164301824

Frame 	Module 	Signature 	Source
0 		@0x2ece5c 	
1 	xul.dll 	nsInputStreamPump::OnInputStreamReady 	netwerk/base/src/nsInputStreamPump.cpp:371
2 	xul.dll 	nsInputStreamReadyEvent::Run 	xpcom/io/nsStreamUtils.cpp:82
3 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:612
4 	xul.dll 	NS_ProcessNextEvent_P 	obj-firefox/xpcom/build/nsThreadUtils.cpp:220
5 	xul.dll 	nsThread::Shutdown 	xpcom/threads/nsThread.cpp:465
6 	xul.dll 	mozilla::crashreporter::LSPAnnotationGatherer::Annotate 	widget/windows/LSPAnnotator.cpp:46
7 	xul.dll 	nsRunnableMethodImpl<void 	obj-firefox/dist/include/nsThreadUtils.h:349
8 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:612
9 	xul.dll 	mozilla::ipc::MessagePump::Run 	ipc/glue/MessagePump.cpp:82
10 	xul.dll 	MessageLoop::RunHandler 	ipc/chromium/src/base/message_loop.cc:208
11 	xul.dll 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:182
12 	xul.dll 	nsBaseAppShell::Run 	widget/xpwidgets/nsBaseAppShell.cpp:163
13 	xul.dll 	nsAppShell::Run 	widget/windows/nsAppShell.cpp:232
14 	xul.dll 	nsAppStartup::Run 	toolkit/components/startup/nsAppStartup.cpp:290
15 	xul.dll 	XREMain::XRE_mainRun 	toolkit/xre/nsAppRunner.cpp:3792
16 	xul.dll 	XREMain::XRE_main 	toolkit/xre/nsAppRunner.cpp:3858
17 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3933
18 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:105

More reports at:
https://crash-stats.mozilla.com/report/list?signature=nsInputStreamPump%3A%3AOnInputStreamReady%28nsIAsyncInputStream*%29
Josh, can you please help find an assignee for this bug who can help here as it is a top crasher . Thanks !
Assignee: nobody → joshmoz
Networking folks - any idea what is going on here?
Assignee: joshmoz → honzab.moz
There is a set of crashes [1] where no malware seems to be involved.  Low frequency up to 14.0.1 version, all just Firefox and seems to be different from the major set.

The major set [2] seems to be mostly from infected machines.  I found a lot (actually almost all I checked had this) of reports that have apparently a random-generated name modules loaded (google finds *nothing* for them), I assume a malware.

For me this is a virus-caused crash.  Could be Vundo trojan/worm [3] or something quit new.  Could we reach some of the reporters?



[1] https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=nsInputStreamPump%3A%3AOnInputStreamReady&reason_type=contains&date=10%2F29%2F2012%2019%3A06%3A50&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=nsCOMPtr_base%3A%3Aassign_assuming_AddRef%28nsISupports*%29%20|%20nsInputStreamPump%3A%3AOnInputStreamReady%28nsIAsyncInputStream*%29

[2] https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=nsInputStreamPump%3A%3AOnInputStreamReady&reason_type=contains&date=10%2F29%2F2012%2019%3A06%3A50&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=nsInputStreamPump%3A%3AOnInputStreamReady%28nsIAsyncInputStream*%29

[3] http://en.wikipedia.org/wiki/Vundo
Typically, reaching out to users that are running into issues like this hasn't yielded any actionable results. If this were malware, I imagine we'd expect this to have a DLL correlation. KaiRo - can you help us grab a DLL correlation?
Flags: needinfo?(kairo)
(In reply to Alex Keybl [:akeybl] from comment #4)
> Typically, reaching out to users that are running into issues like this
> hasn't yielded any actionable results. If this were malware, I imagine we'd
> expect this to have a DLL correlation. KaiRo - can you help us grab a DLL
> correlation?

Robert, please see comment 3:

"I found a lot of reports that have apparently a random-generated name modules loaded (google finds *nothing* for them)".  Those also have no debug info key.
Missed that line. Given that, I don't believe this bug is going to be actionable in the short term, until somebody comments and leaves their email address. I imagine we'd ask them to do a virus scan.

Including Benjamin so that he's aware of this bug, which is an instance of DLL malware that we are unable to blocklist.
Flags: needinfo?(kairo)
It's still #5 in 18.0b1, but I also still see those randomly named DLLs and even a randomly named add-on in correlations.
Summary: crash in nsInputStreamPump::OnInputStreamReady → crash in nsInputStreamPump::OnInputStreamReady with randomly named DLL (malware)
More reports also:
https://crash-stats.mozilla.com/report/list?signature=nsInputStreamPump%3A%3AOnStateStart%28%29
Crash Signature: [@ nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*)] → [@ nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*)] [@ nsInputStreamPump::OnStateStart()]
Honza, is there anything we can do here? If not, I wonder if tracking this bug is worth it at all.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #9)
> Honza, is there anything we can do here? If not, I wonder if tracking this
> bug is worth it at all.

Juan performed outreach to affected users in preparation for working with AV vendors - I'm not sure he's heard back yet. We're just keeping this on our radar for that purpose. Sorry there's not been comment to that end in the bug.
Flags: needinfo?(jbecerra)
I haven't heard back from the people I wrote to. I'll give it another try with a few more users and see what we get.
Flags: needinfo?(jbecerra)
There's nothing actionable here for us for FF18. Leaving it on the tracking list to make sure we check back as we come up to release and after.
I may have a user on the support forums who would be willing to work with us on this. His crash is bp-ce27f149-7c9c-4a20-b5d7-733712130111. If you'd like I'll put him in contact with the correct person, or talk to him myself.
-> tdowner for now to get a copy of the malware DLL. tyler please hand this bug back to me when ready!
Assignee: honzab.moz → tdowner
We have a .dll at https://dl.dropbox.com/u/1037410/qhkwkyza.rar. Please take caution as it is malware. Passing back to bsmedberg.
Assignee: tdowner → benjamin
(In reply to Tyler Downer [:Tyler] from comment #15)
> We have a .dll at https://dl.dropbox.com/u/1037410/qhkwkyza.rar. Please take
> caution as it is malware. Passing back to bsmedberg.

https://www.virustotal.com/file/a453dd9a3b3ed497f288eccb74707b2bbe32c0dc3b1c777eafc739972ca8012f/analysis/1357943833/
I've tried placing this file in the installation directory and browsing for a bit. I've also looked for software that this file has been reported being part of, but I haven't been able to locate these viruses so I can install them. I'm not sure how to proceed with this.
Somebody brave could create a winxp image installation in a virtual box and try to reproduce with installing the dll (see [1]).  This may be a firefox bug, still.  Just spiked by the virus.

[1] http://www.microsoft.com/security/portal/threat/encyclopedia/entry.aspx?Name=Trojan%3AWin32%2FTracur.AU
I tried some of the suggestions at the article in comment 18, mainly:

Copy qhkwkyza.dll to
> %LOCALAPPDATA%\Local AppWizard-Generated Applications\qhkwkyza.dll
> %LOCALAPPDATA%\Microsoft\qhkwkyza.dll

...and add the following registry key
Hive: HKCU\Software\Microsoft\Windows\CurrentVersion\Run
Name: "Local AppWizard-Generated Applications"
Value: "rundll32.exe "C:\Users\mozilla\AppData\Local\Local AppWizard-Generated Applications\qhkwkyza.dll", CheckCTCRCVersion"

I've been trying various search engines including AOL, Google, Yahoo, and Bing but no crashes or search redirections yet. I'll keep trying and report back if I stumble upon anything.
PS, I'm also trying out the other methods mentioned in the article:
TX-Export and mpegInVideoAuxinfo
Add a run entry as:

In subkey: HKCU\Software\Microsoft\Windows\CurrentVersion\Run
Sets value: "<malware's sub-folder>"
With data: "rundll32.exe "%LOCALAPPDATA%\<malware's sub-folder>\<random>.dll",<export function>"


where <export function> = JbdDOnnNp and restart your machine.

be sure that %LOCALAPPDATA%\<malware's sub-folder>\<random>.dll points to the dll correctly.
(In reply to Honza Bambas (:mayhemer) from comment #21)
> where <export function> = JbdDOnnNp and restart your machine.

I set this and no joy yet.
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #22)
> (In reply to Honza Bambas (:mayhemer) from comment #21)
> > where <export function> = JbdDOnnNp and restart your machine.
> 
> I set this and no joy yet.

Check with e.g. ProcessExplorer whether the dll is loaded in Firefox (Ctrl-L to view DLLs loaded by the process).  Also try with admin-level account.
(In reply to Honza Bambas (:mayhemer) from comment #23)

> Check with e.g. ProcessExplorer whether the dll is loaded in Firefox (Ctrl-L
> to view DLLs loaded by the process).  Also try with admin-level account.

Heh, you beat me to the punch. I was just about to post this:

As an aside I tried using Process Explorer to see if the DLL was being loaded at all and it's not, as far as I can tell. I also ran a Kaspersky AV scan (mentioned as the reporter in the above article) and it couldn't find it. I actually saw it scan the offending DLL and it did not detect it as malware.
I tried to run the latest Kaspersky Security Scanner and MS Safety Scanner and it was always found, in my Downloads dir (in the rar) and either in RecycleBin after deletion.  So it is quit strange.
I've just submitted the affected DLL to all major a/v vendors - hopefully they'll resolve soon.
(In reply to Alex Keybl [:akeybl] from comment #26)
> I've just submitted the affected DLL to all major a/v vendors - hopefully
> they'll resolve soon.

According comment 16 I think they know already.
(In reply to Honza Bambas (:mayhemer) from comment #27)
> According comment 16 I think they know already.

Some subset does, but I submitted to a more complete list (see https://www.techsupportalert.com/content/how-report-malware-or-false-positives-multiple-antivirus-vendors.htm)
Sorry, I was incorrect, Kaspersky did find it. I saw references to it in the detailed report. Still no crashes, redirects, or DLL references in Process Explorer. I'm giving up for today. Sorry I couldn't be more helpful.
Here is another dll that is causing the crash on Windows XP, my crash report ID is: fcfe6f90-11a1-4d3a-bfe7-9fc9d2130113

https://crash-stats.mozilla.com/report/index/fcfe6f90-11a1-4d3a-bfe7-9fc9d2130113

Here is the dll:
http://dl.dropbox.com/u/71853669/xasrqgsx.zip

Be careful as this is malware!
https://www.virustotal.com/file/accd8054533f02b3151a8fedb8bc470c36ce6620364e5da6b51b958c485b3ae1/analysis/1358090720/

Put it in this folder:
C:\Documents and Settings\{USERNAME}\Local Settings\Application Data\MainConcept\

Add this reg entry:

String value name: "MainConcept"
Contents: "Rundll32.exe "C:\Documents and Settings\{USERNAME}\Local Settings\Application Data\MainConcept\xasrqgsx.dll",??0CIcdSpiAuto@@QAE@XZ"

Feel free to upload this file to other anti-virus companies as I am not a Windows user anymore ;)
(In reply to Andrew from comment #30)
> Here is another dll that is causing the crash on Windows XP

I haven't checked checksums, but this could in theory be the same DLL. As this bug report says, it comes with random names, and the file name you provide has the same type of random name that the other does. Interestingly, your virustotal check and the one from comment #16 report different detections, and even by different AV vendors. Fun.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #31)
> (In reply to Andrew from comment #30)
> > Here is another dll that is causing the crash on Windows XP
> 
> I haven't checked checksums, but this could in theory be the same DLL. As
> this bug report says, it comes with random names, and the file name you
> provide has the same type of random name that the other does. Interestingly,
> your virustotal check and the one from comment #16 report different
> detections, and even by different AV vendors. Fun.


Yup, I checked the virustotal scan first so I didn't waste anyone's time here. I also noticed it had a different entry point and installation folder. 

Interestingly the DLL had a date from early December but it seems like the latest update broke the malware.
(In reply to Honza Bambas (:mayhemer) from comment #18)
> This may be a firefox
> bug, still.  Just spiked by the virus.

Is there any way we can find out?
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #34)
> (In reply to Honza Bambas (:mayhemer) from comment #18)
> > This may be a firefox
> > bug, still.  Just spiked by the virus.
> 
> Is there any way we can find out?

My plan was to install the virus and try to reproduce the crash with searching for the cause in a traditional way.
(In reply to Andrew from comment #30)
> Here is another dll that is causing the crash on Windows XP
[...]
> Here is the dll:
> http://dl.dropbox.com/u/71853669/xasrqgsx.zip

I submitted this one to AV vendors as well, using the mechanism/steps pointed out in https://www.techsupportalert.com/content/how-report-malware-or-false-positives-multiple-antivirus-vendors.htm

The other one was submitted by Alex in comment #28.

It looks like the volume of this crash is dropping, hopefully because of AV detection uptake.
Unfortunately, I was just hit with this malware myself. The fact that my search results were redirected to shady websites gave it away and FF 18.0.1 started crashing on startup. With the following instructions, this bug should be reproducible in a reliable way:
1. Add a REG_SZ entry to HKCU\Software\Microsoft\Windows\CurrentVersion\Run as follows:
  Name: Microsoft
  Data: "rundll32.exe "<path-to-dll>\mpiwurlq.dll",DllUnregisterServer"
2. Place mpiwurlq.dll at <path-to-dll> chosen in 1 above.
3. Restart system.
4. FF 18.0.1 will crash on startup.

Also, this Microsoft threat encyclopedia entry describes exactly the behavior that I've observed:
http://www.microsoft.com/security/portal/threat/encyclopedia/Entry.aspx?Name=Trojan%3aWin32%2fTracur.AU

The malware (mpiwurlq.dll) has been emailed to bsmith.
Thanks spohl. I attached the malware to this bug.
(In reply to Stephen Pohl [:spohl] from comment #38)
> Virustotal scan for mpiwurlq.dll mentioned in comment 37:
> https://www.virustotal.com/file/
> d8f701e3e604700b2c2fbce22b000a9cf0cb3ed8ddb8876711e674157dad01c6/analysis/
> 1359337603/

Looks like this one is also not detected by some major AV vendors yet, should we also submit it like we did with the others (see comment #36 and comment #28)?

Also, Honza said the crash itself could very well be a crash on our side, could we debug that with the steps in comment #37? (Of course, given that spohl says this seems to redirect searches, it could be that it accesses nsIPrefBranch and the crash is again just because we didn't rev the UUID...)
I submitted the file attached in comment #39 to the AV vendors as well, using the steps and list from https://www.techsupportalert.com/content/how-report-malware-or-false-positives-multiple-antivirus-vendors.htm
It's #138 top browser crasher in 18.0.2.
Keywords: topcrash
Not much more we're going to do about this malware from an engineering perspective, now that we have samples and have handed them off to A/V vendors.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.