Closed Bug 973138 Opened 10 years ago Closed 10 years ago

startup crash in __fnNCDESTROY (mostly with MovieMode.48CA2AEFA22D.dll)

Categories

(Core :: General, defect)

x86
Windows Vista
defect
Not set
critical

Tracking

()

VERIFIED FIXED
mozilla32
Tracking Status
firefox28 + wontfix
firefox29 + wontfix
firefox30 + verified
firefox31 + verified
firefox32 --- verified
b2g-v1.4 --- fixed

People

(Reporter: kairo, Assigned: away)

References

Details

(Keywords: crash, topcrash, verifyme)

Crash Data

Attachments

(2 files)

This bug was filed from the Socorro interface and is 
report bp-c2a5b81b-4240-4049-bb6e-90aa72140215.
=============================================================

https://crash-stats.mozilla.com/report/list?signature=__fnNCDESTROY is a crash that mostly appears with Firefox 27.0 and later, and correlations say that it's mostly with MovieMode.48CA2AEFA22D.dll, which sounds shady from its name.

It's also interesting that it mostly seems to happen with Vista, and somewhat with XP and 8.1 but not with Win7, for example.


The only pointer to that DLL I could find is in http://www.forumpcbrasil.com/t1611-resolvido-ajuda-para-remover-adwares-e-problemas-no-pc and I don't understand the pt-BR language so I have no idea what it really is talking about.
I can somewhat understand Portuguese via Spanish -- the forum user was directed to run a number of different malware scanners, and eventually reported the problem resolved, but it's unclear what the original symptom was (just "help removing adware") or which particular files were the key.
Some kind of badness happened during the system call for DestroyWindow:

user32!__fnNCDESTROY
ntdll!KiUserCallbackDispatcher
user32!NtUserDestroyWindow
xul!nsUXThemeData::UpdateTitlebarInfo
xul!nsWindow::Create
xul!nsWebShellWindow::Initialize
xul!nsAppShellService::JustCreateTopWindow
xul!nsAppShellService::CreateTopLevelWindow
xul!nsAppStartup::CreateChromeWindow2
xul!nsWindowWatcher::OpenWindowInternal
xul!nsWindowWatcher::OpenWindow
xul!NS_InvokeByIndex
xul!XPC_WN_CallMethod

MovieMode is certainly the most popular but there are some other variants too (which makes it look even more suspicious). Several are pretty recently compiled.

    Image path: C:\Windows\System32\MovieMode.48CA2AEFA22D.dll
    Timestamp:        Mon Feb 10 15:32:30 2014 (52F9618E)
    CheckSum:         0011D46E
    ImageSize:        0011E000

    Image path: C:\Windows\System32\SafeMonitor.5D8B1F66A294.dll
    Timestamp:        Mon Feb 10 16:19:20 2014 (52F96C88)
    CheckSum:         00119C44
    ImageSize:        0011E000

    Image path: C:\Windows\System32\WbShld.384B48E6DC83.dll
    Timestamp:        Thu Dec 05 18:44:42 2013 (52A13A1A)
    CheckSum:         00120034
    ImageSize:        0011D000

    Image path: C:\WINDOWS\system32\SpyAlert.F82A323AD82D.dll
    Timestamp:        Mon Feb 10 17:06:08 2014 (52F97780)
    CheckSum:         001291E4
    ImageSize:        0011E000

Maybe something in that API is different post-Vista?
(And of course, no manufacturer or version number on those binaries)
Crash Signature: [@ __fnNCDESTROY] → [@ __fnNCDESTROY ]
This appears to be spiking a bit in Firefox 28 over the last few days with a 3-day rating of 3.5:
https://crash-analysis.mozilla.com/rkaiser/2014-03-02/2014-03-02.firefox.28.explosiveness.html

Crashes per 1MM ADU:
====================
2014-03-02: 92.986
2014-03-01: 71.573
2014-02-28: 146.119
2014-02-27: 70.550
2014-02-26: 60.141
2014-02-25: 46.240
2014-02-24: 49.088

Crashes per Install:
====================
Firefox 27: 8114 crashes, 4928 installations (~1.65 CPI)
Firefox 28: 1414 crashes, 279 installations (~5.07 CPI)

According to Socorro this is rising significantly in the last week, up to #14 from #33, accounting for 0.71% of our crashes in Firefox 28 now:
https://crash-stats.mozilla.com/topcrasher/products/Firefox/versions/28.0b/date_range_type/report/crash_type/browser/os_name/None/result_count/300?days=7
As per the Crashkill meeting today, Tracy is going to work with Kairo to see if the numbers we're seeing on Firefox 28 is an order of magnitude worse than what we saw when Firefox 27 was in Beta.
Flags: needinfo?(twalker)
To recap from the meeting: MovieMode.48CA2AEFA22D.dll is currently 86% of these crashes. We could stop a bunch of crashes by blocking that one specific filename, but it's unknown how long the relief would last, since the names might change in the future.
This has exploded in release (Fx27*). So it is not a regression on Fx28. No need to compare beta to beta stability regarding this malware.
Flags: needinfo?(twalker)
We still don't have anything actionable yet here, and no reason to suspect there will be higher volume in 28 than 27 so wontfixing and we'll watch what happens.
Product: Firefox → Core
Is there any further steps we could do to mitigate this bug in 29?
As suspected, the filenames are changing. Here is a new one:
    Image name: Websteroids.B324755F3F87.dll
    Timestamp:        Sat Mar 22 12:01:19 2014 (532CC4BF)

We could get some short-term relief by blocking MovieMode+Webstreoids (together they are currently 92%) but it won't last for more than a month or two.

Maybe more long term we could block twelve-digit hex strings, but that feels pretty hacky.

(It's interesting that this is so high on crash-stats even though the crash is Vista only. Filtering down to just Vista users, it's even more dramatic: this is 28% of the crashes on that OS)
(In reply to David Major [:dmajor] (UTC+12) from comment #10)
> We could get some short-term relief by blocking MovieMode+Webstreoids
> (together they are currently 92%) but it won't last for more than a month or
> two.
As a temporary workaround until we have a proper fix, that could be a solution.
(In reply to Sylvestre Ledru [:sylvestre] from comment #11)
> As a temporary workaround until we have a proper fix, that could be a
> solution.

Note that there is no proper fix other than getting this malware/grayware deleted from users' computers.
Here's the patch if we want it. Feel free to r- if not! Also added ZombieAlert which seems to be the next one coming up.

I checked that reports of this signature from recent 29b builds (after the blocklist reordering) have a low rate of User32BeforeBlocklist=1, so this ought to work.
Attachment #8403036 - Flags: review?(benjamin)
Comment on attachment 8403036 [details] [diff] [review]
Block moviemode+websteroids+zombiealert

I don't think we should blocklist the specific versions. Kairo was going to talk to relman at the channel meeting today to decide whether the long-form fix of adding some kind of wildcard support to the blocklist was necessary. Otherwise we should add a support classifier for this and add it to the list of things to warn people about in "support-after-crash".
Attachment #8403036 - Flags: review?(benjamin) → review-
We (Lukas, Kairo and I) discussed on this during the channel meeting and we think it would be a great feature to have. I guess this won't make it for 29 but it is welcome for 30 (or 31).
Please clarify, what type of wildcard support is being requested?

The one-off fix is to add a few lines of pattern-matching for this specific bug. I don't love this type of fix but I'd be okay with it.

For a properly general "feature", we'd basically need a full regex engine. That would add a considerable amount of complexity (synonyms: bugs, maintenance, slowness) which I don't think is currently justified (so far this is the only bug of this nature that I've seen).
Sylvestre, can you get the questions in comment #16 cleared up (might need some coordination on what you in relman wants and what bsmedberg deems useful)?
Flags: needinfo?(sledru)
OK. I think that needs some discussions.
For example, just a simple wildcard support could fix most of the issues we have. I am not sure we have to go to full regexp support here...

Anyway, until we have this support (which is unlikely for 29), the attachment 8403036 [details] [diff] [review] could be a potential temporary workaround. What do you think Benjamin?
Flags: needinfo?(sledru) → needinfo?(benjamin)
As I said earlier, I don't think attachment 8403036 [details] [diff] [review] is an acceptable workaround. Given the historical release cadence of this software, it probably won't last for more than a few weeks into the 29 release anyway.
Flags: needinfo?(benjamin)
(In reply to Sylvestre Ledru [:sylvestre] from comment #18)
> For example, just a simple wildcard support could fix most of the issues we
> have. I am not sure we have to go to full regexp support here...

If we used primitive wildcards like DOS-style then it would be a pattern like:
*.????????????.dll
That seems too broad to me. I would like to be able to search for 12 hex digits. That needs a more complex pattern matcher.
This is the one-off pattern match that I had in mind. Maybe it can be a decent compromise for now. (It's a rough draft, needs to be better integrated into surrounding code, etc.)

If later we have reason to do a real pattern matcher then this could be rolled into it.
Attachment #8406503 - Flags: feedback?(sledru)
Attachment #8406503 - Flags: feedback?(benjamin)
Comment on attachment 8406503 [details] [diff] [review]
One-off pattern match

This will block anything which is {hex16}.* right? Don't we want to limit this to the three DLL names in question here?
Attachment #8406503 - Flags: feedback?(benjamin) → feedback-
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #22)
> This will block anything which is {hex16}.* right? Don't we want to limit
> this to the three DLL names in question here?

The name also changes when the hex value changes.
I also think that valid software very probably won't use *.{random-hex-string}.dll so I don't see much harm in blocking that whole pattern.
Comment on attachment 8406503 [details] [diff] [review]
One-off pattern match

Thanks for the patch. Just like Kairo.

This is better than the current situation.
Attachment #8406503 - Flags: feedback?(sledru) → feedback+
Depends on: 997319
I'm very concerned about blocking unexpected DLLs. I filed bug 997319 to get a report of modules that might be affected by this, but I think our default position should be to not block this. Especially because we're apparently facing a determined adversary and all they'd have to do is change their pattern by a little bit to defeat the blocklist.
I agree with Benjamin and think we should wait and see what would be affected (and can we test this over a beta cycle to confirm we don't accidentally block anything not-malware?) so marking this wontfix for 29 and we can continue tracking potential solutions here for FF30
Comment on attachment 8406503 [details] [diff] [review]
One-off pattern match

ok, let's do this. I'm still a little worried that we'll catch future non-bad DLLs, but the evidence from the other bug shows that there's no current false-positives.
Attachment #8406503 - Flags: feedback- → review+
https://hg.mozilla.org/mozilla-central/rev/1fe69cad9713
Assignee: nobody → dmajor
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla32
Comment on attachment 8406503 [details] [diff] [review]
One-off pattern match

[Approval Request Comment]
Bug caused by (feature/regressing bug #): External software
User impact if declined: Startup topcrash (1/3 of all Vista crashes on 30b)
Testing completed (on m-c, etc.): It's been on nightly for a few days with no crashes, but this signature is rare on the nightly channel. No issues on mozillazine.
Risk to taking this patch (and alternatives if risky): There is a chance that the pattern matching could block legitimate DLLs, but bug 997319 found no such modules in the wild currently.
String or IDL/UUID changes made by this patch: None
Attachment #8406503 - Flags: approval-mozilla-beta?
Attachment #8406503 - Flags: approval-mozilla-aurora?
Attachment #8406503 - Flags: approval-mozilla-beta?
Attachment #8406503 - Flags: approval-mozilla-beta+
Attachment #8406503 - Flags: approval-mozilla-aurora?
Attachment #8406503 - Flags: approval-mozilla-aurora+
This in the top #5 with current Fx29 data. There are several malware-sounding dlls in the correlations tab, in addition to the moviemode one, but all those have a 0% next to them, while most others seem legitimate. Given the rank, I'm not sure how to proceed, but I will reopen this bug so we can re-evaluate.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Why are you re-opening it? It didn't land for FF29, so it still being present in 29 is expected.
Status: REOPENED → RESOLVED
Closed: 10 years ago10 years ago
Resolution: --- → FIXED
Keywords: verifyme
For the last 7 days: 
No crashes on 32, and around 15 crashes on 31 but they are all dupes from one user. 

There are somewhere on the order of 10 people on 30 still crashing, with a lot of duplicate crash reports, who seem stuck on the 20140428174145 build, probably unable to update.
You need to log in before you can comment on or make changes to this bug.