Closed
Bug 973138
Opened 10 years ago
Closed 10 years ago
startup crash in __fnNCDESTROY (mostly with MovieMode.48CA2AEFA22D.dll)
Categories
(Core :: General, defect)
Tracking
()
People
(Reporter: kairo, Assigned: away)
References
Details
(Keywords: crash, topcrash, verifyme)
Crash Data
Attachments
(2 files)
1.31 KB,
patch
|
benjamin
:
review-
|
Details | Diff | Splinter Review |
1.05 KB,
patch
|
benjamin
:
review+
Sylvestre
:
feedback+
Sylvestre
:
approval-mozilla-aurora+
Sylvestre
:
approval-mozilla-beta+
|
Details | Diff | Splinter Review |
This bug was filed from the Socorro interface and is report bp-c2a5b81b-4240-4049-bb6e-90aa72140215. ============================================================= https://crash-stats.mozilla.com/report/list?signature=__fnNCDESTROY is a crash that mostly appears with Firefox 27.0 and later, and correlations say that it's mostly with MovieMode.48CA2AEFA22D.dll, which sounds shady from its name. It's also interesting that it mostly seems to happen with Vista, and somewhat with XP and 8.1 but not with Win7, for example. The only pointer to that DLL I could find is in http://www.forumpcbrasil.com/t1611-resolvido-ajuda-para-remover-adwares-e-problemas-no-pc and I don't understand the pt-BR language so I have no idea what it really is talking about.
I can somewhat understand Portuguese via Spanish -- the forum user was directed to run a number of different malware scanners, and eventually reported the problem resolved, but it's unclear what the original symptom was (just "help removing adware") or which particular files were the key.
Some kind of badness happened during the system call for DestroyWindow: user32!__fnNCDESTROY ntdll!KiUserCallbackDispatcher user32!NtUserDestroyWindow xul!nsUXThemeData::UpdateTitlebarInfo xul!nsWindow::Create xul!nsWebShellWindow::Initialize xul!nsAppShellService::JustCreateTopWindow xul!nsAppShellService::CreateTopLevelWindow xul!nsAppStartup::CreateChromeWindow2 xul!nsWindowWatcher::OpenWindowInternal xul!nsWindowWatcher::OpenWindow xul!NS_InvokeByIndex xul!XPC_WN_CallMethod MovieMode is certainly the most popular but there are some other variants too (which makes it look even more suspicious). Several are pretty recently compiled. Image path: C:\Windows\System32\MovieMode.48CA2AEFA22D.dll Timestamp: Mon Feb 10 15:32:30 2014 (52F9618E) CheckSum: 0011D46E ImageSize: 0011E000 Image path: C:\Windows\System32\SafeMonitor.5D8B1F66A294.dll Timestamp: Mon Feb 10 16:19:20 2014 (52F96C88) CheckSum: 00119C44 ImageSize: 0011E000 Image path: C:\Windows\System32\WbShld.384B48E6DC83.dll Timestamp: Thu Dec 05 18:44:42 2013 (52A13A1A) CheckSum: 00120034 ImageSize: 0011D000 Image path: C:\WINDOWS\system32\SpyAlert.F82A323AD82D.dll Timestamp: Mon Feb 10 17:06:08 2014 (52F97780) CheckSum: 001291E4 ImageSize: 0011E000 Maybe something in that API is different post-Vista?
(And of course, no manufacturer or version number on those binaries)
Reporter | ||
Updated•10 years ago
|
Crash Signature: [@ __fnNCDESTROY] → [@ __fnNCDESTROY ]
This appears to be spiking a bit in Firefox 28 over the last few days with a 3-day rating of 3.5: https://crash-analysis.mozilla.com/rkaiser/2014-03-02/2014-03-02.firefox.28.explosiveness.html Crashes per 1MM ADU: ==================== 2014-03-02: 92.986 2014-03-01: 71.573 2014-02-28: 146.119 2014-02-27: 70.550 2014-02-26: 60.141 2014-02-25: 46.240 2014-02-24: 49.088 Crashes per Install: ==================== Firefox 27: 8114 crashes, 4928 installations (~1.65 CPI) Firefox 28: 1414 crashes, 279 installations (~5.07 CPI) According to Socorro this is rising significantly in the last week, up to #14 from #33, accounting for 0.71% of our crashes in Firefox 28 now: https://crash-stats.mozilla.com/topcrasher/products/Firefox/versions/28.0b/date_range_type/report/crash_type/browser/os_name/None/result_count/300?days=7
status-firefox28:
--- → affected
status-firefox29:
--- → affected
status-firefox30:
--- → affected
tracking-firefox28:
--- → ?
tracking-firefox29:
--- → ?
tracking-firefox30:
--- → ?
Keywords: topcrash
As per the Crashkill meeting today, Tracy is going to work with Kairo to see if the numbers we're seeing on Firefox 28 is an order of magnitude worse than what we saw when Firefox 27 was in Beta.
Flags: needinfo?(twalker)
To recap from the meeting: MovieMode.48CA2AEFA22D.dll is currently 86% of these crashes. We could stop a bunch of crashes by blocking that one specific filename, but it's unknown how long the relief would last, since the names might change in the future.
Updated•10 years ago
|
Comment 7•10 years ago
|
||
This has exploded in release (Fx27*). So it is not a regression on Fx28. No need to compare beta to beta stability regarding this malware.
Flags: needinfo?(twalker)
Comment 8•10 years ago
|
||
We still don't have anything actionable yet here, and no reason to suspect there will be higher volume in 28 than 27 so wontfixing and we'll watch what happens.
Updated•10 years ago
|
Product: Firefox → Core
Comment 9•10 years ago
|
||
Is there any further steps we could do to mitigate this bug in 29?
Assignee | ||
Comment 10•10 years ago
|
||
As suspected, the filenames are changing. Here is a new one: Image name: Websteroids.B324755F3F87.dll Timestamp: Sat Mar 22 12:01:19 2014 (532CC4BF) We could get some short-term relief by blocking MovieMode+Webstreoids (together they are currently 92%) but it won't last for more than a month or two. Maybe more long term we could block twelve-digit hex strings, but that feels pretty hacky. (It's interesting that this is so high on crash-stats even though the crash is Vista only. Filtering down to just Vista users, it's even more dramatic: this is 28% of the crashes on that OS)
Comment 11•10 years ago
|
||
(In reply to David Major [:dmajor] (UTC+12) from comment #10) > We could get some short-term relief by blocking MovieMode+Webstreoids > (together they are currently 92%) but it won't last for more than a month or > two. As a temporary workaround until we have a proper fix, that could be a solution.
Reporter | ||
Comment 12•10 years ago
|
||
(In reply to Sylvestre Ledru [:sylvestre] from comment #11) > As a temporary workaround until we have a proper fix, that could be a > solution. Note that there is no proper fix other than getting this malware/grayware deleted from users' computers.
Assignee | ||
Comment 13•10 years ago
|
||
Here's the patch if we want it. Feel free to r- if not! Also added ZombieAlert which seems to be the next one coming up. I checked that reports of this signature from recent 29b builds (after the blocklist reordering) have a low rate of User32BeforeBlocklist=1, so this ought to work.
Attachment #8403036 -
Flags: review?(benjamin)
Comment 14•10 years ago
|
||
Comment on attachment 8403036 [details] [diff] [review] Block moviemode+websteroids+zombiealert I don't think we should blocklist the specific versions. Kairo was going to talk to relman at the channel meeting today to decide whether the long-form fix of adding some kind of wildcard support to the blocklist was necessary. Otherwise we should add a support classifier for this and add it to the list of things to warn people about in "support-after-crash".
Attachment #8403036 -
Flags: review?(benjamin) → review-
Comment 15•10 years ago
|
||
We (Lukas, Kairo and I) discussed on this during the channel meeting and we think it would be a great feature to have. I guess this won't make it for 29 but it is welcome for 30 (or 31).
status-firefox31:
--- → affected
tracking-firefox31:
--- → +
Assignee | ||
Comment 16•10 years ago
|
||
Please clarify, what type of wildcard support is being requested? The one-off fix is to add a few lines of pattern-matching for this specific bug. I don't love this type of fix but I'd be okay with it. For a properly general "feature", we'd basically need a full regex engine. That would add a considerable amount of complexity (synonyms: bugs, maintenance, slowness) which I don't think is currently justified (so far this is the only bug of this nature that I've seen).
Reporter | ||
Comment 17•10 years ago
|
||
Sylvestre, can you get the questions in comment #16 cleared up (might need some coordination on what you in relman wants and what bsmedberg deems useful)?
Flags: needinfo?(sledru)
Comment 18•10 years ago
|
||
OK. I think that needs some discussions. For example, just a simple wildcard support could fix most of the issues we have. I am not sure we have to go to full regexp support here... Anyway, until we have this support (which is unlikely for 29), the attachment 8403036 [details] [diff] [review] could be a potential temporary workaround. What do you think Benjamin?
Flags: needinfo?(sledru) → needinfo?(benjamin)
Comment 19•10 years ago
|
||
As I said earlier, I don't think attachment 8403036 [details] [diff] [review] is an acceptable workaround. Given the historical release cadence of this software, it probably won't last for more than a few weeks into the 29 release anyway.
Flags: needinfo?(benjamin)
Assignee | ||
Comment 20•10 years ago
|
||
(In reply to Sylvestre Ledru [:sylvestre] from comment #18) > For example, just a simple wildcard support could fix most of the issues we > have. I am not sure we have to go to full regexp support here... If we used primitive wildcards like DOS-style then it would be a pattern like: *.????????????.dll That seems too broad to me. I would like to be able to search for 12 hex digits. That needs a more complex pattern matcher.
Assignee | ||
Comment 21•10 years ago
|
||
This is the one-off pattern match that I had in mind. Maybe it can be a decent compromise for now. (It's a rough draft, needs to be better integrated into surrounding code, etc.) If later we have reason to do a real pattern matcher then this could be rolled into it.
Attachment #8406503 -
Flags: feedback?(sledru)
Attachment #8406503 -
Flags: feedback?(benjamin)
Comment 22•10 years ago
|
||
Comment on attachment 8406503 [details] [diff] [review] One-off pattern match This will block anything which is {hex16}.* right? Don't we want to limit this to the three DLL names in question here?
Attachment #8406503 -
Flags: feedback?(benjamin) → feedback-
Assignee | ||
Comment 23•10 years ago
|
||
(In reply to Benjamin Smedberg [:bsmedberg] from comment #22) > This will block anything which is {hex16}.* right? Don't we want to limit > this to the three DLL names in question here? The name also changes when the hex value changes.
Reporter | ||
Comment 24•10 years ago
|
||
I also think that valid software very probably won't use *.{random-hex-string}.dll so I don't see much harm in blocking that whole pattern.
Comment 25•10 years ago
|
||
Comment on attachment 8406503 [details] [diff] [review] One-off pattern match Thanks for the patch. Just like Kairo. This is better than the current situation.
Attachment #8406503 -
Flags: feedback?(sledru) → feedback+
Comment 26•10 years ago
|
||
I'm very concerned about blocking unexpected DLLs. I filed bug 997319 to get a report of modules that might be affected by this, but I think our default position should be to not block this. Especially because we're apparently facing a determined adversary and all they'd have to do is change their pattern by a little bit to defeat the blocklist.
Comment 27•10 years ago
|
||
I agree with Benjamin and think we should wait and see what would be affected (and can we test this over a beta cycle to confirm we don't accidentally block anything not-malware?) so marking this wontfix for 29 and we can continue tracking potential solutions here for FF30
Comment 28•10 years ago
|
||
Comment on attachment 8406503 [details] [diff] [review] One-off pattern match ok, let's do this. I'm still a little worried that we'll catch future non-bad DLLs, but the evidence from the other bug shows that there's no current false-positives.
Attachment #8406503 -
Flags: feedback- → review+
Assignee | ||
Comment 29•10 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/1fe69cad9713
Comment 30•10 years ago
|
||
https://hg.mozilla.org/mozilla-central/rev/1fe69cad9713
Assignee: nobody → dmajor
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla32
Assignee | ||
Comment 31•10 years ago
|
||
Comment on attachment 8406503 [details] [diff] [review] One-off pattern match [Approval Request Comment] Bug caused by (feature/regressing bug #): External software User impact if declined: Startup topcrash (1/3 of all Vista crashes on 30b) Testing completed (on m-c, etc.): It's been on nightly for a few days with no crashes, but this signature is rare on the nightly channel. No issues on mozillazine. Risk to taking this patch (and alternatives if risky): There is a chance that the pattern matching could block legitimate DLLs, but bug 997319 found no such modules in the wild currently. String or IDL/UUID changes made by this patch: None
Attachment #8406503 -
Flags: approval-mozilla-beta?
Attachment #8406503 -
Flags: approval-mozilla-aurora?
Updated•10 years ago
|
Attachment #8406503 -
Flags: approval-mozilla-beta?
Attachment #8406503 -
Flags: approval-mozilla-beta+
Attachment #8406503 -
Flags: approval-mozilla-aurora?
Attachment #8406503 -
Flags: approval-mozilla-aurora+
Comment 32•10 years ago
|
||
This in the top #5 with current Fx29 data. There are several malware-sounding dlls in the correlations tab, in addition to the moviemode one, but all those have a 0% next to them, while most others seem legitimate. Given the rank, I'm not sure how to proceed, but I will reopen this bug so we can re-evaluate.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 33•10 years ago
|
||
Why are you re-opening it? It didn't land for FF29, so it still being present in 29 is expected.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Comment 34•10 years ago
|
||
https://hg.mozilla.org/releases/mozilla-aurora/rev/90c4d8602b35 https://hg.mozilla.org/releases/mozilla-beta/rev/743afab610fa
Updated•10 years ago
|
status-b2g-v1.4:
--- → fixed
Comment 37•10 years ago
|
||
I landed trivial mingw fixup: https://hg.mozilla.org/integration/mozilla-inbound/rev/fa9b99291387
Comment 39•10 years ago
|
||
For the last 7 days: No crashes on 32, and around 15 crashes on 31 but they are all dupes from one user. There are somewhere on the order of 10 people on 30 still crashing, with a lot of duplicate crash reports, who seem stuck on the 20140428174145 build, probably unable to update.
Status: RESOLVED → VERIFIED
You need to log in
before you can comment on or make changes to this bug.
Description
•