|Submitter||Diff||Changes||Open Issues||Last Updated|
|Error loading review requests:|
This bug was filed from the Socorro interface and is report bp-80ca6476-c188-4a39-90a8-5a6df2161021. ============================================================= Seen while looking at Nightly crash stats: http://bit.ly/2enm9HX. Crash appears to be associated with one of the anti virus products from http://global.ahnlab.com/site/main.do. There are 30 crashes showing up, but the summary shows only 9 installations.
https://crash-stats.mozilla.com/search/?signature=%3Dnzbrcom.dll%400x20a76&accessibility=__true__&product=Firefox&date=%3E%3D2016-10-14T17%3A47%3A00.000Z&date=%3C2016-10-21T17%3A47%3A00.000Z&_sort=-date&_facets=signature&_facets=version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-version - accessibility is active 100% - limited to nightly 52 - 3-day this is #27 currently
Some of these stacks contain RPC code that then jumps into this DLL. I am beginning to suspect that they're hooking RPC for some reason and a11y+e10s exercises the affected code. :-(
No crashes have been reported by builds newer than Oct 27. I'll continue monitoring this for a while but if things continue to look okay I'm going to RESOLVE WORKSFORME.
(In reply to Aaron Klotz [:aklotz] from comment #3) > No crashes have been reported by builds newer than Oct 27. I'll continue > monitoring this for a while but if things continue to look okay I'm going to > RESOLVE WORKSFORME. That's because the signature changed! I'll try a DLL blocklist patch once I have a better understanding of DLL version correlations.
I think this patent tells us what they're doing: https://www.google.ca/patents/US9185131 They're definitely hooking into RPC :-(
Comment on attachment 8807290 [details] Bug 1311969: Add nzbrcom.dll to blocklist; https://reviewboard.mozilla.org/r/90506/#review90848
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/autoland/rev/cd40a468ed9c Add nzbrcom.dll to blocklist; r=bsmedberg
a year ago
I still see crashes with this signature in builds as late as Nightly 20161119030204. Also, I see a few crashes with nzbrcom32.dll in the signature; that DLL was not added to the blocklist. bsmedberg, any suggestions how to proceed here? Adding nzbrcom32.dll to the blocklist would be easy, but it appears the blocklist is not having an effect anyway...
The nzbrcom32.dll ones have spiked in the past 3 days and now outnumber the nzbrcom.dll ones.
Grrr... probably the first step is to determine the injection mechanism by debugging an installation of this product running against Firefox with a11y and e10s enabled. Probably in a VM. :-)
njn, what aklotz said.
Dees made contact with an AnhLab engineer via private email. The engineer asked the following questions. > 1. What are the repro steps? We don't know. All the data we have comes from crash reports submitted by Firefox users. Here is a search of all the submitted crash reports for the last 6 months where the crash signature contains the substring "nzbrcom": https://crash-stats.mozilla.com/search/?signature=~nzbrcom&date=%3E%3D2016-12-08T05%3A41%3A00.000Z&date=%3C2016-12-15T05%3A41%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature Current results: > 1 nzbrcom32.dll@0x1c25b 181 73.58% > 2 nzbrcom.dll@0x20846 30 12.20% > 3 nzbrcom32.dll@0x1c51b 30 12.20% > 4 hang | NtUserMessageCall | NtUserMessageCall | nzbrcom32.dll@0x212cf 1 0.41 % > 5 nzbrcom.dll@0x20d96 1 0.41% > 6 nzbrcom.dll@0x2d6c6 1 0.41% > 7 nzbrcom.dll@0x46a0 1 0.41% > 8 nzbrcom32.dll@0x1b92b 1 0.41% The crash reports are grouped by "signature", which is based on the stack trace for the crash. You can see almost 3/4 of the crashes have a top stack frame of "nzbrcom32.dll@0x1c25b". If we focus on just those crashes: https://crash-stats.mozilla.com/signature/?_sort=-date&signature=nzbrcom32.dll%400x1c25b&date=%3E%3D2016-12-08T05%3A41%3A00.000Z&date=%3C2016-12-15T05%3A41%3A00.000Z You can see the individual crash reports by clicking on the "Reports" tab. > 2. What is the AhnLab product related to the issue? V3 Internet Security 8.0? or 9.0? or AhnLab Safe Transaction? I don't know. I looked at the "modules" tab for a few of the individual crash reports and they all had the following module info about nzbrcom32.cll: - module name: nzbrcom32.dll - version: 22.214.171.1240 - debug identifier: BA1C6709FE2F4FA69B7617EB1E9594AF5 - debug filename: NzBrcom32.pdb Here is this info for an example crash report: https://crash-stats.mozilla.com/report/index/add26a83-62b8-4453-b063-db1f42161209#tab-modules We also have the following correlations for this crash signature: > (100.0% in signature vs 00.46% overall) Module "nzbrcom32.dll" = true > (100.0% in signature vs 00.46% overall) Module "IAccessible2Proxy32.dll" = true > (100.0% in signature vs 00.55% overall) Module "klib.dll" = true > (100.0% in signature vs 00.56% overall) Module "imkrapi.dll" = true > (100.0% in signature vs 00.56% overall) Module "imkrtip.dll" = true > (100.0% in signature vs 00.58% overall) Module "mfc90u.dll" = true > (100.0% in signature vs 00.91% overall) Module "IMETIP.DLL" = true > (100.0% in signature vs 01.52% overall) Module "msvcp90.dll" = true > (100.0% in signature vs 02.60% overall) Module "msvcr90.dll" = true > (99.26% in signature vs 00.84% overall) Module "IMJKAPI.DLL" = true > (100.0% in signature vs 05.40% overall) Module "dui70.dll" = true > (100.0% in signature vs 05.52% overall) Module "imagehlp.dll" = true It's possible that this information might help identify which product is being used. > 3. What is the exact version of Firefox which has the problem? I can find > 52.0a2 on developer edition web site but can’t find 52.0a1. All the crashes are happening in 52.0a2, which is the current Developer Edition version. (52.0a1 was the version number used when that version of Firefox was in active development.) > 4. I need a technical analysis report. There is no technical analysis of the > crash. There is no direct evidence concerned with AhnLab module and no call > stacks. So I need a reasonable analysis description why you think the problem > is caused by AhnLab module, especially, which module in AhnLab product and > what action affected your module. Every individual crash report has a variety of information, including the stack trace for the crashing thread. https://crash-stats.mozilla.com/report/index/add26a83-62b8-4453-b063-db1f42161209#tab-details has the following stack trace: > 0 nzbrcom32.dll nzbrcom32.dll@0x1c25b > 1 oleacc.dll AccWrap_Base::get_accChildCount(long*) > 2 nzbrcom32.dll nzbrcom32.dll@0x17807 > 3 ntdll.dll ntdll.dll@0x4e4f4 > 4 rpcrt4.dll rpcrt4.dll@0x1b51f > 5 nzbrcom32.dll nzbrcom32.dll@0x6d13f > 6 nzbrcom32.dll nzbrcom32.dll@0x6d809 > 7 ntdll.dll ntdll.dll@0x6ff4f > 8 nzbrcom32.dll nzbrcom32.dll@0x6ddf7 > 9 ntdll.dll ntdll.dll@0x6ff4f > 10 xul.dll _SEH_epilog4 > 11 firefox.exe _SEH_epilog4 > 12 ntdll.dll ntdll.dll@0x75eef > 13 ntdll.dll ntdll.dll@0x8261e > 14 firefox.exe __scrt_common_main_seh f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:282 Not all the crash reports have exactly this stack trace, but they're all similar. The crash is clearly happening within nzbrcom32.dll.
Adding another signature which only seems to show up in Aurora 52.0a2 starting with 20161201004019.
As an aside, this should probably be moved out of Firefox:Untriaged at this point. Perhaps accessibility (per comment 1) or External Software makes more sense.
Hi I'm developer of AhnLab Safe Transaction Development Team I tried to reproduce it. I can not reproduce well, but I found several reproduction cases. This issue is not a problem until Firefox 51, but it has been reproduced since Firefox 52. Crash occurs when using VARIANTs obtained through the AccessibleChildren function. The problem occurs when the vt type of the VARIANT is VT_DISPATCH and the value of pdispVal is NULL. We will be distributing AhnLab SafeTransaction by adding a defense code for this problem. Please check the reason for this problem in Firefox 52.
(In reply to deguls from comment #18) > Please check the reason for this problem in Firefox 52. I fixed bug 1319640 which landed on 52 for the December 15 build. Are you seeing this on the most recent build?
It has not been reproduced in Firefox 52 2016-12-15 build. It appears to be resolved by bug 1319640.
Hello Aaron Klotz Because the cause of the problem is known I am asking you to remove nzbrcom.dll that you have added to WindowsDllBlocklist.cpp If you add nzbrcom.dll, Korean Firefox users will not be able to use Internet banking.
Hi Deguls, The method you're using to support your product in Firefox is not supported. The supported mechanism for communicating with host processes/libraries is to use Native Messaging with a WebExtension (https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Native_messaging), vs. any kind of hook or injection of a binary into the Firefox process. I'll send an intro email, but DLL injection/loading in the Firefox process is something we're going to continue to move away from. (In reply to deguls from comment #21) > Hello Aaron Klotz > Because the cause of the problem is known > I am asking you to remove nzbrcom.dll that you have added to > WindowsDllBlocklist.cpp > If you add nzbrcom.dll, Korean Firefox users will not be able to use > Internet banking.
hi. I'm a product planner of AhnLab Safe Transaction. Ahnlab Safe Transaction is an online bank security software that contains antivirus, personal firewall, anti-keylogger, and other functionality. In korea, it is a mandatory security software in order to use any kind of online banking services. The purpose of using dll injection is to check product running on web browser and detect WinAPI hooking viruses(zeus etc). We are aware of that it is against your policy. However dll injection is the only way to meet online banking compliance in Korea(for zeus detection, etc). The crash issue was resolved(bug 1319640). it is all taken care of and won’t happen anymore. So I am requesting you to remove nzbrcom.dll from your blocklist.