Crash in nzbrcom.dll@0x20a76

RESOLVED WORKSFORME

Status

()

Firefox
Untriaged
--
critical
RESOLVED WORKSFORME
a year ago
a year ago

People

(Reporter: marcia, Assigned: aklotz)

Tracking

({crash})

Trunk
Firefox 52
Unspecified
Windows 10
crash
Points:
---

Firefox Tracking Flags

(firefox52 fixed)

Details

(Whiteboard: aes+, crash signature)

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

58 bytes, text/x-review-board-request
Benjamin Smedberg
: review+
Details | Review
(Reporter)

Description

a year ago
This bug was filed from the Socorro interface and is 
report bp-80ca6476-c188-4a39-90a8-5a6df2161021.
=============================================================

Seen while looking at Nightly crash stats: http://bit.ly/2enm9HX. Crash appears to be associated with one of the anti virus products from http://global.ahnlab.com/site/main.do.

There are 30 crashes showing up, but the summary shows only 9 installations.
https://crash-stats.mozilla.com/search/?signature=%3Dnzbrcom.dll%400x20a76&accessibility=__true__&product=Firefox&date=%3E%3D2016-10-14T17%3A47%3A00.000Z&date=%3C2016-10-21T17%3A47%3A00.000Z&_sort=-date&_facets=signature&_facets=version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-version

- accessibility is active 100%
- limited to nightly 52
- 3-day this is #27 currently
Whiteboard: aes+

Updated

a year ago
Component: General → Untriaged
(Assignee)

Comment 2

a year ago
Some of these stacks contain RPC code that then jumps into this DLL. I am beginning to suspect that they're hooking RPC for some reason and a11y+e10s exercises the affected code. :-(
(Assignee)

Comment 3

a year ago
No crashes have been reported by builds newer than Oct 27. I'll continue monitoring this for a while but if things continue to look okay I'm going to RESOLVE WORKSFORME.
(Reporter)

Updated

a year ago
Crash Signature: [@ nzbrcom.dll@0x20a76] → [@ nzbrcom.dll@0x20a76] [@ nzbrcom.dll@0x1a6b1]
(Assignee)

Updated

a year ago
Crash Signature: [@ nzbrcom.dll@0x20a76] [@ nzbrcom.dll@0x1a6b1] → [@ nzbrcom.dll@0x20a76] [@ nzbrcom.dll@0x1a6b1] [@ nzbrcom.dll@0x20846]
(Assignee)

Updated

a year ago
Duplicate of this bug: 1313963
(Assignee)

Updated

a year ago
Depends on: 1314693
(Assignee)

Comment 5

a year ago
(In reply to Aaron Klotz [:aklotz] from comment #3)
> No crashes have been reported by builds newer than Oct 27. I'll continue
> monitoring this for a while but if things continue to look okay I'm going to
> RESOLVE WORKSFORME.

That's because the signature changed! I'll try a DLL blocklist patch once I have a better understanding of DLL version correlations.
(Assignee)

Comment 6

a year ago
I think this patent tells us what they're doing:
https://www.google.ca/patents/US9185131

They're definitely hooking into RPC :-(
Comment hidden (mozreview-request)
Attachment #8807290 - Flags: review?(dmajor) → review?(benjamin)
(Reporter)

Updated

a year ago
Crash Signature: [@ nzbrcom.dll@0x20a76] [@ nzbrcom.dll@0x1a6b1] [@ nzbrcom.dll@0x20846] → [@ nzbrcom.dll@0x20a76] [@ nzbrcom.dll@0x1a6b1] [@ nzbrcom.dll@0x20846] [@ nzbrcom.dll@0x1a701]

Comment 8

a year ago
mozreview-review
Comment on attachment 8807290 [details]
Bug 1311969: Add nzbrcom.dll to blocklist;

https://reviewboard.mozilla.org/r/90506/#review90848
Attachment #8807290 - Flags: review?(benjamin) → review+

Comment 9

a year ago
Pushed by aklotz@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cd40a468ed9c
Add nzbrcom.dll to blocklist; r=bsmedberg

Comment 10

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/cd40a468ed9c
Status: NEW → RESOLVED
Last Resolved: a year ago
status-firefox52: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 52
Assignee: nobody → aklotz
I still see crashes with this signature in builds as late as Nightly 20161119030204.

Also, I see a few crashes with nzbrcom32.dll in the signature; that DLL was not added to the blocklist.

bsmedberg, any suggestions how to proceed here? Adding nzbrcom32.dll to the blocklist would be easy, but it appears the blocklist is not having an effect anyway...
Flags: needinfo?(benjamin)
The nzbrcom32.dll ones have spiked in the past 3 days and now outnumber the nzbrcom.dll ones.
Status: RESOLVED → REOPENED
Crash Signature: [@ nzbrcom.dll@0x20a76] [@ nzbrcom.dll@0x1a6b1] [@ nzbrcom.dll@0x20846] [@ nzbrcom.dll@0x1a701] → [@ nzbrcom.dll@0x20a76] [@ nzbrcom.dll@0x1a6b1] [@ nzbrcom.dll@0x20846] [@ nzbrcom.dll@0x1a701] [@ nzbrcom32.dll@0x1c25b] [@ nzbrcom32.dll@0x1b92b] [@ nzbrcom32.dll@0x1c38b]
Resolution: FIXED → ---
(Assignee)

Comment 13

a year ago
Grrr... probably the first step is to determine the injection mechanism by debugging an installation of this product running against Firefox with a11y and e10s enabled. Probably in a VM. :-)

Comment 14

a year ago
njn, what aklotz said.
Flags: needinfo?(benjamin)
Dees made contact with an AnhLab engineer via private email. The engineer asked the following questions.


> 1. What are the repro steps?

We don't know. All the data we have comes from crash reports submitted by Firefox users.

Here is a search of all the submitted crash reports for the last 6 months where the crash signature contains the substring "nzbrcom":

https://crash-stats.mozilla.com/search/?signature=~nzbrcom&date=%3E%3D2016-12-08T05%3A41%3A00.000Z&date=%3C2016-12-15T05%3A41%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Current results:

> 1 nzbrcom32.dll@0x1c25b  181  73.58%
> 2 nzbrcom.dll@0x20846     30  12.20%
> 3 nzbrcom32.dll@0x1c51b   30  12.20% 	
> 4 hang | NtUserMessageCall | NtUserMessageCall | nzbrcom32.dll@0x212cf 1 0.41 % 	
> 5 nzbrcom.dll@0x20d96      1 	0.41% 
> 6 nzbrcom.dll@0x2d6c6      1  0.41% 
> 7 nzbrcom.dll@0x46a0       1  0.41% 
> 8 nzbrcom32.dll@0x1b92b    1  0.41%

The crash reports are grouped by "signature", which is based on the stack trace
for the crash. You can see almost 3/4 of the crashes have a top stack frame of
"nzbrcom32.dll@0x1c25b". If we focus on just those crashes:

https://crash-stats.mozilla.com/signature/?_sort=-date&signature=nzbrcom32.dll%400x1c25b&date=%3E%3D2016-12-08T05%3A41%3A00.000Z&date=%3C2016-12-15T05%3A41%3A00.000Z

You can see the individual crash reports by clicking on the "Reports" tab.


> 2. What is the AhnLab product related to the issue? V3 Internet Security 8.0? or 9.0? or AhnLab Safe Transaction?

I don't know. I looked at the "modules" tab for a few of the individual crash reports and they all had the following module info about nzbrcom32.cll:

- module name: nzbrcom32.dll
- version: 1.0.0.280
- debug identifier: BA1C6709FE2F4FA69B7617EB1E9594AF5
- debug filename: NzBrcom32.pdb

Here is this info for an example crash report: https://crash-stats.mozilla.com/report/index/add26a83-62b8-4453-b063-db1f42161209#tab-modules

We also have the following correlations for this crash signature:

> (100.0% in signature vs 00.46% overall) Module "nzbrcom32.dll" = true
> (100.0% in signature vs 00.46% overall) Module "IAccessible2Proxy32.dll" = true
> (100.0% in signature vs 00.55% overall) Module "klib.dll" = true
> (100.0% in signature vs 00.56% overall) Module "imkrapi.dll" = true
> (100.0% in signature vs 00.56% overall) Module "imkrtip.dll" = true
> (100.0% in signature vs 00.58% overall) Module "mfc90u.dll" = true
> (100.0% in signature vs 00.91% overall) Module "IMETIP.DLL" = true
> (100.0% in signature vs 01.52% overall) Module "msvcp90.dll" = true
> (100.0% in signature vs 02.60% overall) Module "msvcr90.dll" = true
> (99.26% in signature vs 00.84% overall) Module "IMJKAPI.DLL" = true
> (100.0% in signature vs 05.40% overall) Module "dui70.dll" = true
> (100.0% in signature vs 05.52% overall) Module "imagehlp.dll" = true

It's possible that this information might help identify which product is being used.


> 3. What is the exact version of Firefox which has the problem? I can find
> 52.0a2 on developer edition web site but can’t find 52.0a1.

All the crashes are happening in 52.0a2, which is the current Developer Edition
version. (52.0a1 was the version number used when that version of Firefox was
in active development.)


> 4. I need a technical analysis report. There is no technical analysis of the
> crash. There is no direct evidence concerned with AhnLab module and no call
> stacks. So I need a reasonable analysis description why you think the problem
> is caused by AhnLab module, especially, which module in AhnLab product and
> what action affected your module.

Every individual crash report has a variety of information, including the stack trace for the crashing thread.
https://crash-stats.mozilla.com/report/index/add26a83-62b8-4453-b063-db1f42161209#tab-details has the following stack trace:

> 0 	nzbrcom32.dll 	nzbrcom32.dll@0x1c25b 	
> 1 	oleacc.dll 	AccWrap_Base::get_accChildCount(long*) 	
> 2 	nzbrcom32.dll 	nzbrcom32.dll@0x17807 	
> 3 	ntdll.dll 	ntdll.dll@0x4e4f4 	
> 4 	rpcrt4.dll 	rpcrt4.dll@0x1b51f 	
> 5 	nzbrcom32.dll 	nzbrcom32.dll@0x6d13f 	
> 6 	nzbrcom32.dll 	nzbrcom32.dll@0x6d809 	
> 7 	ntdll.dll 	ntdll.dll@0x6ff4f 	
> 8 	nzbrcom32.dll 	nzbrcom32.dll@0x6ddf7 	
> 9 	ntdll.dll 	ntdll.dll@0x6ff4f 	
> 10 	xul.dll 	_SEH_epilog4 	
> 11 	firefox.exe 	_SEH_epilog4 	
> 12 	ntdll.dll 	ntdll.dll@0x75eef 	
> 13 	ntdll.dll 	ntdll.dll@0x8261e 	
> 14 	firefox.exe 	__scrt_common_main_seh 	f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:282 

Not all the crash reports have exactly this stack trace, but they're all
similar. The crash is clearly happening within nzbrcom32.dll.
Adding another signature which only seems to show up in Aurora 52.0a2 starting with 20161201004019.
Crash Signature: [@ nzbrcom.dll@0x20a76] [@ nzbrcom.dll@0x1a6b1] [@ nzbrcom.dll@0x20846] [@ nzbrcom.dll@0x1a701] [@ nzbrcom32.dll@0x1c25b] [@ nzbrcom32.dll@0x1b92b] [@ nzbrcom32.dll@0x1c38b] → [@ nzbrcom.dll@0x20a76] [@ nzbrcom.dll@0x1a6b1] [@ nzbrcom.dll@0x20846] [@ nzbrcom.dll@0x1a701] [@ nzbrcom32.dll@0x1c25b] [@ nzbrcom32.dll@0x1b92b] [@ nzbrcom32.dll@0x1c38b] [@ nzbrcom32.dll@0x1c51b]
As an aside, this should probably be moved out of Firefox:Untriaged at this point. Perhaps accessibility (per comment 1) or External Software makes more sense.

Comment 18

a year ago
Hi I'm developer of AhnLab Safe Transaction Development Team
I tried to reproduce it.
I can not reproduce well, but I found several reproduction cases.
This issue is not a problem until Firefox 51, but it has been reproduced since Firefox 52.
Crash occurs when using VARIANTs obtained through the AccessibleChildren function.

The problem occurs when the vt type of the VARIANT is VT_DISPATCH and the value of pdispVal is NULL.
We will be distributing AhnLab SafeTransaction by adding a defense code for this problem.

Please check the reason for this problem in Firefox 52.
(Assignee)

Comment 19

a year ago
(In reply to deguls from comment #18)
> Please check the reason for this problem in Firefox 52.

I fixed bug 1319640 which landed on 52 for the December 15 build. Are you seeing this on the most recent build?
Flags: needinfo?(deguls)

Comment 20

a year ago
It has not been reproduced in Firefox 52 2016-12-15 build.
It appears to be resolved by bug 1319640.

Updated

a year ago
Status: REOPENED → RESOLVED
Last Resolved: a year agoa year ago
Resolution: --- → WORKSFORME

Comment 21

a year ago
Hello Aaron Klotz
Because the cause of the problem is known
I am asking you to remove nzbrcom.dll that you have added to WindowsDllBlocklist.cpp
If you add nzbrcom.dll, Korean Firefox users will not be able to use Internet banking.
Hi Deguls,

The method you're using to support your product in Firefox is not supported. The supported mechanism for communicating with host processes/libraries is to use Native Messaging with a WebExtension (https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Native_messaging), vs. any kind of hook or injection of a binary into the Firefox process.

I'll send an intro email, but DLL injection/loading in the Firefox process is something we're going to continue to move away from.
 

(In reply to deguls from comment #21)
> Hello Aaron Klotz
> Because the cause of the problem is known
> I am asking you to remove nzbrcom.dll that you have added to
> WindowsDllBlocklist.cpp
> If you add nzbrcom.dll, Korean Firefox users will not be able to use
> Internet banking.
(Assignee)

Updated

a year ago
Flags: needinfo?(deguls)

Comment 23

a year ago
hi. I'm a product planner of AhnLab Safe Transaction.

Ahnlab Safe Transaction is an online bank security software that contains antivirus, personal firewall, anti-keylogger, and other functionality.
In korea, it is a mandatory security software in order to use any kind of online banking services.
 
The purpose of using dll injection is to check product running on web browser and detect WinAPI hooking viruses(zeus etc).
We are aware of that it is against your policy. 
However dll injection is the only way to meet online banking compliance in Korea(for zeus detection, etc).

The crash issue was resolved(bug 1319640). 
it is all taken care of and won’t happen anymore.
 
So I am requesting you to remove nzbrcom.dll from your blocklist.
You need to log in before you can comment on or make changes to this bug.