Closed Bug 1229252 Opened 8 years ago Closed 6 years ago

startup crashes in nss3.dll@0x1e5b60 | GetFileInfo, due to software from ESET (eOppMonitor.dll)

Categories

(External Software Affecting Firefox :: Other, defect)

Unspecified
Windows NT
defect
Not set
critical

Tracking

(firefox45 wontfix, firefox46+ wontfix, firefox47- wontfix, firefox48+ wontfix, firefox49+ wontfix, firefox50+ verified, firefox51+ verified)

RESOLVED WONTFIX
Tracking Status
firefox45 --- wontfix
firefox46 + wontfix
firefox47 - wontfix
firefox48 + wontfix
firefox49 + wontfix
firefox50 + verified
firefox51 + verified

People

(Reporter: dbaron, Unassigned)

References

Details

(Keywords: crash, topcrash, Whiteboard: [AV:ESET])

Crash Data

Attachments

(2 files, 1 obsolete file)

This bug was filed from the Socorro interface and is 
report bp-e45e3b91-ce7f-465b-8521-8528a2151117.
=============================================================

This is showing up in the list of top crashes on nightly, although not at the top of the list.  It appears to be a crash we've had for a while across release channels, at relatively low frequency.  (The occurrences on release aren't much higher than those on nightly... although I assume we're only processing a small percentage of release crashes, still?)  The signature seems to move around every so often, possibly in conjunction with NSS updates, and a given signature (i.e., numeric address) seems to propagate its way through the channels.

(Is there a reason we don't have symbols for nss3.dll?)

A list of crashes over the last few months is:
https://crash-stats.mozilla.com/search/?product=Firefox&platform=Windows&signature=^|+GetFileInfo&date=%3E2015-08-01&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature


It's really not clear to me from the stack what's going on here, but I'm going to guess a component.
Any chance you can make sense of the stack, and maybe redirect appropriately?
Flags: needinfo?(nfroyd)
(In reply to David Baron [:dbaron] ⌚UTC-8 from comment #1)
> Any chance you can make sense of the stack, and maybe redirect appropriately?

I will take a look; any deeper examination will need to wait for my minidump access.
Flags: needinfo?(nfroyd)
(In reply to David Baron [:dbaron] ⌚UTC-8 from comment #0)
> This bug was filed from the Socorro interface and is 
> report bp-e45e3b91-ce7f-465b-8521-8528a2151117.
> =============================================================
> 
> This is showing up in the list of top crashes on nightly, although not at
> the top of the list.  It appears to be a crash we've had for a while across
> release channels, at relatively low frequency.  (The occurrences on release
> aren't much higher than those on nightly... although I assume we're only
> processing a small percentage of release crashes, still?)  The signature
> seems to move around every so often, possibly in conjunction with NSS
> updates, and a given signature (i.e., numeric address) seems to propagate
> its way through the channels.
> 
> (Is there a reason we don't have symbols for nss3.dll?)

ni? Ted for this one.
Flags: needinfo?(ted)
I don't get the stack from the crash report linked in comment 0:

Source
0 	nss3.dll 	nss3.dll@0x1ddad0 	
1 		@0x500000018 	
2 	xul.dll 	GetFileInfo 	xpcom/io/nsLocalFileWin.cpp
3 	xul.dll 	nsLocalFile::ResolveAndStat() 	xpcom/io/nsLocalFileWin.cpp

The problem is that the only interesting external function that GetFileInfo calls is GetFileAttributesExW...nothing anywhere near nss.dll.  So how are we ending up there?  That 0x50000018 address looks really suspicious.
Crash Signature: [@ nss3.dll@0x1e5b60 | GetFileInfo] → [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e4b60 | GetFileInfo]
We do have symbols for nss3.dll, if we didn't that stack frame would be color-coded red! If you look in the "raw dump" tab, search for "modules" and then scroll down or search for nss3, you can see that in fact we loaded symbols for nss3:
 "filename": "nss3.dll", "loaded_symbols": true,

I think something is calling into bogus memory here. The rest of the stack might be questionable because the top frame is bogus.
Flags: needinfo?(ted)
I ran this through my dumplookup tool and here's a plausible few top frames:
0x000000a0ba9fefe8: nss3.dll!FileAvailable [prfile.c:51fa3e0d4f7b : 123 + 0xd] http://hg.mozilla.org/mozilla-central/annotate/51fa3e0d4f7b/nsprpub/pr/src/io/prfile.c#l123
0x000000a0ba9ff018: xul.dll!CrashReporter::GetFileContents [nsExceptionHandler.cpp:51fa3e0d4f7b : 1406 + 0x19] http://hg.mozilla.org/mozilla-central/annotate/51fa3e0d4f7b/toolkit/crashreporter/nsExceptionHandler.cpp#l1406
0x000000a0ba9ff048: xul.dll!CrashReporter::GetOrInit [nsExceptionHandler.cpp:51fa3e0d4f7b : 1451 + 0xd] hg.mozilla.org/mozilla-central/annotate/51fa3e0d4f7b/toolkit/crashreporter/nsExceptionHandler.cpp#l1451
0x000000a0ba9ff060: xul.dll!CrashReporter::InitInstallTime [nsExceptionHandler.cpp:51fa3e0d4f7b : 1461 + 0x0]
0x000000a0ba9ff098: xul.dll!CrashReporter::SetupExtraData(nsIFile *,nsACString_internal const &) [nsExceptionHandler.cpp:51fa3e0d4f7b : 1529 + 0x3c]

...which makes me think maybe something is interposing itself in our IO calls here?
Crash Signature: [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e4b60 | GetFileInfo] → [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e4b60 | GetFileInfo] [@ nss3.dll@0x1ebb60 | GetFileInfo]
It looks like many of these reports have "eOppMonitor.dll" which appears to be some
security product from "ESET" http://www.eset.com/

There's quite a few crashes matching "GetFileInfo" spread out over many addresses:
https://crash-stats.mozilla.com/search/?signature=~GetFileInfo
I suspect that many of them could be related to this bug.
We might want to reach out to the vendor in this case, who handles that sort of thing?
Flags: needinfo?(dbolter)
Switching NI to dbaron, please re-NI me if needed. (I'm not familiar with ESET)
Flags: needinfo?(dbolter) → needinfo?(dbaron)
Correlation data in https://crash-analysis.mozilla.com/crash_analysis/20160318/ for 45.0 confirms:

  nss3.dll@0x1e5b60 | GetFileInfo|EXCEPTION_ACCESS_VIOLATION_EXEC (50 crashes)                                          
    100% (50/50) vs.   1% (198/32179) eOppMonitor.dll
    100% (50/50) vs.   4% (1290/32179) CRYPTBASE.DLL
     86% (43/50) vs.  17% (5479/32179) dbgcore.dll
    100% (50/50) vs.  31% (10093/32179) kernel.appcore.dll
    100% (50/50) vs.  32% (10411/32179) WINMMBASE.dll
     etc.

which definitely implicates whatever is associated with eOppMonitor.dll (and maybe CRYPTBASE.DLL).
I think who can contact / deal with products that cause Firefox to crash is perhaps a question for Sylvestre or KaiRo (but KaiRo is away).
Flags: needinfo?(dbaron) → needinfo?(sledru)
Crash Signature: [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e4b60 | GetFileInfo] [@ nss3.dll@0x1ebb60 | GetFileInfo] → [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e4b60 | GetFileInfo] [@ nss3.dll@0x1ebb60 | GetFileInfo] [@ nss3.dll@0x1e9900 | GetFileInfo]
This was mentioned in the channel meeting today as a crash that is spiking on Fx47.
Kai, I might be totally off here, but when I saw NSS3.dll, I wondered if that is something that you might be able to investigate? Thanks!
Flags: needinfo?(kaie)
This is unrelated to NSS. Per comment 10 / comment 11, this is a third-party product inserting itself into Firefox and doing something bad.
Flags: needinfo?(kaie)
We can block eOppMonitor.dll if necessary, if the vendor doesn't come through any time soon.
I don't have a contact there. However, I believe we should blocklist it on beta asap. Aaron, can you take the lead here?
Flags: needinfo?(sledru)
Assignee: nobody → aklotz
Status: NEW → ASSIGNED
There are multiple versions of that DLL causing the crash. Since there are at least three affected versions, I plan on blocking all versions until we have verification from them that they have fixed the problem on their end.
Comment on attachment 8733530 [details]
MozReview Request: Bug 1229252: Add eOppMonitor.dll (all versions) to the dll blocklist; r?bsmedberg

https://reviewboard.mozilla.org/r/41799/#review38779
Attachment #8733530 - Flags: review?(benjamin) → review+
https://hg.mozilla.org/mozilla-central/rev/683aaa453534
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Comment on attachment 8733530 [details]
MozReview Request: Bug 1229252: Add eOppMonitor.dll (all versions) to the dll blocklist; r?bsmedberg

Approval Request Comment
[Feature/regressing bug #]: Crashes caused by third-party dll
[User impact if declined]: DLL will continue to interfere with our code
[Describe test coverage new/current, TreeHerder]: On m-c
[Risks and why]: Low, trivial blocklist update
[String/UUID change made/needed]: None
Attachment #8733530 - Flags: approval-mozilla-beta?
Attachment #8733530 - Flags: approval-mozilla-aurora?
Comment on attachment 8733530 [details]
MozReview Request: Bug 1229252: Add eOppMonitor.dll (all versions) to the dll blocklist; r?bsmedberg

DLL blocklist for a signature that is spiking on all channels, Aurora47+, Beta46+
Attachment #8733530 - Flags: approval-mozilla-beta?
Attachment #8733530 - Flags: approval-mozilla-beta+
Attachment #8733530 - Flags: approval-mozilla-aurora?
Attachment #8733530 - Flags: approval-mozilla-aurora+
It's a blocking issue as it's ranked 2 in the list of 47 top-crashers.
Did this work? I think there is a bug in this patch where eOppMonitor was mixed-case and therefore won't block anything (we normalize the compared DLL name to lowercase). Reopening
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
MozReview-Commit-ID: K3NPE0eOmCl
Assignee: aklotz → benjamin
Status: REOPENED → ASSIGNED
Comment on attachment 8737210 [details] [diff] [review]
part B - Add Trusteer Rapport DLLs to the blocklist,

I am having major bzexport fail.
Attachment #8737210 - Attachment is obsolete: true
Attachment #8737213 - Flags: review?(aklotz)
Assignee: benjamin → aklotz
Comment on attachment 8737213 [details] [diff] [review]
followup - normalize to lowercase,

Review of attachment 8737213 [details] [diff] [review]:
-----------------------------------------------------------------

Doh! Thanks!
Attachment #8737213 - Flags: review?(aklotz) → review+
Comment on attachment 8737213 [details] [diff] [review]
followup - normalize to lowercase,

Approval Request Comment
[Feature/regressing bug #]: this bug
[User impact if declined]: dll block that was previously added in this bug will be ineffective
[Describe test coverage new/current, TreeHerder]: locally
[Risks and why]: None, trivial change to dll blocklist entry
[String/UUID change made/needed]: None
Attachment #8737213 - Flags: approval-mozilla-beta?
Attachment #8737213 - Flags: approval-mozilla-aurora?
Comment on attachment 8737213 [details] [diff] [review]
followup - normalize to lowercase,

Tweak for dll blocklist, fix for topcrash, please uplift.
Attachment #8737213 - Flags: approval-mozilla-beta?
Attachment #8737213 - Flags: approval-mozilla-beta+
Attachment #8737213 - Flags: approval-mozilla-aurora?
Attachment #8737213 - Flags: approval-mozilla-aurora+
https://hg.mozilla.org/mozilla-central/rev/6bd704474773
Status: ASSIGNED → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → FIXED
this blocklist entry doesn't seem to be effective unfortunately & crashes continue in 46.0b9 showing eOppMonitor.dll loaded in the list of modules. sample: https://crash-stats.mozilla.com/report/index/9bbaaf3b-2c6c-40f8-bc38-3adab2160409
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Yeah, I suspect that the DLL is being injected prior to blocklist initialization. We'll need to debug an actual instance with the ESET software installed to know what's going on.
Looks like we should back this out for 46.
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #42)
> Looks like we should back this out for 46.

backed out from aurora and beta per request from lizzard
Crash Signature: [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e4b60 | GetFileInfo] [@ nss3.dll@0x1ebb60 | GetFileInfo] [@ nss3.dll@0x1e9900 | GetFileInfo] → [@ nss3.dll@0x1e4b60 | GetFileInfo] [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e9900 | GetFileInfo] [@ nss3.dll@0x1ebb60 | GetFileInfo] [@ nss3.dll@0x1ec920 | GetFileInfo]
With the new signature "[@ nss3.dll@0x1ee700 | GetFileInfo]" this has occurred 16 times in the two Nightly builds from April 12 (20160412030235 and 20160412050029), making it easily the top crasher for those two builds.
Crash Signature: [@ nss3.dll@0x1e4b60 | GetFileInfo] [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e9900 | GetFileInfo] [@ nss3.dll@0x1ebb60 | GetFileInfo] [@ nss3.dll@0x1ec920 | GetFileInfo] → [@ nss3.dll@0x1e4b60 | GetFileInfo] [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e9900 | GetFileInfo] [@ nss3.dll@0x1ebb60 | GetFileInfo] [@ nss3.dll@0x1ec920 | GetFileInfo] [@ nss3.dll@0x1ee700 | GetFileInfo]
I'm not sure why we're backing this out; it's true that it doesn't work, but if we improve the blocklisting mechanism we'd still need to have this in the blocklist list.

Liz, you asked in email what next steps would be, and that would be installing ESET and writing down precise steps to reproduce. If you feel that's worth the effort, please talk to Softvision about taking that on.
Flags: needinfo?(lhenry)
Andrei, can someone on your team install ESET and try to reproduce the crash?
Flags: needinfo?(lhenry) → needinfo?(andrei.vaida)
Keywords: topcrash
Until now, we were not able to reproduce this crash on Windows 10x64.

Ran through various scenarios, as it follows:
*Updating from 43.0 & 44.0.2 to 45.0.2
*Navigating on the Sparkasse website (DE and AU)
*Clean installation of Firefox 45.0.2
*Stressed Fx 45.0.2 with several heavy content websites and no performance issues were noticed compared to Firefox 43 and Firefox 44 (both x86 and x64 builds).

The Nod32 version used into testing is 9.0.375.0 (30 days free trial) - downloaded from http://www.eset.com/us/free-trial/

We're currently setting up a few VMs as well, we'll follow up with the results.
Flags: needinfo?(andrei.vaida) → needinfo?(cornel.ionce)
Crash Signature: [@ nss3.dll@0x1e4b60 | GetFileInfo] [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e9900 | GetFileInfo] [@ nss3.dll@0x1ebb60 | GetFileInfo] [@ nss3.dll@0x1ec920 | GetFileInfo] [@ nss3.dll@0x1ee700 | GetFileInfo] → [@ nss3.dll@0x1e4b60 | GetFileInfo] [@ nss3.dll@0x1e5b60 | GetFileInfo] [@ nss3.dll@0x1e9900 | GetFileInfo] [@ nss3.dll@0x1ebb60 | GetFileInfo] [@ nss3.dll@0x1ec920 | GetFileInfo] [@ nss3.dll@0x1ee700 | GetFileInfo] [@ nss3.dll@0x1e8700 | GetFileInf…
I've also tried and failed to reproduce this crash using two VMs[1] with clean Windows installs.
Tested with the above mentioned scenarios and no crash occured.
Used the en-GB and zh-CN builds too.
The same free trial of Nod32 version was used (9.0.375.0).

[1] Windows 10x64 and Windows 8.1x64
Flags: needinfo?(cornel.ionce)
Crash Signature: GetFileInfo] → GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo]
Hi Aaron, this was mentioned at the channel meeting today, is there another dll blocklist that might be needed here? Thanks!
Flags: needinfo?(aklotz)
What we need is a successful repro so that we can debug this.
Flags: needinfo?(aklotz)
I just installed ESET Smart Security on a VM and intercepted the DLL injection. They are intercepting process creation and modifying the initial thread's entry point to execute a stub that loads eOppMonitor.dll before firefox!wmainCRTStartup is ever even called.

That explains why the DLL blocklist is ineffective for this case -- not a single instruction of our code has even run yet when this injection occurs!
(In reply to Aaron Klotz [:aklotz] from comment #51)
> What we need is a successful repro so that we can debug this.

This is a pretty old comment from crash signature reports but worth copying here if it helps with trying out a repro scenario "Have been usingThunderbird with Firefox and Mailwasher. But quiter often have open a Firefox window from Thunderbird and then close firefox and try to delete email from Thunderbird when it crashes or stops responding. I have try restarting thinderbird in order to delete but eventually have to restart the system."
I did some more investigation about how the DLL injection works because I can't help myself.

AFAICT when the initial thread starts running, the process address space has already been modified to include the injection stub code. The entry point of the thread is adjusted by detouring ntdll!NtContinue and doing a compare-and-swap on its Context->Eax parameter, replacing the real entry point with the stub if the value of Eax matches.

This is actually kind of scary, since that hook is essentially assuming that NtContinue is only being used for thread startup. It's doing CAS on CONTEXT::Eax which is the location of the entry point when going through the user thread start thunk. OTOH, if a thread were to return from a user-mode APC and the previous context's EAX register contained a value that coincidentally happened to equal the address of the stub, that value will be swapped out! (That is another bug in this product, but it's not THE bug that we're dealing with here.)

From what I can see, the NtContinue patch and the stub code are already present before the Win32 DLLs are initialized. Given that the product includes numerous device drivers, I can only assume that they're calling something like PsSetCreateProcessNotifyRoutineEx from kernel mode to register a callback that is responsible for augmenting the process prior to initial thread execution.

TL;DR:
We need to open a channel with people at ESET to fix this because we are not going to be able to successfully create a block for eoppmonitor.dll.
s/coincidentally happened to equal the address of the stub/coincidentally happened to equal the address of the process entry point/
Crash Signature: GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] → GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo]
Crash Signature: GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] → GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5]
Crash Signature: GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] → GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5]
Liz, can you please clarify the status of this bug? The current combo of approval and tracking flags seems contradictory and could use some cleaning up.
Flags: needinfo?(lhenry)
This isn't in the top 10 crash signatures  for beta or release -- it's #35 on beta 11, not in the top 50 on release).  But I can track it for 46 anyway to make sure it doesn't become more widespread in 46. 
However this is the #3 crash in aurora, almost all windows 10.
Flags: needinfo?(lhenry)
Pinging some people at ESET.
Flags: needinfo?(szklarzewicz)
Flags: needinfo?(cook)
Crash Signature: GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] → GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e5b70 | GetFileInfo]
Crash Signature: GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e5b70 | GetFileInfo] → GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e5b70 | GetFileInfo] [@ nss3.dll@0x1e9700 | GetFileInfo]
Removing myself from assignment as there is not currently any actionable work for me here...
Assignee: aklotz → nobody
Summary: crash in nss3.dll@0x1e5b60 | GetFileInfo → startup crashes in nss3.dll@0x1e5b60 | GetFileInfo, due to software from ESET (eOppMonitor.dll)
Flags: needinfo?(mozillamarcia.knous)
Still in the top 10 crashes on aurora 48 and 49, but not showing up in the top 50 for beta or release. 
That seems odd, as it's the opposite of the pattern we usually see.
This was a blocking issue when 47 was on Aurora and it seems have gone down significantly since 47 moved to Beta channel. 47 tracking -.
Could possibly be that .default profile is safe from the crash. (unverified)
Crash Signature: GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e5b70 | GetFileInfo] [@ nss3.dll@0x1e9700 | GetFileInfo] → GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e5b70 | GetFileInfo] [@ nss3.dll@0x1e9700 | GetFileInfo] [@ nss3.…
This is the top #2 crashes for the 20160516030211 Nightly although the symbol is missing.
I tried reaching out to Customer Support via the website and did not hear back. I also emailed the two individuals who we have ni on in this bug. I will report back when I have more information.
Flags: needinfo?(mozillamarcia.knous)
(In reply to Marcia Knous [:marcia - use ni] from comment #64)
> I tried reaching out to Customer Support via the website and did not hear
> back. I also emailed the two individuals who we have ni on in this bug. I
> will report back when I have more information.

I got a return email that indicated the following: "The team responsible has been notified of the issue. They will contact you directly, or possibly through the Bugzilla."
I am in email communication with a developer who works at ESET - he is requesting crash dumps. I assume that will be difficult since they are in Slovakia, so what other path do we want to follow here?
We can't give them crash dumps because that contains too much private user information.
Cleaning up the flags here. We can't fix this and can't blocklist eset before the crash happens. 
Marking wontfix for 46/47 mostly so that we don't end up checking this every few days.  
And tracking/affected for 48 and 49.  Sorry this was confusing. 
I hope that if eset is able to fix this on their end, it will fix the crash for all versions.
(In reply to Marcia Knous [:marcia - use ni] from comment #66)
> I am in email communication with a developer who works at ESET - he is
> requesting crash dumps. I assume that will be difficult since they are in
> Slovakia, so what other path do we want to follow here?

Benjamin, which options do we have there? I know we can't just hand out crash dumps but you have explored options for what we can do there in the past, any way we can help them there to get this debugged?
Flags: needinfo?(benjamin)
I heard back from the developer and he said they fixed a similar issue with the same symptoms that will eventually be pushed out to their users - but he is not certain whether it will fix this issue.
Crash Signature: nss3.dll@0x1e9700] → nss3.dll@0x1e0b40 | GetFileInfo] [@ nss3.dll@0x1e76d0 | GetFileInfo] [@ nss3.dll@0x1e66d0 | GetFileInfo] [@ nss3.dll@0x1dba10 | GetFileInfo] [@ nss3.dll@0x1e9700]
We can give them minidumps from particular users who have given specific consent, or dumps from our internal QA if they have reproduced the crash. Otherwise, there is no effective way to give out dumps.
Flags: needinfo?(benjamin)
Crash Signature: nss3.dll@0x1e0b40 | GetFileInfo] [@ nss3.dll@0x1e76d0 | GetFileInfo] [@ nss3.dll@0x1e66d0 | GetFileInfo] [@ nss3.dll@0x1dba10 | GetFileInfo] [@ nss3.dll@0x1e9700] → nss3.dll@0x1e0b40 | GetFileInfo] [@ nss3.dll@0x1e76d0 | GetFileInfo] [@ nss3.dll@0x1e66d0 | GetFileInfo] [@ nss3.dll@0x1dba10 | GetFileInfo] [@ nss3.dll@0x1e9700] [@ nss3.dll@0x1e8700 | nsLocalFile::Append ]
Update from my contact at ESET:

I looked thoroughly to all available information about the issue at Bug 1229252#c70 and I think problem I fixed and problem we have now are different. This week I’m going to review code and fix all possible problematic places. After that I’ll initiate release process of update. As soon as I will get information about possible release date I’ll let you know.
Flags: needinfo?(szklarzewicz)
Flags: needinfo?(cook)
Crash Signature: nss3.dll@0x1e0b40 | GetFileInfo] [@ nss3.dll@0x1e76d0 | GetFileInfo] [@ nss3.dll@0x1e66d0 | GetFileInfo] [@ nss3.dll@0x1dba10 | GetFileInfo] [@ nss3.dll@0x1e9700] [@ nss3.dll@0x1e8700 | nsLocalFile::Append ] → nss3.dll@0x1e0b40 | GetFileInfo] [@ nss3.dll@0x1e76d0 | GetFileInfo] [@ nss3.dll@0x1e66d0 | GetFileInfo] [@ nss3.dll@0x1dba10 | GetFileInfo] [@ nss3.dll@0x1e9700] [@ nss3.dll@0x1e8700 | nsLocalFile::Append ] [@ nss3.dll@0x1e7700 | GetFileInfo]
A user on Reddit can reproduce this. Apparently ESET stops working correctly after the crash.
https://www.reddit.com/r/firefox/comments/4m0i9j/startup_crash_cause_nss3dll/
Crash Signature: nss3.dll@0x1e0b40 | GetFileInfo] [@ nss3.dll@0x1e76d0 | GetFileInfo] [@ nss3.dll@0x1e66d0 | GetFileInfo] [@ nss3.dll@0x1dba10 | GetFileInfo] [@ nss3.dll@0x1e9700] [@ nss3.dll@0x1e8700 | nsLocalFile::Append ] [@ nss3.dll@0x1e7700 | GetFileInfo] → nss3.dll@0x1e0b40 | GetFileInfo] [@ nss3.dll@0x1e76d0 | GetFileInfo] [@ nss3.dll@0x1e66d0 | GetFileInfo] [@ nss3.dll@0x1dba10 | GetFileInfo] [@ nss3.dll@0x1e9700] [@ nss3.dll@0x1e8700 | nsLocalFile::Append ] [@ nss3.dll@0x1e7700 | GetFileInfo] [@ …
Crash Signature: nss3.dll@0x1e7700 | kernelbase.dll@0x98b5] → nss3.dll@0x1e7700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e8720 | GetFileInfo]
Crash Signature: nss3.dll@0x1e7700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e8720 | GetFileInfo] → nss3.dll@0x1e7700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e8720 | GetFileInfo] [@ nss3.dll@0x1e9720 | GetFileInfo]
Crash Signature: nss3.dll@0x1e7700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e8720 | GetFileInfo] [@ nss3.dll@0x1e9720 | GetFileInfo] → nss3.dll@0x1e7700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e8720 | GetFileInfo] [@ nss3.dll@0x1e9720 | GetFileInfo] [@ nss3.dll@0x1e7720 | GetFileInfo]
Crash Signature: nss3.dll@0x1e7700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e8720 | GetFileInfo] [@ nss3.dll@0x1e9720 | GetFileInfo] [@ nss3.dll@0x1e7720 | GetFileInfo] → nss3.dll@0x1e7700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e8720 | GetFileInfo] [@ nss3.dll@0x1e9720 | GetFileInfo] [@ nss3.dll@0x1e7720 | GetFileInfo] [@ nss3.dll@0x1e7720 | nsLocalFile::Append]
Hi, I have two days for this  bug, i cant use the Firefox.
If you need to remote desktop for checking and finding a solution for this bug,I can do it

my report id :bp-91aa229e-b579-4d4a-999b-071702160624.
Crash Signature: nss3.dll@0x1e7700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e8720 | GetFileInfo] [@ nss3.dll@0x1e9720 | GetFileInfo] [@ nss3.dll@0x1e7720 | GetFileInfo] [@ nss3.dll@0x1e7720 | nsLocalFile::Append] → nss3.dll@0x1e7700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e8720 | GetFileInfo] [@ nss3.dll@0x1e9720 | GetFileInfo] [@ nss3.dll@0x1e7720 | GetFileInfo] [@ nss3.dll@0x1e7720 | nsLocalFile::Append] [@ nss3.dll@0x1e2b10 | GetFileInfo] [@ nss3.dll@0x1e1…
The developer from ESET advises that an update is coming, but I don't yet have any further information about the date. It is also uncertain as to whether it will fix these crashes.
Crash Signature: nss3.dll@0x1e1ad0 | GetFileInfo] → nss3.dll@0x1e1ad0 | GetFileInfo] [@ nss3.dll@0x1e56d0 | GetFileInfo] [@ nss3.dll@0x1eb720 | GetFileInfo]
Crash Signature: nss3.dll@0x1e1ad0 | GetFileInfo] [@ nss3.dll@0x1e56d0 | GetFileInfo] [@ nss3.dll@0x1eb720 | GetFileInfo] → nss3.dll@0x1e1ad0 | GetFileInfo] [@ nss3.dll@0x1e56d0 | GetFileInfo] [@ nss3.dll@0x1eb720 | GetFileInfo] [@ nss3.dll@0x1ec720 | GetFileInfo] [@ nss3.dll@0x1ec720 | GetFileInfo] [@ nss3.dll@0x1ddad0 | GetFileInfo] [@ nss3.dll@0x1d99e0 | GetFileInfo]…
Crash Signature: GetFileInfo] [@ nss3.dll@0x1d89e0 | GetFileInfo] → GetFileInfo] [@ nss3.dll@0x1d89e0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | GetFileInfo]
Crash volume for signature 'nss3.dll@0x1e7720 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e9720 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1d89e0 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1d99e0 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e8720 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e7720 | nsLocalFile::Append': None.
Crash volume for signature 'nss3.dll@0x1ebb60 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1ea700 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e9700': None.
Crash volume for signature 'nss3.dll@0x1e8700 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e7700 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e1ad0 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1ee700 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e56d0 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e5b70 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e7700 | kernelbase.dll@0x98b5': None.
Crash volume for signature 'nss3.dll@0x1e8700 | nsLocalFile::Append': None.
Crash volume for signature 'nss3.dll@0x1e4b60 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e2b10 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e5b60 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1eb700 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e9900 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e76d0 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1eb700 | kernelbase.dll@0x98b5': None.
Crash volume for signature 'nss3.dll@0x1ddad0 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1dba10 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1ec720 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1e6b70 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1ec920 | GetFileInfo': None.
Crash volume for signature 'nss3.dll@0x1eb720 | GetFileInfo': None.

Crash volume for signature 'nss3.dll@0x17a3a0 | GetFileInfo':
 - Ni.  (version 51): 75 crashes from 2016-08-01.
 - Aur. (version 50): 68 crashes from 2016-08-01.
 - Beta (version 49): 0 crashes from 2016-08-02.
 - Rel. (version 48): 0 crashes from 2016-07-25.
 - Esr  (version 45): 0 crashes from 2016-04-20.

Crash volume on the last weeks (Week N is from 08-08 to 08-14):
              W. N-1
 - Ni.  (51)      20
 - Aur. (50)      34

Affected platform: Windows

Crash volume for signature 'nss3.dll@0x1e9700 | GetFileInfo':
 - Ni.  (version 51): 0 crashes from 2016-08-01.
 - Aur. (version 50): 0 crashes from 2016-08-01.
 - Beta (version 49): 368 crashes from 2016-08-02.
 - Rel. (version 48): 0 crashes from 2016-07-25.
 - Esr  (version 45): 0 crashes from 2016-04-20.

Crash volume on the last weeks (Week N is from 08-08 to 08-14):
              W. N-1
 - Beta (49)      92

Affected platform: Windows

Crash volume for signature 'nss3.dll@0x1e0b40 | GetFileInfo':
 - Ni.  (version 51): 0 crashes from 2016-08-01.
 - Aur. (version 50): 0 crashes from 2016-08-01.
 - Beta (version 49): 0 crashes from 2016-08-02.
 - Rel. (version 48): 0 crashes from 2016-07-25.
 - Esr  (version 45): 422 crashes from 2016-04-20.

Crash volume on the last weeks (Week N is from 08-08 to 08-14):
              W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - Esr  (45)      19      26      59      33      18      25      17

Affected platform: Windows

Crash volume for signature 'nss3.dll@0x1e66d0 | GetFileInfo':
 - Ni.  (version 51): 0 crashes from 2016-08-01.
 - Aur. (version 50): 0 crashes from 2016-08-01.
 - Beta (version 49): 0 crashes from 2016-08-02.
 - Rel. (version 48): 544 crashes from 2016-07-25.
 - Esr  (version 45): 0 crashes from 2016-04-20.

Crash volume on the last weeks (Week N is from 08-08 to 08-14):
              W. N-1  W. N-2
 - Rel. (48)     282       5

Affected platform: Windows
Hello Saleh, could we share the information from your crash report with the ESET developers? This might help fixing the issue. (Please note that the report may contain private information, so we understand if you don't want to share it).
Flags: needinfo?(saleh.souzanchi)
Too late for 48 but still interested to take a fix in 49
Crash Signature: GetFileInfo] [@ nss3.dll@0x1d89e0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | GetFileInfo] → GetFileInfo] [@ nss3.dll@0x1d89e0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | nsLocalFile::Append] [@ nss3.dll@0x17a3a0 | GetVolumeNameForRoot] [@ nss3.dll@0x17a3a0 | GetFileAttributesExW] [@ nss3.dll@0x1e9900 | GetFil…
Crash Signature: GetFileInfo] [@ nss3.dll@0x1d89e0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | nsLocalFile::Append] [@ nss3.dll@0x17a3a0 | GetVolumeNameForRoot] [@ nss3.dll@0x17a3a0 | GetFileAttributesExW] [@ nss3.dll@0x1e9900 | GetFil… → GetFileInfo] [@ nss3.dll@0x1d89e0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | GetFileInfo] [@ nss3.dll@0x17b500 | GetFileInfo] [@ nss3.dll@0x17a3a0 | nsLocalFile::Append] [@ nss3.dll@0x17a3a0 | GetVolumeNameForRoot] [@ nss3.dll@0x17a3a0 | GetFileAttribut…
I think that people may still be crashing, but using 9.0.376.0 of the dll. There was a fix that went in for 9.0.383.x or above. Calixte: Can you scrape the various crash reports and report if anyone is crashing on Version 9.0.383.x or above? If so, the fix didn't work.
Flags: needinfo?(cdenizet)
I analyzed a lot of recent crashes (2604) with one of the above signatures from the 2016-08-10 to today, and I didn't find anything with a version greater than 9.0.383.x.
Flags: needinfo?(cdenizet)
Crash Signature: GetFileInfo] [@ nss3.dll@0x1d89e0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | GetFileInfo] [@ nss3.dll@0x17b500 | GetFileInfo] [@ nss3.dll@0x17a3a0 | nsLocalFile::Append] [@ nss3.dll@0x17a3a0 | GetVolumeNameForRoot] [@ nss3.dll@0x17a3a0 | GetFileAttribut… → GetFileInfo] [@ nss3.dll@0x1d89e0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | GetFileInfo] [@ nss3.dll@0x17b500 | GetFileInfo] [@ nss3.dll@0x178500 | GetFileInfo] [@ nss3.dll@0x17a3a0 | nsLocalFile::Append] [@ nss3.dll@0x17a3a0 | GetVolumeNameForRoot] […
I think this will need to wait for 50, it's a problem we've had for a while and it doesn't look worse on beta than it's been historically. If it 's significantly worse on release we can try blocklisting more versions.
This is the #1 top crash for 64-bit Firefox (with ~500 reports), but has only ~100 reports on 32-bit and doesn't appear in the top 50.
Crash Signature: GetFileInfo] [@ nss3.dll@0x1d89e0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | GetFileInfo] [@ nss3.dll@0x17b500 | GetFileInfo] [@ nss3.dll@0x178500 | GetFileInfo] [@ nss3.dll@0x17a3a0 | nsLocalFile::Append] [@ nss3.dll@0x17a3a0 | GetVolumeNameForRoot] [… → GetFileInfo] [@ nss3.dll@0x1d89e0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | GetFileInfo] [@ nss3.dll@0x17b500 | GetFileInfo] [@ nss3.dll@0x178500 | GetFileInfo] [@ nss3.dll@0x1773a0 | GetFileInfo] [@ nss3.dll@0x17a3a0 | nsLocalFile::Append] [@ nss3.dl…
Crash Signature: GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x1e5b70 | GetFileInfo] [@ nss3.dll@0x1e9700 | GetFileInfo] [@ nss3.… → GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x17a500 | kernelbase.dll@0x27ea1] [@ nss3.dll@0x1e5b70 | GetFileInfo…
just as a general heads-up - there was a huge spike (2650 recorded crashes) in those startup crashes again earlier today within a 4-hour-window hitting firefox 49 64bit users on windows 10: http://bit.ly/2d52Jqs 

the crashing has gone down since then again, so this indicates that there was a botched signature update coming from eset triggering this issue, that they have quickly pulled again.
(In reply to [:philipp] from comment #83)
> just as a general heads-up - there was a huge spike (2650 recorded crashes)
> in those startup crashes again earlier today within a 4-hour-window hitting
> firefox 49 64bit users on windows 10: http://bit.ly/2d52Jqs 
> 
> the crashing has gone down since then again, so this indicates that there
> was a botched signature update coming from eset triggering this issue, that
> they have quickly pulled again.

Thanks that is a really great investigation! How did you figure out there was a 4-hour-window spike?

How can we know for sure if this was a botched signature update from eset?
Flags: needinfo?(madperson)
this signature was at the top of the list for crashers in 49 when i looked in the morning, so i was going to the individual crash reports at https://crash-stats.mozilla.com/signature/?product=Firefox&process_type=browser&signature=nss3.dll%400x1ea700%20%7C%20GetFileInfo#reports (back to page #54) where it becomes apparent that between 2016-09-23 02:45:10 and 06:29:30 (UTC) we received new instances of this crash every few seconds, whereas otherwise there it's only one every few hours.

i can't say for sure that it was triggered by a signature update (as i was not affected myself), but that was the only plausible explanation i could come up for this pattern...
Flags: needinfo?(madperson)
Crash Signature: GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x17a500 | kernelbase.dll@0x27ea1] [@ nss3.dll@0x1e5b70 | GetFileInfo… → GetFileInfo] [@ nss3.dll@0x1e6b70 | GetFileInfo] [@ nss3.dll@0x1eb700 | GetFileInfo] [@ nss3.dll@0x1ea700 | GetFileInfo] [@ nss3.dll@0x1eb700 | kernelbase.dll@0x98b5] [@ nss3.dll@0x17a500 | kernelbase.dll@0x27ea1] [@ nss3.dll@0x17b500 | kernelbase.…
Hi. This issue happened on my machine. I'm using ESET Smart Security 9.0.375.3.
Some of my crash reports are shown below. It is OK to send my reports to ESET developers, including crash dumps.

bp-ab1ab151-a1b0-4d33-ab48-93f562161002
bp-abe3b140-facd-414a-9d65-d4fc32161001
bp-e02e753b-9c6a-4bcd-aaa9-856c92161001
bp-48d1623c-50f0-473b-8f94-2f4432161001
bp-d08ed49d-0185-4792-9570-aa95b2161001
Marcia, MISE accepted to share his crash dumps with ESET. Could you send them to your
ESET contact?
Flags: needinfo?(mozillamarcia.knous)
MISE: Is it possible to try version 9.0.383.x or higher to see if it crashes? I was told by the developers that update might have addressed the crash.

Otherwise, I think that the developers want the crash dumps, which I will we will have to request on our end.
Flags: needinfo?(yasmise)
Flags: needinfo?(saleh.souzanchi)
Flags: needinfo?(mozillamarcia.knous)
Could we also run a script and find out what is the most common versions that are currently crashing?
Flags: needinfo?(cdenizet)
For 49.0.1, I got:
518 crash reports have been analyzed and the following versions have been found:
 - eoppmonitor.dll
   - 9.0.374.0: 189
   - 9.0.349.0: 104
   - 9.0.376.0: 218
   - 9.0.318.0: 3
   - 9.0.141.0: 4

And for 49.0, I got:
2939 crash reports have been analyzed and the following versions have been found:
 - eoppmonitor.dll
   - 9.0.349.0: 2831
   - 9.0.374.0: 59
   - 9.0.376.0: 48
   - 9.0.141.0: 1
Flags: needinfo?(cdenizet)
Marcia:
Thanks for your reply.
Unfortunately, I couldn't try 9.0.383.x for now because Firefox works fine now. After reporting here on comment 86, I tried "Run as administrator" and it worked. So for me, this issue is resolved.

ESET developers:
Version 9.0.383.x is not available for Japanese. 9.0.375.3 is the latest. I wish I could have a patch for old versions.
Flags: needinfo?(yasmise)
Hi Marco, Calixte, is there a blocklist mechanism that might work on this one? Or an add-on block perhaps?
Flags: needinfo?(mcastelluccio)
Flags: needinfo?(cdenizet)
Unfortunately we've already tried blocking the DLL, but it was ineffective.

Marcia received a reply from an ESET person, who said that he looked at the crash dump and the problem was actually fixed a long time ago. Last week he "updated the product not to use the old version of eOppMonitor.dll (below 9.0.378.0)" and he was expecting the number of crashes to go down.

Looking at crash stats, this is indeed the case. We still have a few reports, probably from people who haven't updated yet, but the volume is quite small.
Flags: needinfo?(mcastelluccio)
Flags: needinfo?(cdenizet)
Thanks Marco! That's great. I am updating the status on this one to verified even though the fix didn't come from our side.
I also don't see crashes in 51. Mark 51 as verified.
The crash volume seems low enough that this bug doesn't need to block the 64-bit stub installer roll-out (for 52-53).
No longer blocks: support-win64
Crash Signature: GetFileAttributesExW] [@ nss3.dll@0x1e66d0 | GetFileAttributesExW] → GetFileAttributesExW] [@ nss3.dll@0x1e66d0 | GetFileAttributesExW] [@ nss3.dll@0x17e4a0 | GetFileAttributesExW]
Closing because no crash reported since 12 weeks.
Status: REOPENED → RESOLVED
Closed: 8 years ago6 years ago
Resolution: --- → WONTFIX
See Also: → 1251020
Component: XPCOM → Other
Product: Core → External Software Affecting Firefox
Whiteboard: [AV:ESET]
Target Milestone: mozilla48 → ---
Version: Trunk → unspecified
You need to log in before you can comment on or make changes to this bug.