Closed Bug 1268470 Opened 4 years ago Closed 2 years ago

crash in klsihk64.dll@0x3ad7 (Kaspersky DLL)

Categories

(External Software Affecting Firefox :: Other, defect, critical)

x86_64
Windows 8
defect
Not set
critical

Tracking

(firefox47 wontfix, firefox48 wontfix, firefox49 wontfix, firefox-esr45 wontfix, relnote-firefox 56+, firefox50 wontfix, firefox51 wontfix, firefox52 wontfix, firefox-esr52 wontfix, firefox53 wontfix, firefox55 wontfix, firefox56blocking fixed, firefox57 fixed)

RESOLVED FIXED
Tracking Status
firefox47 --- wontfix
firefox48 --- wontfix
firefox49 --- wontfix
firefox-esr45 --- wontfix
relnote-firefox --- 56+
firefox50 --- wontfix
firefox51 --- wontfix
firefox52 --- wontfix
firefox-esr52 --- wontfix
firefox53 --- wontfix
firefox55 --- wontfix
firefox56 blocking fixed
firefox57 --- fixed

People

(Reporter: marco, Assigned: jcristau)

References

Details

(Keywords: 64bit, crash)

Crash Data

Attachments

(5 files, 1 obsolete file)

This bug was filed from the Socorro interface and is 
report bp-4c7f0179-d2e8-4d60-8bcd-458c22160427.
=============================================================

Looks like this dll is included in an antivirus (Kaspersky).

This is a startup crash.
Low-frequency startup crash. Do we know anybody at Kaspersky?
Crash Signature: [@ klsihk64.dll@0x3ad7] → [@ klsihk64.dll@0x3ad7] [@ klsihk64.dll@0x5337] [@ klsihk64.dll@0x3807] [@ klsihk64.dll@0x2b37] [@ klsihk64.dll@0x33ef] [@ klsihk64.dll@0x533b]
Component: Untriaged → Other
Product: Core → External Software Affecting Firefox
Could you help us to debug this issue?
Flags: needinfo?(vladimir.kukushkin)
Flags: needinfo?(toivo.peegel)
Flags: needinfo?(kurt.baumgartner)
Flags: needinfo?(denisov_vit)
Flags: needinfo?(alexey.drozdov)
Flags: needinfo?(Andrey.Rubin)
One more needinfo. Please let us know if there is anything we can do to help out. Thanks!
Flags: needinfo?(kaspersky-antivirus)
Flags: needinfo?(kaspersky-antivirus)
Can you provide appropriated crash-dump form analyze trouble?
Unfortunately we have quite a strict policy regarding crash dumps:

* If a community member or user gives explicit permission to share their own
crash minidump with a partner, we may do so.
* We try to resolve without giving a partner direct access: if they provide us
with debug symbols, we will debug and send them non-sensitive
information/stacks.
* If no other resolution can be found, we may have a partner do direct
debugging under the following conditions:
** The partner company must sign a privacy agreement.
** The debugging must occur in a Mozilla office, on Mozilla hardware, and under
the supervision of a Mozilla employee.

---

Looks like in all these crashes there's always "JS_NewRuntime" in the stack, which calls JSRuntime::init, which is pretty complex.

There's often (or maybe always, some stacks are clearly corrupted) some function that uses AddVectoredExceptionHandler. Could it be something related to signal handlers?

Here's the most recurring stack: https://crash-stats.mozilla.com/report/index/6664c3ae-3e91-4453-84f7-56c122160707.

Another thing in common between the crashes is that they're always occurring on Windows 8 (6.2.9200).
Flags: needinfo?(Andrey.Rubin)
Crash volume for signature 'klsihk64.dll@0x5337':
 - nightly (version 50): 17 crashes from 2016-06-06.
 - aurora  (version 49): 0 crash from 2016-06-07.
 - beta    (version 48): 103 crashes from 2016-06-06.
 - release (version 47): 332 crashes from 2016-05-31.
 - esr     (version 45): 6 crashes from 2016-04-07.

Crash volume on the last weeks:
             Week N-1   Week N-2   Week N-3   Week N-4   Week N-5   Week N-6   Week N-7
 - nightly          4          0          8          3          0          0          0
 - aurora           0          0          0          0          0          0          0
 - beta            27          2          8         18          9         30          9
 - release         60         55         58         47         39         30         27
 - esr              0          0          0          4          1          0          1

Affected platform: Windows
Crash volume for signature 'klsihk64.dll@0x5337':
 - nightly(version 50):19 crashes from 2016-06-06.
 - aurora (version 49):4 crashes from 2016-06-07.
 - beta   (version 48):104 crashes from 2016-06-06.
 - release(version 47):383 crashes from 2016-05-31.
 - esr    (version 45):6 crashes from 2016-04-07.

Crash volume on the last weeks:
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       4       4       0       8       3       0       0
 - aurora        4       0       0       0       0       0       0
 - beta          1      27       2       8      18       9      30
 - release      56      60      63      58      47      39      30
 - esr           0       0       0       0       4       1       0

Affected platform: Windows
Crash volume for signature 'klsihk64.dll@0x5337':
 - nightly (version 51): 14 crashes from 2016-08-01.
 - aurora  (version 50): 14 crashes from 2016-08-01.
 - beta    (version 49): 14 crashes from 2016-08-02.
 - release (version 48): 174 crashes from 2016-07-25.
 - esr     (version 45): 13 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly       5       2       7
 - aurora        6       1       4
 - beta          4      10       0
 - release      61      53      31
 - esr           6       0       0

Affected platform: Windows

Crash rank on the last 7 days:
           Browser     Content   Plugin
 - nightly
 - aurora  #205
 - beta    #3264
 - release #455
 - esr     #4826
Does Kaspersky use the ctypes JavaScript API to call into klsihk64.dll? That might explain why the stack traces all have JS_NewRuntime calling into klsihk64.dll. Perhaps this is a binary compatibility problem with 64-bit Firefox?
Hardware: Unspecified → x86_64
The stack reported by crash-stats is inaccurate. It's likely that kaspersky is hooking something under js::wasm::EnsureSignalHandlersInstalled, probably AddVectoredExceptionHandler at https://hg.mozilla.org/mozilla-central/annotate/ab0044bfa1df/js/src/asmjs/WasmSignalHandlers.cpp#l1305

This is very unlikely to be ctypes-related.
Marco, do you have any new information about this klsihk64.dll crash? How can we help move this bug forward?

There were about 540 klsihk64.dll crashes in Firefox 49 release. Of the 4666 crashes over the last six months across all channels, 99.54% were on Windows 8.0, which just 21 crashes on Windows 8.1 or 10.
Crash Signature: [@ klsihk64.dll@0x3ad7] [@ klsihk64.dll@0x5337] [@ klsihk64.dll@0x3807] [@ klsihk64.dll@0x2b37] [@ klsihk64.dll@0x33ef] [@ klsihk64.dll@0x533b] → [@ klsihk64.dll@0x2b37] [@ klsihk64.dll@0x33ef] [@ klsihk64.dll@0x3807] [@ klsihk64.dll@0x3ad7] [@ klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b]
Flags: needinfo?(mcastelluccio)
Keywords: 64bit
OS: Windows → Windows 8
I don't have anything new. Perhaps we can try to contact some of the people who are reporting crashes and ask them for permission to share a crash dump?

Alexey, do you know if Kaspersky could be doing something that would cause AddVectoredExceptionHandler to fail? Is Kaspersky hooking AddVectoredExceptionHandler?
Flags: needinfo?(mcastelluccio)
Nightly 53 is affected. Also adding another crash signature (klsihk64.dll@0x3d19), now the most common klsihk64.dll crash address.
Crash Signature: [@ klsihk64.dll@0x2b37] [@ klsihk64.dll@0x33ef] [@ klsihk64.dll@0x3807] [@ klsihk64.dll@0x3ad7] [@ klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b] → [@ klsihk64.dll@0x2b37] [@ klsihk64.dll@0x33ef] [@ klsihk64.dll@0x3807] [@ klsihk64.dll@0x3ad7] [@ klsihk64.dll@0x3d19] [@ klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b]
Over the past six months, 99.38% of these klsihk64.dll crashes happened on Windows 8.0.
Crash Signature: [@ klsihk64.dll@0x2b37] [@ klsihk64.dll@0x33ef] [@ klsihk64.dll@0x3807] [@ klsihk64.dll@0x3ad7] [@ klsihk64.dll@0x3d19] [@ klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b] → [@ klsihk64.dll@0x1627] [@ klsihk64.dll@0x1ed4] [@ klsihk64.dll@0x2b37] [@ klsihk64.dll@0x33ef] [@ klsihk64.dll@0x33f3] [@ klsihk64.dll@0x3467] [@ klsihk64.dll@0x3807] [@ klsihk64.dll@0x3ad7] [@ klsihk64.dll@0x3adb] [@ klsihk64.dll@0x3d19] [@ kl…
Too late for firefox 52, mass-wontfix.
Crash Signature: klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b] [@ klsihk64.dll@0x183ff] → klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b] [@ klsihk64.dll@0x183ff] [@ klsihk64.dll@0x3d71]
Crash Signature: klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b] [@ klsihk64.dll@0x183ff] [@ klsihk64.dll@0x3d71] → klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b] [@ klsihk64.dll@0x183ff] [@ klsihk64.dll@0x3d71] [@ klsihk64.dll@0x3d7d]
Crash Signature: klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b] [@ klsihk64.dll@0x183ff] [@ klsihk64.dll@0x3d71] [@ klsihk64.dll@0x3d7d] → klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b] [@ klsihk64.dll@0x183ff] [@ klsihk64.dll@0x3d71] [@ klsihk64.dll@0x3d7d] [@ klsihk64.dll@0x3e2d]
[Tracking Requested - why for this release]:
kaspersky dosn't play nice with 64bit versions of firefox as it seems. in early data for 56.0b9 where we started to migrate users to win64 automatically these signatures account for >20% of all browser crashes.
ni ping to make you aware of the issue
Flags: needinfo?(cpeterson)
Track 56+ as top crash.
Here's the patch to block klsihk64.dll, try build at https://treeherder.mozilla.org/#/jobs?repo=try&revision=abbfc4a75aef647e80c0736986282d91a9f08729.

philipp is going to test it.
Flags: needinfo?(madperson)
yes, putting klsihk64.dll on our regular dll blocklist seems to be successful in getting it out of our process. that's a start :-)
Flags: needinfo?(madperson)
Attachment #8905447 - Flags: review?(aklotz)
the crash is on win8 - not sure how feasible it would be to implement a block just for this environment...
We reproduce defect in Kaspersky Lab on FF52x64 + KIS2018 under Win8x64. We still not sure about root cause, but we will find it in a days. I will update topic, when we get more details.

It’s a pity, that we did not get this bug year ago. Please use BrowserIssues@kaspersky.com, when you suspect Kaspersky products.

Alexey Totmakov
Head of Network Technologies Development
Kaspersky Lab
Attachment #8905447 - Flags: review?(aklotz) → review+
(In reply to [:philipp] from comment #21)
> the crash is on win8 - not sure how feasible it would be to implement a
> block just for this environment...

Pushed a couple of patches that should achieve that.  Philipp tested that klsihk64.dll is correctly blocked on win8 but not on win7.  Jim, do you have a preference on going this way vs blocking on all versions as in marco's patch?
Flags: needinfo?(jmathies)
We found root cause of the problem. Sure enough, it is reproducing only on Win8. We update all our products on next week. If you want, I can provide module with fix earlier, to make sure that fix work.
 
I think it is not necessary to blocklist klsihk64.dll right now, the problem one year old, and in the next week it should be gone.
Thanks Alexey. Yes, it will be great if you update next week. 
Can you send us the updated version with the fix? That will help us with the decision before the Firefox 56 release.
Attached file klsihk64.dll
fix from Kasperksy
Attached file klsihk.dll
fix from Kaspersky
Liz, i have attached two files, you should replace C:\ProgramData\Kaspersky Lab\AVP17.0.0\Bases\klsihk*.dll. 
Before it, it is necessary to disable Self Defense in product settings.
Alexy, thanks for the fast fix!

We don't need to block all versions of klsihk64.dll, but we might want to block the older, unfixed versions. I don't know how to search Socorro for all klsihk64.dll version numbers, but I did a random spot check of some crash reports and saw the following DLL version numbers:

10.0.0.1523
12.0.282.0
13.0.459.0
14.0.245.0
14.0.455.0
Crash Signature: klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b] [@ klsihk64.dll@0x183ff] [@ klsihk64.dll@0x3d71] [@ klsihk64.dll@0x3d7d] [@ klsihk64.dll@0x3e2d] → klsihk64.dll@0x5337] [@ klsihk64.dll@0x533b] [@ klsihk64.dll@0x5377] [@ klsihk64.dll@0x537b] [@ klsihk64.dll@0x183ff] [@ klsihk64.dll@0x3d71] [@ klsihk64.dll@0x3d7d] [@ klsihk64.dll@0x3e2d] [@ klsihk64.dll@0x1fec] [@ klsihk64.dll@0x2b3b]
Flags: needinfo?(vladimir.kukushkin)
Flags: needinfo?(toivo.peegel)
Flags: needinfo?(kurt.baumgartner)
Flags: needinfo?(jmathies)
Flags: needinfo?(denisov_vit)
Flags: needinfo?(cpeterson)
Flags: needinfo?(alexey.drozdov)
Summary: crash in klsihk64.dll@0x3ad7 → crash in klsihk64.dll@0x3ad7 (Kaspersky DLL)
dmajor or handyman, do you have time to confirm whether the attached DLL fixes this Kaspersky crash?
Flags: needinfo?(dmajor)
Flags: needinfo?(davidp99)
i can confirm that replacing the dll with the fixed version provided by kaspersky in comment #28 no longer causes crashes at startup for firefox 64bit versions on windows 8.

based on that, i think we could narrow the block from comment #24 to klsihk64.dll versions below but not including 14.0.557.0.
Flags: needinfo?(dmajor)
Flags: needinfo?(davidp99)
(In reply to Chris Peterson [:cpeterson] from comment #31)
> We don't need to block all versions of klsihk64.dll, but we might want to
> block the older, unfixed versions. 

Actually, it is updatable module for Kaspersky products, so all users of all product versions will receive it, when it will be on public. There is no reason to block any version.
(In reply to Alexey Totmakov from comment #34)
> (In reply to Chris Peterson [:cpeterson] from comment #31)
> > We don't need to block all versions of klsihk64.dll, but we might want to
> > block the older, unfixed versions. 
> 
> Actually, it is updatable module for Kaspersky products, so all users of all
> product versions will receive it, when it will be on public. There is no
> reason to block any version.

Hey Alexey, if we block the old version, the DLL will automatically be unblocked as soon as the user updates.
(In reply to Marco Castelluccio [:marco] from comment #35)

> Hey Alexey, if we block the old version, the DLL will automatically be
> unblocked as soon as the user updates.

I understand it :) If you already have blocking mechanism by module version, I do not see any problems to block older version of module. For me it is important, that 14.0.457.0+ version will not blocked.
Assignee: nobody → jcristau
i've tested a try build based on julien's patches and can confirm it's working as intended:
* win 8 with klsihk64.dll 14.0.455.0: no startup crash and the module doesn't hook into the process (regular nightly is crashing in this situation)
* win 8 with klsihk64.dll 14.0.457.0: no startup crash and klsihk64.dll is present
* win 7 with klsihk64.dll 14.0.455.0: no crash and klsihk64.dll is present
Attachment #8906007 - Flags: review?(jmathies)
Attachment #8906008 - Flags: review?(jmathies)
Attachment #8906582 - Flags: review?(jmathies)
Comment on attachment 8906007 [details]
Bug 1268470 - Part 1: Add BLOCK_WIN8_ONLY flag to dll blocklist

https://reviewboard.mozilla.org/r/177764/#review183368
Attachment #8906007 - Flags: review?(jmathies) → review+
Comment on attachment 8906008 [details]
Bug 1268470 - Part 2: Block klsihk64.dll on Windows 8

https://reviewboard.mozilla.org/r/177766/#review183370
Attachment #8906008 - Flags: review?(jmathies) → review+
Comment on attachment 8906582 [details]
Bug 1268470 - Part 3: Restrict klsihk64.dll block to versions < 14.0.457.0

https://reviewboard.mozilla.org/r/178326/#review183372
Attachment #8906582 - Flags: review?(jmathies) → review+
Attachment #8905447 - Attachment is obsolete: true
Keywords: checkin-needed
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/70acb4b69beb
Part 1: Add BLOCK_WIN8_ONLY flag to dll blocklist r=jimm
https://hg.mozilla.org/integration/autoland/rev/fda9e39cdb5d
Part 2: Block klsihk64.dll on Windows 8 r=jimm
https://hg.mozilla.org/integration/autoland/rev/91ebea077390
Part 3: Restrict klsihk64.dll block to versions < 14.0.457.0 r=jimm
Keywords: checkin-needed
Comment on attachment 8906007 [details]
Bug 1268470 - Part 1: Add BLOCK_WIN8_ONLY flag to dll blocklist

Probably too late for today's build, and this isn't in m-c yet, but seeing how it's flagged as blocking...  This is for all 3 patches.

Approval Request Comment
[Feature/Bug causing the regression]: the 64-bit migration made this a topcrash
[User impact if declined]: startup crashes for users of 64-bit firefox with Kaspersky third-party software
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: no but philipp verified on a try build (comment 38)
[Needs manual test from QE? If yes, steps to reproduce]: no
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: no
[Why is the change risky/not risky?]: blocks a dll that causes startup crashes, can't get any worse
[String changes made/needed]: none
Flags: needinfo?(lhenry)
Attachment #8906007 - Flags: approval-mozilla-beta?
Comment on attachment 8906007 [details]
Bug 1268470 - Part 1: Add BLOCK_WIN8_ONLY flag to dll blocklist

I'd like to take this for beta 11 if it lands cleanly
Flags: needinfo?(lhenry)
Attachment #8906007 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Alexey, can you share any more details about why klsihk64.dll is crashing in 64-bit Firefox? Is there a Firefox code change we could make to avoid the scenario that lead to the crash?

We started seeing more klsihk64.dll crashes after we upgraded 32-bit Firefox Beta users to 64-bit builds. The 64-bit upgrade procedure replaced the existing 32-bit Firefox binaries with 64-bit binaries in the existing "C:\Program Files (x86)" directory. Our testing didn't find any problems running 64-bit binaries from the 32-bit "C:\Program Files (x86)" directory, but perhaps 64-bit third-party software is surprised when 64-bit Firefox is in 32-bit "C:\Program Files (x86)" directory
Flags: needinfo?(alexey.totmakov)
Chris,
klsihk64.dll was incompatible with C++ runtime, shipped with Windows 8.0 x64. It is not depend of upgrade or clear installation of FF. In addition, it should not depend from browser, for crash it is enough Windows 8.0 x64 C++ runtime and any browser. Currently, we are investigate the behavior of other browsers.
Flags: needinfo?(alexey.totmakov)
We update klsihk64.dll for 100% users of 2016, 2018 Kaspersky products, and for 50% users of 2017. All 100% users of all our products will receive this update to 20 September.
Thanks, Alexey! That was fast. :)
[Why is this notable]:
Windows users running 64-bit Firefox and an old version of the Kaspersky security software will hit startup crashes. We plan to automatically migrate some 32-bit Firefox users to 64-bit during the 56 release, so some users may hit this crash but not know they are now running 64-bit Firefox.
 
[Affects Firefox for Android]:
No

[Suggested wording]:
Some versions of the Kaspersky security software can cause 64-bit Firefox startup crashes on Windows 8. Kaspersky has fixed the crash in the latest version of their software. To fix this crash, please install the latest Kaspersky update or [re-install 32-bit Firefox](https://www.mozilla.org/firefox/all/).

[Links (documentation, blog post, etc)]:
https://www.mozilla.org/firefox/all/
relnote-firefox: --- → ?
Liz, I think we should relnote this Kaspersky crash and resolution for Firefox 56.
Flags: needinfo?(lhenry)
Are we certain that reverting to 32-bits fixes this issue? Sure, the user will no longer see "klsihk64.dll" crashes after that, but I see plenty of crashes coming from "klsihk.dll" too.
Flags: needinfo?(cpeterson)
(In reply to David Major [:dmajor] from comment #53)
> Are we certain that reverting to 32-bits fixes this issue? Sure, the user
> will no longer see "klsihk64.dll" crashes after that, but I see plenty of
> crashes coming from "klsihk.dll" too.

That's a good question. I haven't confirmed that downgrading to 32-bit avoids the crashes, but it seems like a safe suggestion. If migrating a user from 32- to 64-bit causes enough crashes that the user seeks out information in the Firefox release notes, then they were probably have no or few crashes with 32-bit Firefox.

In theory, no one should hit this crash when we migrate users to 64-bit Firefox 56 because Kaspersky has fixed their code and we are blocking the old unfixed klsihk64.dll versions.
Flags: needinfo?(cpeterson)
i don't think there's a need to add this into the release notes - the blocklisting patch apparently has worked as these crashes have totally stopped in recent nightly and beta builds.
(In reply to [:philipp] from comment #55)
> i don't think there's a need to add this into the release notes - the
> blocklisting patch apparently has worked as these crashes have totally
> stopped in recent nightly and beta builds.

Looks like you are correct. Awesome! I was afraid that the crashes stopped because Beta users stopped using Firefox. However, we have seen klsihk64.dll crashes on Nightly for a long time and the crashes stopped after Nightly build id 20170911100210 [1]. The blocklist patch landed around September 12 (comment 46).

OTOH, I don't think there is much harm mentioning the crash in the release notes and that Kaspersky has already fixed it in their latest update. The blocklist prevents the crash but also disables Kaspersky's Firefox integration until the user updates to the latest version.

[1] https://crash-stats.mozilla.com/search/?signature=~klsihk64.dll&release_channel=nightly&product=Firefox&platform=Windows&date=%3E%3D2017-03-22T19%3A15%3A03.000Z&date=%3C2017-09-22T19%3A15%3A03.000Z&_sort=-date&_facets=signature&_facets=build_id&_columns=date&_columns=signature&_columns=version&_columns=build_id&_columns=platform#facet-build_id
Flags: needinfo?(lhenry)
I don't think there's anything actionable for ESR52 in this bug, but correct me if I'm wrong.
Added to the release notes with
"Some third party software (Comodo Internet Security, Kaspersky, Quick Heal Antivirus) are known to cause issues with Firefox 64-bit. These vendors have published new releases which addresses the issues."
as wording
You need to log in before you can comment on or make changes to this bug.