1777960 - Crash in [@ TF_Notify] - Crash with ZoneAlarm Anti-Keylogger on Windows 11 22H2 when starting to type

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect)

Description

[Gabriele Svelto [:gsvelto] (PTO)]

Crash report: https://crash-stats.mozilla.org/report/index/5a1cb5e2-adb7-480c-b31b-e59890220703

Reason: EXCEPTION_ACCESS_VIOLATION_READ

Top 10 frames of crashing thread:

0 msctf.dll TF_Notify 
1 user32.dll int64_t CtfHookProcWorker 
2 user32.dll uint64_t CallHookWithSEH 
3 user32.dll _fnHkINLPCHARHOOKSTRUCT 
4 ntdll.dll KiUserCallbackDispatch 
5 win32u.dll NtUserPeekMessage 
6 eoppbrowser.dll eoppbrowser.dll@0x000000000000c0f1 
7 xul.dll profiler_thread_wake tools/profiler/core/platform.cpp:6573
8 xul.dll static mozilla::widget::WinUtils::WaitForMessage widget/windows/WinUtils.cpp:860
9 xul.dll nsAppShell::ProcessNextNativeEvent widget/windows/nsAppShell.cpp:707

This is happening with eOppBrowser.dll version 1.0.67.0 or 1.0.68.0 on the stack.

Comment 1

[Yannis Juglaret [:yannis]]

This crash signature currently has rather high volume. It has evolved compared to the initial description. Currently, all reports are from Windows 11 22H2 10.0.22621 users (or above: 10.0.22631), using ZoneAlarm Anti-Keylogger. The crash occurs because of a DLL injected by the product: icsak.dll.

New example crash report: here

New example crashing stack (partially incorrect because icsak.dll is unavailable):

0 	msctf.dll 	TF_Notify
1 	icsak.dll 	icsak.dll@0x3a17
2 	icsak.dll 	icsak.dll@0x5df27
3 	icsak.dll 	icsak.dll@0x8cdf7
4 	icsak.dll 	icsak.dll@0x8ce97
5 	icsak.dll 	icsak.dll@0x8cdf7
6 	user32.dll 	CtfHookProcWorker(int, uint64_t, int64_t, uint64_t) 		scan
7 	user32.dll 	CallHookWithSEH(_GENERICHOOKHEADER*, void*, unsigned long*, uint64_t) 		cfi
8 	user32.dll 	_fnHkINLPCHARHOOKSTRUCT 		cfi
9 	ntdll.dll 	KiUserCallbackDispatch 		cfi
10 	win32u.dll 	ZwUserPeekMessage 		cfi
11 	icsak.dll 	icsak.dll@0x50b3 		cfi
12 	user32.dll 	_PeekMessage(tagMSG*, HWND__*, unsigned int, unsigned int, unsigned int, unsigned int, int) 		scan
13 	user32.dll 	PeekMessageW 		cfi
14 	icsak.dll 	icsak.dll@0x4e3c 		cfi
15 	user32.dll 	PeekMessageW 		scan
16 	icsak.dll 	icsak.dll@0x4a20 		cfi
17 	icsak.dll 	icsak.dll@0x8ccff 		scan
18 	icsak.dll 	icsak.dll@0x8cdf7 		scan
19 	icsak.dll 	icsak.dll@0x8ce07 		scan
20 	icsak.dll 	icsak.dll@0x4fdc 		scan
21 	xul.dll 	mozilla::widget::WinUtils::WaitForMessage(unsigned long) 	widget/windows/WinUtils.cpp:509 	scan
22 	xul.dll 	mozilla::widget::WinUtils::PeekMessage(tagMSG*, HWND__*, unsigned int, unsigned int, unsigned int) 	widget/windows/WinUtils.cpp:419 	inlined
22 	xul.dll 	nsAppShell::ProcessNextNativeEvent(bool) 	widget/windows/nsAppShell.cpp:717 	cfi
23 	xul.dll 	nsBaseAppShell::DoProcessNextNativeEvent(bool) 	widget/nsBaseAppShell.cpp:131 	inlined
23 	xul.dll 	nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal*, bool) 	widget/nsBaseAppShell.cpp:267 	inlined
23 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1154 	inlined
23 	xul.dll 	NS_ProcessNextEvent(nsIThread*, bool) 	xpcom/threads/nsThreadUtils.cpp:479 	cfi
24 	xul.dll 	mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp:107 	cfi
25 	xul.dll 	MessageLoop::RunInternal() 	ipc/chromium/src/base/message_loop.cc:368 	inlined
25 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc:361 	cfi
26 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc:343 	cfi
27 	xul.dll 	nsBaseAppShell::Run() 	widget/nsBaseAppShell.cpp:148 	cfi
28 	xul.dll 	nsAppShell::Run() 	widget/windows/nsAppShell.cpp:615 	cfi
29 	xul.dll 	nsAppStartup::Run() 	toolkit/components/startup/nsAppStartup.cpp:295 	cfi
30 	xul.dll 	XREMain::XRE_mainRun() 	toolkit/xre/nsAppRunner.cpp:5659 	cfi
31 	xul.dll 	XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) 	toolkit/xre/nsAppRunner.cpp:5859 	cfi
32 	xul.dll 	XRE_main(int, char**, mozilla::BootstrapConfig const&) 	toolkit/xre/nsAppRunner.cpp:5915 	cfi
33 	firefox.exe 	do_main(int, char**, char**) 	browser/app/nsBrowserApp.cpp:227 	inlined
33 	firefox.exe 	NS_internal_main(int, char**, char**) 	browser/app/nsBrowserApp.cpp:445 	inlined
33 	firefox.exe 	wmain(int, wchar_t**) 	toolkit/xre/nsWindowsWMain.cpp:167 	cfi
34 	firefox.exe 	invoke_main() 	/builds/worker/workspace/obj-build/browser/app/D:/a/_work/1/s/src/vctools/crt/vcstartup/src/startup/exe_common.inl:90 	inlined
34 	firefox.exe 	__scrt_common_main_seh() 	/builds/worker/workspace/obj-build/browser/app/D:/a/_work/1/s/src/vctools/crt/vcstartup/src/startup/exe_common.inl:288 	cfi
35 	kernel32.dll 	BaseThreadInitThunk 		cfi
36 	ntdll.dll 	RtlUserThreadStart 

User comments suggest that the crash happens with other browsers as well (possibly Opera and Chrome, but not Edge and Brave?), and that it occurs after they start to type a message, e.g. when sending a message on a Facebook Messenger or when searching on DuckDuckGo.

icsak.dll is part of a feature of ZoneAlarm called Anti-Keylogger which requires a premium account and presumably a manual activation.

We should block the current versions of icsak.dll and ping Check Point Software about this crash so that they fix it. Most likely, they don't adapt well to something new with Windows 11 22H2.

NI [:gstoll] to be sure this gets proper attention.

Comment 2

[Greg Stoll :gstoll]

1. How were we aware of the problem?
   Found via topcrasher

2. What is a suspicious product causing the problem?
   ZoneAlarm Anti-Keylogger, published by Check Point Software Technologies Ltd.

3. Is the product downloadable? If so, do we have a local repro?
   It is downloadable but requires a premium account to use, so we haven't been able to reproduce.

4. Which OS versions does the problem occur on?
   Seems to only happen on Windows 11 22H2, but we don't have a good way to only block on this OS.

5. Which process types does the problem occur on?
   parent process only

6. What is the maximum version of the module in the crash reports?
   1.5.393.2181

7. Is the issue fixed by a newer version of the product?
   Telemetry does not show any newer versions.

8. Do we have data about the module in the third-party-module ping?
   We do have a little bit of data, but seemingly only for a very old version of Firefox (78.9) on Windows 7, which we no longer support. (the version of icsak.dll is also an older one) I suspect all newer versions of icsak.dll are crashing on launch (or shortly after launch?), so we're not getting third-party-module ping data from them.

9. Do we know how the module is loaded?
   With the limited data, I see this call stack:
   0 firefox+0xeb2b
   1 <unknown>+0xffffffff
   2 ntdll.dll!MD4Transform+0xb3d
   3 ntdll.dll!WinSqmEndSession+0x3d4
   4 ntdll.dll!RtlQueryImageFileKeyOption+0x1d3
   5 ntdll.dll!RtlQueryImageFileKeyOption+0xd5
   6 ntdll.dll!LdrInitializeThunk+0x10

I'm not sure what exactly this is but it doesn't look like known bad cases we've seen before.

10. Describe your conclusion.
    We should block icsak.dll versions 1.5.393.2181 and earlier in the parent process.

Comment 3

[Greg Stoll :gstoll]

Attachment: Bug 1777960 - block ZoneAlarm Anti-Keylogger r=yjuglaret

Comment 4

[Yannis Juglaret [:yannis]]

Note for ZoneAlarm devs: We have these crash reports only with versions 1.5.393.2163 and 1.5.393.2181 of icsak.dll as far as I can tell.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 20 desktop browser crashes on beta

For more information, please visit BugBot documentation.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3e34a525b26a
block ZoneAlarm Anti-Keylogger r=yjuglaret

Comment 7

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/3e34a525b26a

Comment 8

[Yannis Juglaret [:yannis]]

Comment on attachment 9345412 [details]
Bug 1777960 - block ZoneAlarm Anti-Keylogger r=yjuglaret

Beta/Release Uplift Approval Request

• User impact if declined: This is a 115.0.2 top crasher, also a 116 beta top crasher. The impacted users crash when they start typing text, which is clearly not ideal for web browsing.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This only blocks the faulty DLL from injecting into Firefox.
• String changes made/needed:
• Is Android affected?: No

Comment 9

[Yannis Juglaret [:yannis]]

This is an issue that only seems to affect the most up-to-date users of Windows 11. Although it should be affected, we did not receive any report on the ESR channel, probably because it's unlikely for these users to not want to use the most up-to-date version of Firefox as well. So, it's probably best to not port the patch to ESR, so that ESR users still benefit from the ZoneAlarm Anti-Keylogger feature.

Comment 10

[Dianna Smith [:diannaS]]

Comment on attachment 9345412 [details]
Bug 1777960 - block ZoneAlarm Anti-Keylogger r=yjuglaret

Switching flags because we are no longer in beta

Approved for 116.0rc2

Comment 11

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/091efe4fd76e

Comment 12

[Greg Stoll :gstoll]

I have reproduced this crash and have a minidump I can share if it would be helpful.

Comment 13

[Greg Stoll :gstoll]

As we're backing the fix out because of the regressor bug 1847033, reopening this.

Comment 14

[Dianna Smith [:diannaS]]

Affected in nightly as of 2023-08-04 and 117.0b4

Comment 15

[Dianna Smith [:diannaS]]

Affected in Release when we rollout 116.0.2

Comment 16

[BugBot [:suhaib / :marco/ :calixte]]

Based on the topcrash criteria, the crash signature linked to this bug is not a topcrash signature anymore.

For more information, please visit BugBot documentation.

Comment 17

[Yannis Juglaret [:yannis]]

The current volume is correlated to msctf.dll 10.0.22621.1928 and with [:gstoll] we think we have found the root cause. We have forwarded the details to the vendor. I'm adding the signature for msctf.dll 10.0.22621.2215, with same root cause.

Comment 18

[Yannis Juglaret [:yannis]]

Adding the low-volume signatures for x86. Here is the full list of signatures on which we receive variations of this crash currently:

TF_Notify: msctf.dll 10.0.22621.1928 x64;
msctf.dll | icsak.dll | RtlpFreeHeapInternal | free | icsak.dll | CtfHookProcWorker: msctf.dll 10.0.22621.2215 x64;
IMProcessKeyInput: msctf.dll 10.0.22621.1928 x86;
msctf.dll | icsak.dll | CtfHookProcWorker: msctf.dll 10.0.22621.2215 x86.

For the record, what we described to the vendor with [:gstoll] is the following. We believe that this crash occurs because ZoneAlarm Anti-Keylogger hooks TF_Notify(UINT message, WPARAM wParam, LPARAM lParam) from msctf.dll to alter the Microsoft-internal message 0x20000, but starting with msctf.dll 10.0.22621.1928 Microsoft has changed the way the lParam should be used for that message. It was a scalar value before and is now a pointer to data instead. ZoneAlarm always uses the old convention and thus forwards a scalar value, which is dereferenced by msctf.dll, causing the crash.

Comment 19

[Yannis Juglaret [:yannis]]

In the absence of any improvement over the course of the past month, I will be submitting an ad-hoc patch to mitigate this issue.

Comment 20

[Yannis Juglaret [:yannis]]

Attachment: Bug 1777960 - Patch msctf.dll to prevent a crash with ZoneAlarm Anti-Keylogger. r=gstoll

Microsoft has made internal changes in msctf.dll starting with version
10.0.22621.1928. The TF_Notify function uses a new convention for
one of its arguments. These changes are incompatible with current
versions of ZoneAlarm Anti-Keylogger, resulting in crashes in our main
process.

This patch converts messages forwarded by ZoneAlarm Anti-Keylogger to
the new convention. If we detect the product and an incompatible version
of msctf.dll, then we hook TF_Notify and detect any message using the
old convention, and convert it to the new convention.

Comment 21

[Yannis Juglaret [:yannis]]

Some updates on this:

• The change in msctf.dll seems to actually date back to the very first versions of Windows 11 22H2 (even preview versions), e.g. here we have a crash with version 10.0.22621.755. I have confirmed this by analyzing the DLL.
• ZoneAlarm Anti-Keylogger (icsak.dll) is responsible for ~84% of the crash volume here, ESET (eoppbrowser.dll, bug 1846512) is responsible for ~1%, and Citrix Systems (epclient64.dll, no bug yet) for ~14%. The symptoms are very similar for all three: same crashing instruction, also with Windows 11 22H2. I will share the information we have about this change in msctf.dll with ESET and Citrix Systems and report about that in bug 1846512 (ESET) and a new bug for Citrix Systems.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c8c454aa97ec
Patch msctf.dll to prevent a crash with ZoneAlarm Anti-Keylogger. r=gstoll,win-reviewers

Comment 23

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/c8c454aa97ec

Comment 24

[BugBot [:suhaib / :marco/ :calixte]]

Since nightly and release are affected, beta will likely be affected too.
For more information, please visit BugBot documentation.

Comment 25

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:gstoll, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval. Also, don't forget to request an uplift for the patches in the regression caused by this fix.
• If no, please set status-firefox119 to wontfix.

For more information, please visit BugBot documentation.

Comment 26

[Yannis Juglaret [:yannis]]

Steps to reproduce:

Requires a Windows 11 22H2 machine (i.e. an up-to-date Windows 11 machine). To check:

• Open C:\Windows\System32\ directory.
• Right click on msctf.dll, click Properties, click Details.
• File version should show 10.0.22621.XXXX (for example: 10.0.22621.2361).

Requires ZoneAlarm with the Anti-Keylogger feature, here is how to activate it:

• Click in the WEB & PRIVACY panel of the ZoneAlarm software client.
• Move the Anti-Keylogger slider to ON position.
• Click OK.
• Restart your web browser.

Reproducing the issue:

• Start Firefox.
• Click on the awesome bar and type a few words of text.

What happens:

• The browser crashes.

Expected behavior:

• The words that were typed appear and Firefox can be used normally.

Note: If different words appear than the ones that were typed, we do not have the expected behavior, even if we don't crash. Please report.

Comment 27

[Yannis Juglaret [:yannis]]

Comment on attachment 9355606 [details]
Bug 1777960 - Patch msctf.dll to prevent a crash with ZoneAlarm Anti-Keylogger. r=gstoll

Beta/Release Uplift Approval Request

• User impact if declined: Users on Windows 11 22H2 with ZoneAlarm's Anti-Keylogger feature activated will continue to crash as soon as they type.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Follow steps in comment 26 in Firefox 118.0.1 64-bit and 32-bit for Windows, and confirm that the browser crashes. Then follow the same steps with patched builds, and confirm normal behavior. Please try at least 5 times per build to confirm the absence of crashes (see comment 28).
• List of other uplifts needed: None
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): If we discover a problem with this patch, it will affect at most all Windows 11 22H2 users of ZoneAlarm. This patch will not alter behavior for users that don't run with icsak.dll (ZoneAlarm's Anti-Keylogger DLL) or don't run on Windows 11 22H2. We also checked the compatibility of the patch with ZoneAlarm's hooking and everything seems fine. Note that according to [:gstoll]'s tests, ZoneAlarm users should have icsak.dll in their Firefox process even when they are not using the anti-keylogger feature, so behavior will be altered for them as well, though not in a way that should break anything.
• String changes made/needed:
• Is Android affected?: No

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: High impact for the low number of affected users on esr channel.
• User impact if declined: Users on Windows 11 22H2 with ZoneAlarm's Anti-Keylogger feature activated will continue to crash as soon as they type.
• Fix Landed on Version: 120
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch specifically targets users that are impacted by the crash. This patch will not alter behavior for users that don't run with icsak.dll (ZoneAlarm's Anti-Keylogger DLL) or don't run on Windows 11 22H2. We also checked the compatibility of our patch with ZoneAlarm's hooking and everything seems fine. By the time the patch reaches ESR it will presumably have already spent some time in Beta and/or Release.

Comment 28

[Greg Stoll :gstoll]

Note that the Steps to Reproduce don't crash Firefox every time - on my machine it's more like 1 in 2 times I try I get the crash. (and if you don't get the crash right away, quit Firefox and try again)

Comment 29

[Yannis Juglaret [:yannis]]

Comment on attachment 9355606 [details]
Bug 1777960 - Patch msctf.dll to prevent a crash with ZoneAlarm Anti-Keylogger. r=gstoll

ZoneAlarm plans to release an update that should fix this issue on Oct 9. Since our next planned dot release is on Oct 10, I think it is better to trust them to handle the issue for this release. Given such close dates, if we uplift to release and a new issue appears, it would be hard to know if it's caused by their update, by our patch, or by the mixing of both. If we later notice that their update doesn't fix the crash, we can consider uplifting the patch to beta.

Comment 30

[Yannis Juglaret [:yannis]]

(These signatures were probably not related in second thought)

Comment 31

[Yannis Juglaret [:yannis]]

We are currently receiving crashes for the following versions of icask.dll:

• 1.5.393.2181 for x64 (icsak.dll/a8894590237a48b3ad6030ebdadea76a1 when aggregating over modules in stack);
• 1.5.393.2163 for x64 (icsak.dll/bba941bab537409ab521393ab401910b1);
• 1.5.393.2181 for x86 (icsak.dll/3f9ccc819c4941a49f1a31e2540d8c721);
• 1.5.393.2163 for x86 (icsak.dll/20dc63e38988471bba7d05c295c7a7231).

Hopefully the crash volume should start to go down in the next few days and we should not see a new entry for icsak.dll appear when aggregating over modules in stack.

Comment 32

[Yannis Juglaret [:yannis]]

(In reply to Yannis Juglaret [:yannis] from comment #31)

Hopefully the crash volume should start to go down in the next few days and we should not see a new entry for icsak.dll appear when aggregating over modules in stack.

This is what we are observing now, so everything looks fine at last!

Comment 33

[Yannis Juglaret [:yannis]]

Attachment: Bug 1777960 - Restrict the ZoneAlarm icsak.dll patch to buggy versions. r=gstoll

Now that we know that 1.5.393.2181 is the final version of icsak.dll
from ZoneAlarm that causes the crash from bug 1777960, we can adapt our
patch to make sure that it does not alter behavior for more recent
versions of icsak.dll, where the issue has been fixed. In a few months,
when everybody has updated, we can remove the patch altogether.

Comment 34

[Phabricator Automation]

Comment on attachment 9359141 [details]
Bug 1777960 - Restrict the ZoneAlarm icsak.dll patch to buggy versions. r=gstoll

Revision D191329 was moved to bug 1860013. Setting attachment 9359141 [details] to obsolete.