1794064 - Crash in [@ RtlpFreeHeapInternal | RtlFreeHeap | aswjsflt64.dll.t01]

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect, P1)

Description

[Ryan VanderMeulen [:RyanVM]]

We're seeing a big release crash spike with this signature. Looks like it's an Avast DLL causing the problem.

Crash report: https://crash-stats.mozilla.org/report/index/632c8c58-93d2-4dec-aa8e-819cf0221007

Reason: STATUS_HEAP_CORRUPTION

Top 9 frames of crashing thread:

0 ntdll.dll RtlReportFatalFailure 
1 ntdll.dll RtlReportCriticalFailure 
2 ntdll.dll RtlpHeapHandleError 
3 ntdll.dll RtlpHpHeapHandleError 
4 ntdll.dll RtlpLogHeapFailure 
5 ntdll.dll RtlpFreeHeapInternal 
6 ntdll.dll RtlFreeHeap 
7 aswJsFlt64.dll.t01 aswJsFlt64.dll.t01@0x0000000000057d5b 
8 aswJsFlt64.dll.t01 aswJsFlt64.dll.t01@0x000000000002a780 

Comment 1

[Gabriele Svelto [:gsvelto]]

I've inspected a bunch of crashes and it seems that versions prior and including 18.0.1473.0 are affected. I'm going to analyze how it is injected and add it to the block-list. Given the release channel is affected we might have to do a chem-spill as we can't wait for weeks for the fix to reach them.

Comment 2

[Gabriele Svelto [:gsvelto]]

Quick update: I'm trying to identify the way the module is being injected to be sure that we can block it. Version 18.0.1477.0 of the module seems available and we don't seem to have crashes on file caused by that version which suggests the vendor was already aware of the bug and addressed it.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to topcrash signatures, which match the following criteria:

• Top 20 desktop browser crashes on release
• Top 5 desktop browser crashes on Windows on release

For more information, please visit auto_nag documentation.

Comment 4

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1794064 - Block older crash-prone versions of Avast Antivirus r=gcp

Comment 5

[Gabriele Svelto [:gsvelto]]

Windows DLLBlocklist request form

1) How were we aware of the problem?

This bug tracks all the crash reports associated with the problem.

2) What is a suspicious product causing the problem?

Avast Antivirus

3) Is the product downloadable?  If so, do we have a local repro?

It can be downloaded and I've tested the latest version but I cannot reproduce
the crash.

4) Which OS versions does the problem occur on?

It affects all versions of Windows.

5) Which process types does the problem occur on?

Only the main process seems affected though the module is loaded in other
processes too.

6) What is the maximum version of the module in the crash reports?

18.0.1473.0

7) Is the issue fixed by a newer version of the product?

Version 18.0.1477.0 appears in the telemetry module pings but not in crashes,
suggesting that the problem was fixed by the vendor.

8) Do we have data about the module in the third-party-module ping?

Yes, the third party module ping shows the module being loaded in all processes
with the following stack:

0 firefox.exe!mozilla::freestanding::patched_LdrLoadDll(wchar_t*, unsigned long*, _UNICODE_STRING*, void**)+0x18e
1 aswhook+0x25c6
2 <unknown>+0xffffffff
3 <unknown>+0xffffffff
4 <unknown>+0xffffffff
5 KERNELBASE.dll!LoadLibraryExW+0xc6
6 firefox.exe!mozilla::GetBootstrap(char const*, mozilla::LibLoadingStrategy)+0x200
7 firefox.exe!InitXPCOMGlue(mozilla::LibLoadingStrategy)+0x8f
8 firefox.exe!wmain(int, wchar_t**)+0x687
9 firefox.exe!__scrt_common_main_seh()+0xfa
10 wkernel32!BaseThreadInitThunk+0x24
11 wntdll!_RtlUserThreadStart+0x2f
12 wntdll!_RtlUserThreadStart+0x1b

9) Do we know how the module is loaded?

Yes, it's being loaded via LoadLibraryExW() and can be blocked by our machinery.

10) Describe your conclusion.

We should block old versions of this module to shield users from crashes.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9832e48a30c5
Block older crash-prone versions of Avast Antivirus r=gcp

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9297571 [details]
Bug 1794064 - Block older crash-prone versions of Avast Antivirus r=gcp

Approved for 106.0rc1, 105.0.3, and 102.4esr.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/cb42fa8d0f66

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/5a2af3ed86a6

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/d6d9f8de34ab

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Added to the 105.0.3 release notes:

Mitigated frequent crashes for Windows users with Avast or AVG Antivirus software installed

Comment 12

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/9832e48a30c5

Comment 13

[Catalin Sasca, Desktop QA [:csasca]]

Checked that Firefox works as expected with the latest version of Avast Free (aswjsflt64.dll - 18.0.1479.0). We don't have access to older affected versions unfortunately. Tests were performed with Firefox 106.0 on Windows 10 & 11, while browsing several random websites and trying other scenarios as well.

Please let me know if I can help with anything here.

Comment 14

[Tom25519]

Firefox 105.0.3 seems random freezed when I surfing the Internet, but it won't happen after downgrade to 105.0.2.

Comment 15

[Gian-Carlo Pascutto [:gcp]]

Firefox 105.0.3 seems random freezed when I surfing the Internet, but it won't happen after downgrade to 105.0.2.

The (literally) only change is that it will block the crashing Avast DLL. Are you using Avast/AVG?

Comment 16

[Tom25519]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #15)

Firefox 105.0.3 seems random freezed when I surfing the Internet, but it won't happen after downgrade to 105.0.2.

The (literally) only change is that it will block the crashing Avast DLL. Are you using Avast/AVG?

No, I using ESET Internet Security 15.2.17.0, today I update Firefox to 105.0.3, this bug seems doesn't reproduce for now.

Comment 17

[Yannis Juglaret [:yannis]]

We have finished analyzing what happened with this incident. It involves named pipe IPC between a server that runs in the Firefox main process and its clients who run in child processes, in particular content processes.

The root cause of the incident is that the new protocol used to communicate on the named pipe in the newest versions of aswJsFlt.dll is incompatible with the protocol used in older versions, and this can trigger a server-side crash. The block from comment 5 was exactly what was required to mitigate this crash, and has likely allowed the vast majority of Firefox users to have their DLL updated without crashing. The server-side bug should now be patched in the latest versions of the DLL. If I understood correctly, Avast intends to patch the upcoming version of the DLL as well, so that updating to it will not crash older versions of the named pipe server anymore: new clients will not communicate with old version servers.

The scenario to reproduce the crash is as follows:

• Install an old version of Avast with version 18.0.1473.0 or older of aswJsFlt.dll.
• Start a Firefox version that is supported by your version of aswJsFlt.dll (in my case, 105.0.1 for 18.0.1473.0). A named pipe server thread will run in the main process using version 18.0.1473.0 or older of aswJsFlt.dll.
• Receive an Avast update in the background that changes the version of aswJsFlt.dll to something more recent than 18.0.1473.0.
• Open new tabs. The new child processes will run with the new version of the DLL and communicate on the named pipe.
• Visit a website which uses a JavaScript script with a size of at least 2KB in one of the new tabs.

Comment 18

[Yannis Juglaret [:yannis]]

This crash was a double-free memory corruption in the main process. It was possible to trigger it from sandboxed processes. Assuming exploitability, it could have let an attacker escape the sandbox. Avast has listed it under CVE-2022-4291 and Norton has published two security advisories: one listed as CVE-2022-4291, focused on the security vulnerability aspect, and one listed as NLOKSA1509, focused on the stability impact in Firefox.