Closed Bug 1794064 Opened 2 months ago Closed 2 months ago

Crash in [@ RtlpFreeHeapInternal | RtlFreeHeap | aswjsflt64.dll.t01]

Categories

(External Software Affecting Firefox :: Other, defect, P1)

Unspecified
Windows

Tracking

(relnote-firefox 105+, firefox-esr102106+ fixed, firefox105+ fixed, firefox106+ fixed, firefox107+ fixed)

RESOLVED FIXED
Tracking Status
relnote-firefox --- 105+
firefox-esr102 106+ fixed
firefox105 + fixed
firefox106 + fixed
firefox107 + fixed

People

(Reporter: RyanVM, Assigned: gsvelto)

References

(Blocks 1 open bug)

Details

(Keywords: crash, topcrash)

Crash Data

Attachments

(1 file)

We're seeing a big release crash spike with this signature. Looks like it's an Avast DLL causing the problem.

Crash report: https://crash-stats.mozilla.org/report/index/632c8c58-93d2-4dec-aa8e-819cf0221007

Reason: STATUS_HEAP_CORRUPTION

Top 9 frames of crashing thread:

0 ntdll.dll RtlReportFatalFailure 
1 ntdll.dll RtlReportCriticalFailure 
2 ntdll.dll RtlpHeapHandleError 
3 ntdll.dll RtlpHpHeapHandleError 
4 ntdll.dll RtlpLogHeapFailure 
5 ntdll.dll RtlpFreeHeapInternal 
6 ntdll.dll RtlFreeHeap 
7 aswJsFlt64.dll.t01 aswJsFlt64.dll.t01@0x0000000000057d5b 
8 aswJsFlt64.dll.t01 aswJsFlt64.dll.t01@0x000000000002a780 
Flags: needinfo?(salomon)
Flags: needinfo?(rypacek)

I've inspected a bunch of crashes and it seems that versions prior and including 18.0.1473.0 are affected. I'm going to analyze how it is injected and add it to the block-list. Given the release channel is affected we might have to do a chem-spill as we can't wait for weeks for the fix to reach them.

Quick update: I'm trying to identify the way the module is being injected to be sure that we can block it. Version 18.0.1477.0 of the module seems available and we don't seem to have crashes on file caused by that version which suggests the vendor was already aware of the bug and addressed it.

The bug is linked to topcrash signatures, which match the following criteria:

  • Top 20 desktop browser crashes on release
  • Top 5 desktop browser crashes on Windows on release

For more information, please visit auto_nag documentation.

Keywords: topcrash
Assignee: nobody → gsvelto
Status: NEW → ASSIGNED
Windows DLLBlocklist request form

1) How were we aware of the problem?

This bug tracks all the crash reports associated with the problem.

2) What is a suspicious product causing the problem?

Avast Antivirus

3) Is the product downloadable?  If so, do we have a local repro?

It can be downloaded and I've tested the latest version but I cannot reproduce
the crash.

4) Which OS versions does the problem occur on?

It affects all versions of Windows.

5) Which process types does the problem occur on?

Only the main process seems affected though the module is loaded in other
processes too.

6) What is the maximum version of the module in the crash reports?

18.0.1473.0

7) Is the issue fixed by a newer version of the product?

Version 18.0.1477.0 appears in the telemetry module pings but not in crashes,
suggesting that the problem was fixed by the vendor.

8) Do we have data about the module in the third-party-module ping?

Yes, the third party module ping shows the module being loaded in all processes
with the following stack:

0 firefox.exe!mozilla::freestanding::patched_LdrLoadDll(wchar_t*, unsigned long*, _UNICODE_STRING*, void**)+0x18e
1 aswhook+0x25c6
2 <unknown>+0xffffffff
3 <unknown>+0xffffffff
4 <unknown>+0xffffffff
5 KERNELBASE.dll!LoadLibraryExW+0xc6
6 firefox.exe!mozilla::GetBootstrap(char const*, mozilla::LibLoadingStrategy)+0x200
7 firefox.exe!InitXPCOMGlue(mozilla::LibLoadingStrategy)+0x8f
8 firefox.exe!wmain(int, wchar_t**)+0x687
9 firefox.exe!__scrt_common_main_seh()+0xfa
10 wkernel32!BaseThreadInitThunk+0x24
11 wntdll!_RtlUserThreadStart+0x2f
12 wntdll!_RtlUserThreadStart+0x1b

9) Do we know how the module is loaded?

Yes, it's being loaded via LoadLibraryExW() and can be blocked by our machinery.

10) Describe your conclusion.

We should block old versions of this module to shield users from crashes.
Pushed by gpascutto@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9832e48a30c5
Block older crash-prone versions of Avast Antivirus r=gcp

Comment on attachment 9297571 [details]
Bug 1794064 - Block older crash-prone versions of Avast Antivirus r=gcp

Approved for 106.0rc1, 105.0.3, and 102.4esr.

Attachment #9297571 - Flags: approval-mozilla-release+
Attachment #9297571 - Flags: approval-mozilla-esr102+
Attachment #9297571 - Flags: approval-mozilla-beta+
See Also: → 1790160

Added to the 105.0.3 release notes:

Mitigated frequent crashes for Windows users with Avast or AVG Antivirus software installed

Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Crash Signature: | aswjsflt.dll.t01 | BaseThreadInitThunk ] [@ RtlpReportHeapFailure | MD4Transform ] [@ RtlpFreeHeapInternal | RtlFreeHeap ] → | aswjsflt.dll.t01 | BaseThreadInitThunk ] [@ RtlpReportHeapFailure | MD4Transform ] [@ RtlpFreeHeapInternal | RtlFreeHeap ] [@ RtlpFreeHeapInternal | RtlFreeHeap | aswjsflt64.dll.t02 ] [@ RtlpReportHeapFailure | RtlUlonglongByteSwap | HeapFree | Bas…
Crash Signature: BaseThreadInitThunk ] [@ RtlpFreeHeapInternal | RtlFreeHeap | LdrpFreeTls ] → BaseThreadInitThunk ] [@ RtlpFreeHeapInternal | RtlFreeHeap | LdrpFreeTls ] [@ RtlpFreeHeapInternal | RtlFreeHeap | LocalFree ] [@ RtlpFreeHeapInternal | RtlFreeHeap | WSPSelect ]
Crash Signature: [@ RtlpFreeHeapInternal | RtlFreeHeap | aswjsflt64.dll.t01] [@ RtlpReportHeapFailure | RtlFreeHeap | HeapFree] [@ RtlFreeHeap ] [@ RtlpReportHeapFailure | zzz_AsmCodeRange_End | HeapFree ] [@ RtlpReportHeapFailure | RtlpFreeHeapInternal | RtlFreeHeap … → [@ RtlpFreeHeapInternal | RtlFreeHeap | aswjsflt64.dll.t01] [@ RtlpFreeHeapInternal | RtlFreeHeap | aswjsflt64.dll.t02] [@ RtlpFreeHeapInternal | RtlFreeHeap | aswjsflt64.dll.t03] [@ RtlpReportHeapFailure | RtlFreeHeap | HeapFree] [@ RtlFreeHeap ] [@…

Checked that Firefox works as expected with the latest version of Avast Free (aswjsflt64.dll - 18.0.1479.0). We don't have access to older affected versions unfortunately. Tests were performed with Firefox 106.0 on Windows 10 & 11, while browsing several random websites and trying other scenarios as well.

Please let me know if I can help with anything here.

Flags: needinfo?(Tom25519)
Blocks: 1795864
No longer blocks: 1795864
See Also: → 1795864
Flags: needinfo?(salomon)
Flags: needinfo?(rypacek)
Blocks: 1797269

We have finished analyzing what happened with this incident. It involves named pipe IPC between a server that runs in the Firefox main process and its clients who run in child processes, in particular content processes.

The root cause of the incident is that the new protocol used to communicate on the named pipe in the newest versions of aswJsFlt.dll is incompatible with the protocol used in older versions, and this can trigger a server-side crash. The block from comment 5 was exactly what was required to mitigate this crash, and has likely allowed the vast majority of Firefox users to have their DLL updated without crashing. The server-side bug should now be patched in the latest versions of the DLL. If I understood correctly, Avast intends to patch the upcoming version of the DLL as well, so that updating to it will not crash older versions of the named pipe server anymore: new clients will not communicate with old version servers.

The scenario to reproduce the crash is as follows:

  • Install an old version of Avast with version 18.0.1473.0 or older of aswJsFlt.dll.
  • Start a Firefox version that is supported by your version of aswJsFlt.dll (in my case, 105.0.1 for 18.0.1473.0). A named pipe server thread will run in the main process using version 18.0.1473.0 or older of aswJsFlt.dll.
  • Receive an Avast update in the background that changes the version of aswJsFlt.dll to something more recent than 18.0.1473.0.
  • Open new tabs. The new child processes will run with the new version of the DLL and communicate on the named pipe.
  • Visit a website which uses a JavaScript script with a size of at least 2KB in one of the new tabs.

This crash was a double-free memory corruption in the main process. It was possible to trigger it from sandboxed processes. Assuming exploitability, it could have let an attacker escape the sandbox. Avast has listed it under CVE-2022-4291 and Norton has published two security advisories: one listed as CVE-2022-4291, focused on the security vulnerability aspect, and one listed as NLOKSA1509, focused on the stability impact in Firefox.

You need to log in before you can comment on or make changes to this bug.