1752733 - Crash caused by Sophos AV software in [@ hmpalert.dll | NS_FaultTolerantHeap::APIHook_RtlAllocateHeap]

Status: NEW

(External Software Affecting Firefox :: Other, defect)

Description

[Gabriele Svelto [:gsvelto]]

Crash report: https://crash-stats.mozilla.org/report/index/9247ceaa-33d5-4b6e-b21b-b13c10220130

Reason: Unhandled C++ Exception

Top 10 frames of crashing thread:

0 kernelbase.dll RaiseException 
1 hmpalert.dll hmpalert.dll@0x000000000009562f 
2 aclayers.dll void* NS_FaultTolerantHeap::APIHook_RtlAllocateHeap 
3 hmpalert.dll hmpalert.dll@0x000000000000d1bc 
4 hmpalert.dll hmpalert.dll@0x00000000000b65bf 
5 hmpalert.dll hmpalert.dll@0x00000000000d0c67 
6 hmpalert.dll hmpalert.dll@0x000000000000ca4b 
7 aclayers.dll void* NS_FaultTolerantHeap::APIHook_RtlAllocateHeap 
8 hmpalert.dll hmpalert.dll@0x0000000000030c5a 
9 ntdll.dll LdrpInitialize 

I added only the top 10 signatures but there's a lot more.

Comment 1

[Gabriele Svelto [:gsvelto]]

The volume is increasing and new signatures are popping up all the time.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Hi Yaniv, could Sophos look into these Firefox crashes for which Sophos' hmpalert.dll is part of the crashing thread, version 3.8.3.808 for bp-50b3960a-7d2b-424a-8e4d-324620220208.

List of crashes with hmpalert.dll in the first 15 frames of the crashing thread.

Comment 3

[Chloe Bell]

Hi,

I work for Sophos. Do you have any more detailed description of what steps you are following that trigger these crashes?

Thanks,

Chloe

Comment 4

[Gabriele Svelto [:gsvelto]]

Looking at the comments from the crash report these seem to happen at random but rather frequently for the affected users. Under the most common signature it seems that a C++ exception is being raised from the injected code (see this crash report). This seems to be happening in the "COM MTA" thread (see here) within hooks inserted by the module. I'm not familiar with that code but maybe some of my colleagues could help.

Comment 5

[Chloe Bell]

Hi Gabriele,
Is there any way I can have access to some of the actual crash dumps as well please?
Thanks

Comment 6

[Gian-Carlo Pascutto [:gcp]]

Hi Chloe, we can't share crash dumps because they might contain sensitive user data, but we can provide you with enough information to reconstruct the call stack in your DLL. Toshihito will help you with this.

Comment 7

[Toshihito Kikuchi [:toshi]]

(In reply to Gabriele Svelto [:gsvelto] from comment #4)

Looking at the comments from the crash report these seem to happen at random but rather frequently for the affected users. Under the most common signature it seems that a C++ exception is being raised from the injected code (see this crash report). This seems to be happening in the "COM MTA" thread (see here) within hooks inserted by the module. I'm not familiar with that code but maybe some of my colleagues could help.

I looked at the crash of 30322557-8a99-41d9-90a3-6f1570220209 you saw, but the crashing thread had nothing to do with COM MTA.

I analyzed two dumps and attached my debug notes including the callstack of the crashing thread. Chloe, can you take a look and share it with engineers. If you need other specific information, please let me know. Please note that we have minidumps, not fulldumps, so some information (TLS, Heap, etc.) are not available.

• 30322557-8a99-41d9-90a3-6f1570220209
  OS: 10.0.19041.1469 / hmpalert.dll: 3.8.3.808

• 1b824757-3ae2-4235-b546-0a7b30220208
  OS: 10.0.18362.815 / hmpalert.dll: 3.8.3.808

I could set up Free Trial version of Sophos (with hmpalert.dll v3.8.3.808) and ran Firefox with Win7 mode, but this crash did not happen. Here are a couple of things I noticed.

• hmpalert.dll explicitly calls RaiseException with 0E06D7363h.
  I don't know what condition triggered this, but basically injected modules should not do this.

• Having AcLayers!NS_FaultTolerantHeap::APIHook_RtlAllocateHeap in the stack (and it appears in the crash signatures) has nothing to do with the exception. This happens because hmpalert+0xd980 called HeapAlloc, which returned normally. It means, however, the process ran with compatibility mode. On Win10, a process does not usually load AcLayers.dll. So running Firefox with compat mode is likely to be one of the factors.

Comment 8

[Toshihito Kikuchi [:toshi]]

Attachment: analysis1-30322557-8a99-41d9-90a3-6f1570220209

Comment 9

[Toshihito Kikuchi [:toshi]]

Attachment: analysis2-1b824757-3ae2-4235-b546-0a7b30220208

Comment 10

[Gabriele Svelto [:gsvelto]]

(In reply to Toshihito Kikuchi [:toshi] from comment #7)

I looked at the crash of 30322557-8a99-41d9-90a3-6f1570220209 you saw, but the crashing thread had nothing to do with COM MTA.

You're right, though I swear I opened several crashes and they were all in the "COM MTA" thread. Can't find them anymore now though, maybe I mixed it up with some other crash.

Comment 11

[Chloe Bell]

Hi Toshihito,

Thank you for providing these analyses.

We believe the issue is a memory leak that occurs when running in compatibility mode (so the aclayers.dll is loaded). This heap has a fixed size and so it runs out of memory and causes the browser to crash.

We have not been able to reproduce the issue but do have a customer who has reported is against other browsers so it does not appear to be specific to Firefox.

We are working to identify a fix and will release this as a hotfix for customers to use.

Thanks

Comment 12

[Toshihito Kikuchi [:toshi]]

Thank you for the updates. It's great news that Sophos engineers are actively working on the case! Please let us know if you need anything from us to help your investigation.

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:haik, since the bug has high severity and recent activity, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 14

[Chloe Bell]

Hi all,

Sophos has released a fix for this issue and it is included in the latest Intercept X Cumulative Hotfix, see https://support.sophos.com/support/s/article/KB-000038477?language=en_US

This will also be rolled out to all customers in the next release.

Thanks,

Chloe

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

Since the crash volume is low (less than 5 per week), the severity is downgraded to S3. Feel free to change it back if you think the bug is still critical.

For more information, please visit auto_nag documentation.

Comment 16

[Matthew Gaudet (he/him) [:mgaudet]]

There does seem to still be ongoing issues with hmpalert.dll, under the ReadProcessMemory signature at least, but the volume does remain relatively low (tho 92 crashes in a month...). We also have an embedder of SpiderMonkey who has run into this too (bug 1887084)