Open Bug 1752733 Opened 8 months ago Updated 4 months ago

Crash caused by Sophos AV software in [@ hmpalert.dll | NS_FaultTolerantHeap::APIHook_RtlAllocateHeap]

Categories

(External Software Affecting Firefox :: Other, defect)

Unspecified
Windows 10
defect

Tracking

(Not tracked)

People

(Reporter: gsvelto, Unassigned)

References

Details

(Keywords: crash)

Crash Data

Attachments

(2 files)

Crash report: https://crash-stats.mozilla.org/report/index/9247ceaa-33d5-4b6e-b21b-b13c10220130

Reason: Unhandled C++ Exception

Top 10 frames of crashing thread:

0 kernelbase.dll RaiseException 
1 hmpalert.dll hmpalert.dll@0x000000000009562f 
2 aclayers.dll void* NS_FaultTolerantHeap::APIHook_RtlAllocateHeap 
3 hmpalert.dll hmpalert.dll@0x000000000000d1bc 
4 hmpalert.dll hmpalert.dll@0x00000000000b65bf 
5 hmpalert.dll hmpalert.dll@0x00000000000d0c67 
6 hmpalert.dll hmpalert.dll@0x000000000000ca4b 
7 aclayers.dll void* NS_FaultTolerantHeap::APIHook_RtlAllocateHeap 
8 hmpalert.dll hmpalert.dll@0x0000000000030c5a 
9 ntdll.dll LdrpInitialize 

I added only the top 10 signatures but there's a lot more.

The volume is increasing and new signatures are popping up all the time.

Crash Signature: EtwpSetProviderTraits] [@ hmpalert.dll | js::frontend::NameOpEmitter::emitAssignment] → EtwpSetProviderTraits] [@ hmpalert.dll | js::frontend::NameOpEmitter::emitAssignment] [@ hmpalert.dll | ReadProcessMemory] [@ hmpalert.dll | js::jit::MoveResolver::resolve] [@ hmpalert.dll | js::frontend::TokenStreamSpecific<T>::getTokenInternal] [@…

Hi Yaniv, could Sophos look into these Firefox crashes for which Sophos' hmpalert.dll is part of the crashing thread, version 3.8.3.808 for bp-50b3960a-7d2b-424a-8e4d-324620220208.

List of crashes with hmpalert.dll in the first 15 frames of the crashing thread.

Flags: needinfo?(yaniv.frank)

Hi,

I work for Sophos. Do you have any more detailed description of what steps you are following that trigger these crashes?

Thanks,

Chloe

Flags: needinfo?(gsvelto)

Looking at the comments from the crash report these seem to happen at random but rather frequently for the affected users. Under the most common signature it seems that a C++ exception is being raised from the injected code (see this crash report). This seems to be happening in the "COM MTA" thread (see here) within hooks inserted by the module. I'm not familiar with that code but maybe some of my colleagues could help.

Flags: needinfo?(gsvelto)

Hi Gabriele,
Is there any way I can have access to some of the actual crash dumps as well please?
Thanks

Flags: needinfo?(gsvelto)

Hi Chloe, we can't share crash dumps because they might contain sensitive user data, but we can provide you with enough information to reconstruct the call stack in your DLL. Toshihito will help you with this.

Flags: needinfo?(gsvelto) → needinfo?(tkikuchi)

(In reply to Gabriele Svelto [:gsvelto] from comment #4)

Looking at the comments from the crash report these seem to happen at random but rather frequently for the affected users. Under the most common signature it seems that a C++ exception is being raised from the injected code (see this crash report). This seems to be happening in the "COM MTA" thread (see here) within hooks inserted by the module. I'm not familiar with that code but maybe some of my colleagues could help.

I looked at the crash of 30322557-8a99-41d9-90a3-6f1570220209 you saw, but the crashing thread had nothing to do with COM MTA.

I analyzed two dumps and attached my debug notes including the callstack of the crashing thread. Chloe, can you take a look and share it with engineers. If you need other specific information, please let me know. Please note that we have minidumps, not fulldumps, so some information (TLS, Heap, etc.) are not available.

I could set up Free Trial version of Sophos (with hmpalert.dll v3.8.3.808) and ran Firefox with Win7 mode, but this crash did not happen. Here are a couple of things I noticed.

  • hmpalert.dll explicitly calls RaiseException with 0E06D7363h.
    I don't know what condition triggered this, but basically injected modules should not do this.

  • Having AcLayers!NS_FaultTolerantHeap::APIHook_RtlAllocateHeap in the stack (and it appears in the crash signatures) has nothing to do with the exception. This happens because hmpalert+0xd980 called HeapAlloc, which returned normally. It means, however, the process ran with compatibility mode. On Win10, a process does not usually load AcLayers.dll. So running Firefox with compat mode is likely to be one of the factors.

Flags: needinfo?(tkikuchi)

(In reply to Toshihito Kikuchi [:toshi] from comment #7)

I looked at the crash of 30322557-8a99-41d9-90a3-6f1570220209 you saw, but the crashing thread had nothing to do with COM MTA.

You're right, though I swear I opened several crashes and they were all in the "COM MTA" thread. Can't find them anymore now though, maybe I mixed it up with some other crash.

Flags: needinfo?(chloe.bell)

Hi Toshihito,

Thank you for providing these analyses.

We believe the issue is a memory leak that occurs when running in compatibility mode (so the aclayers.dll is loaded). This heap has a fixed size and so it runs out of memory and causes the browser to crash.

We have not been able to reproduce the issue but do have a customer who has reported is against other browsers so it does not appear to be specific to Firefox.

We are working to identify a fix and will release this as a hotfix for customers to use.

Thanks

Flags: needinfo?(chloe.bell)

Thank you for the updates. It's great news that Sophos engineers are actively working on the case! Please let us know if you need anything from us to help your investigation.

See Also: → 1759168
See Also: → 1769309

Redirect a needinfo that is pending on an inactive user to the triage owner.
:haik, since the bug has high severity and recent activity, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(yaniv.frank) → needinfo?(haftandilian)

Hi all,

Sophos has released a fix for this issue and it is included in the latest Intercept X Cumulative Hotfix, see https://support.sophos.com/support/s/article/KB-000038477?language=en_US

This will also be rolled out to all customers in the next release.

Thanks,

Chloe

Flags: needinfo?(haftandilian)
You need to log in before you can comment on or make changes to this bug.