1743427 - Socket process top crash in IPSEng64 (Symantec Intrusion Detection)

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect, P2)

Description

[Jim Mathies [:jimm]]

New top crash in the Socket process for 94. This module represents six of the top ten crashes for this process in 94 and accounts fro around 95% of the crashes we see on a regular basis. There is no visibility into these crashes on crash stats currently.

Signatures -
IPSEng64.pdb@0xfc1a2
IPSEng32.pdb@0xe31fd
IPSEng64.pdb@0xf89e2
IPSEng64.pdb@0xfbc72
IPSEng64.pdb@0x112d37
IPSEng32.pdb@0xe0ddd
IPSEng32.pdb@0xe2d9d
IPSEng32.pdb@0xf374a

The stacks are pretty worthless. Every frame is in this module after a thread start. Unknown what thread this is on. Example stack -

0
KERNELBASE.dll
RaiseException
1
IPSEng64.pdb
0xfc1a2
2
IPSEng64.pdb
0xfc468
3
IPSEng64.pdb
0x1e15ad
4
IPSEng64.pdb
0x1e1930
5
IPSEng64.pdb
0xd6c77
6
IPSEng64.pdb
0x31426
7
IPSEng64.pdb
0x50a45
8
kernel32.dll
BaseThreadInitThunk
9
ntdll.dll
RtlUserThreadStart

Comment 1

[Gian-Carlo Pascutto [:gcp]]

The fact that these are all RROR_MOD_NOT_FOUND makes me wonder about https://bugzilla.mozilla.org/show_bug.cgi?id=1546154 which landed in Fx94.

Comment 2

[Jim Mathies [:jimm]]

We have similar crashes for another 3rd party (HitmanPro) in the same release.

https://www.mathies.com/mozilla/crashes/sockrelease.html

Comment 3

[Jim Mathies [:jimm]]

I wonder if all the new delay loading triggered an issue with some 3rd party libs. We can post to the symantec list about this if you think it would help.

Comment 4

[Bob Owen (:bobowen)]

(In reply to Jim Mathies [:jimm] from comment #3)

I wonder if all the new delay loading triggered an issue with some 3rd party libs. We can post to the symantec list about this if you think it would help.

I don't quite see how it would directly affect a third party DLL, but maybe they are relying on something being loaded which isn't.
Either way I guess they would have a better chance of working out exactly what is failing, given that we don't even have crash dumps.

Comment 5

[Toshihito Kikuchi [:toshi]]

Added the signature indicating a socket process crash with ERROR_MOD_NOT_FOUND.

Comment 6

[Toshihito Kikuchi [:toshi]]

Ipseng64.dll that I have delayloads the following modules.

• oleaut32.dll
• shell32.dll
• shlwapi.dll
• user32.dll
• userenv.dll
• version.dll
• ws2_32.dll
• ole32.dll
• urlmon.dll

Comment 7

[Gian-Carlo Pascutto [:gcp]]

Guessing at severity, users with this software might have broken WebRTC? Toshihito, do we have a contact at Symantec we can alert to this?

Comment 8

[Jim Mathies [:jimm]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #7)

Guessing at severity, users with this software might have broken WebRTC? Toshihito, do we have a contact at Symantec we can alert to this?

From looking at the code and testing a bit by killing the socket process in task manager, networking will fall back to parent and socket process use will be disabled for the session. So generally everything will continue to work and active connections should continue.

Comment 9

[Toshihito Kikuchi [:toshi]]

I downloaded Norton 360 from https://us.norton.com/downloads and installed it. The latest version contains IPSEng64.dll 17.2.6.25, which matches the crash at IPSEng64+0xfc1a2, the 1st rank crash at https://www.mathies.com/mozilla/crashes/sockrelease.html.

The crash ping data indicates this happens on all platforms Win7/8/8.1/10.

I could not reproduce the crash, but upon analyzing the assembly, I figured out the crashing address IPSEng64+0xfc1a2 does delay-load shlwapi.dll. Pre-loading shlwapi.dll could be a workaround, but this can be caused by any modules to be delayloaded.

(In reply to Gian-Carlo Pascutto [:gcp] from comment #7)

Guessing at severity, users with this software might have broken WebRTC? Toshihito, do we have a contact at Symantec we can alert to this?

We have a discussion group with Symantec. I'll post an email to let them know this issue. A possible fix on their side is to use dynamic linking instead of delayload.

Comment 10

[Toshihito Kikuchi [:toshi]]

Sent an email to our discussion group with Symantec on Nov-30

Comment 11

[Jim Mathies [:jimm]]

Hey Toshi, can we block the injection of libraries like IPSEng*, hmpalert*, and kwsui* from the Socket process?

Comment 12

[Toshihito Kikuchi [:toshi]]

For socket process, we've enabled delayed CIG for years (bug 1611290). I think it's time to move on to pre-spawn CIG for socket.

Comment 13

[Toshihito Kikuchi [:toshi]]

Attachment: Bug 1743427 - Enable pre-spawn CIG for socket process. r=bobowen

Since socket process can be launched before GeckoDependentInitialize,
we cache the install directory in InitSignedPolicyRulesToBypassCig
instead of using sBinDir.

This patch moves GetInstallDirectory to WinHeaderOnlyUtils.h to reuse
and removes WinUtils::GetModuleFullPath that is unused.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2e0fda2938e9
Enable pre-spawn CIG for socket process. r=bobowen

Comment 15

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/2e0fda2938e9