2005255 - startup Crash in [@ ntdll.dll | inprocessclient64.dll | mozilla::interceptor::FuncHook<T>::operator()] with SentinelOne

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

33 startup crashes from 23 installs of Firefox 146.0 on Windows 11, no crash reports for version <=145.

Greg:
a) Do you contacts there?
b) Should the .dll be blocklisted is a short term mitigation?

Crash report: https://crash-stats.mozilla.org/report/index/470c4f88-f246-4e7e-8266-8ee2b0251210

Reason:

EXCEPTION_ILLEGAL_INSTRUCTION

Top 10 frames:

0  ntdll.dll  ntdll.dll@0xb9b3c
1  InProcessClient64.dll  InProcessClient64.dll@0x81140
2  InProcessClient64.dll  InProcessClient64.dll@0x80e80
3  mozglue.dll  mozilla::interceptor::FuncHook<mozilla::interceptor::WindowsDllInterceptor<mo...  toolkit/xre/dllservices/mozglue/nsWindowsDllInterceptor.h:144
3  mozglue.dll  patched_LdrLoadDll(wchar_t*, unsigned long*, _UNICODE_STRING*, void**)  toolkit/xre/dllservices/mozglue/WindowsDllBlocklist.cpp:513
4  KERNELBASE.dll  KERNELBASE.dll@0xf4944
5  mozglue.dll  mozilla::detail::DynamicallyLinkedFunctionPtrBase<int (*)(void*, _PROCESS_MIT...  mozglue/misc/DynamicallyLinkedFunctionPtr.h:50
5  mozglue.dll  mozilla::StaticDynamicallyLinkedFunctionPtr<int (*)(void*, _PROCESS_MITIGATIO...  mozglue/misc/DynamicallyLinkedFunctionPtr.h:106
5  mozglue.dll  mozilla::FetchGetProcessMitigationPolicyFunc()  mozglue/misc/WindowsProcessMitigations.cpp:19
5  mozglue.dll  mozilla::IsWin32kLockedDown::<lambda_0>::operator()() const  mozglue/misc/WindowsProcessMitigations.cpp:30

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

SentinelOne has been contacted.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox146 (release). However, the bug still isn't assigned and has low severity.

:jstutte, could you please find an assignee and increase the severity for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 3

[Greg Stoll :gstoll]

This is a fairly high-volume startup crash, so I'm going to block the offending DLL.

Comment 4

[Greg Stoll :gstoll]

It looks like the crash is happening when we call LoadLibrary on kernel32.dll in order to call GetProcessMitigationPolicy, and that LoadLibrary call is being intercepted by InProcessClient64.dll, which is crashing.

I'm not sure why this only shows up in Fx 146; it doesn't seem like the relevant Fx code has changed?

Comment 5

[Greg Stoll :gstoll]

Windows DLLBlocklist request form

1. How were we aware of the problem?
   topcrasher in Fx 146

2. What is a suspicious product causing the problem?
   InProcessClient64.dll published by SentinelOne.

3. Is the product downloadable? If so, do we have a local repro?
   It does not appear to be downloadable.

4. Which OS versions does the problem occur on?
   Windows 11 only

5. Which process types does the problem occur on?
   parent process

6. What is the maximum version of the module in the crash reports?
   25.1.4.434

7. Is the issue fixed by a newer version of the product?
   I don't think so; there are no newer versions in our telemetry.

8. Do we have data about the module in the third-party-module ping?
   Yes

9. Do we know how the module is loaded?
   It appears to load through a normal mechanism

10. Describe your conclusion.
    We should block all versions of InProcessClient64.dll in the parent process.

Comment 6

[Greg Stoll :gstoll]

Attachment: Bug 2005255 - block InProcessClient64.dll for causing startup crashes r=gsvelto

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/62ef1c5e829c
https://hg.mozilla.org/integration/autoland/rev/957f4fb5a2af
block InProcessClient64.dll for causing startup crashes r=win-reviewers,handyman

Comment 8

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/957f4fb5a2af

Comment 9

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined: topcrash startup crash in Fx 146
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: low
• Explanation of risk level: just blocking a DLL that's crashing on startup, probably can't make things worse :-)
• String changes made/needed: no
• Is Android affected?: no

Comment 10

[Greg Stoll :gstoll]

Attachment: Bug 2005255 - block InProcessClient64.dll for causing startup crashes

Original Revision: https://phabricator.services.mozilla.com/D276001

Comment 11

[Phabricator Automation]

firefox-release Uplift Approval Request

• User impact if declined: topcrash startup crash in Fx 146
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: low
• Explanation of risk level: just blocking a DLL that's crashing on startup, probably can't make things worse :-)
• String changes made/needed: no
• Is Android affected?: no

Comment 12

[Greg Stoll :gstoll]

Attachment: Bug 2005255 - block InProcessClient64.dll for causing startup crashes

Original Revision: https://phabricator.services.mozilla.com/D276001

Comment 13

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/63b968ad5c18
https://hg.mozilla.org/releases/mozilla-beta/rev/611472ec94cf

Comment 14

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/70917f648846
https://hg.mozilla.org/releases/mozilla-release/rev/da2efca7fd68

Comment 15

[Greg Stoll :gstoll]

Sigh, I didn't notice that these are all ARM64 crashes. This is basically a duplicate of bug 1936029.