1804023 - Crashes caused by MSI Nahimic audio driver/effects software

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect, P2)

Description

[Gabriele Svelto [:gsvelto]]

Crash report: https://crash-stats.mozilla.org/report/index/c69e4b5e-d4d3-488e-919a-a71f20221204

Reason: EXCEPTION_ACCESS_VIOLATION_READ

Top 10 frames of crashing thread:

0  NahimicOSD.dll  NahimicOSD.dll@0x14aa0  
1  nss3.dll  _MD_CURRENT_THREAD  nsprpub/pr/src/md/windows/w95thred.c:373
1  nss3.dll  PR_GetCurrentThread  nsprpub/pr/src/threads/prcthr.c:151
1  nss3.dll  PR_GetThreadPrivate  nsprpub/pr/src/threads/prtpd.c:205
2  ntdll.dll  LdrpApplyFileNameRedirection  
3  ntdll.dll  LdrpPreprocessDllName  
4  ntdll.dll  LdrpFindLoadedDll  
5  ntdll.dll  LdrGetDllHandleEx  
6  KERNELBASE.dll  GetModuleHandleExW  
7  NahimicOSD.dll  NahimicOSD.dll@0x118f6  

The software is available here. There are several more crash signatures for Thunderbird, I've filed just the Firefox ones.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:haik, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[Haik Aftandilian [:haik]]

I've sent a message to the MSI support Twitter account (@msiUSA_Tech) asking them to look into this issue.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 10 desktop browser crashes on nightly (startup)

:haik, could you consider increasing the severity of this top-crash bug?

For more information, please visit auto_nag documentation.

Comment 4

[Haik Aftandilian [:haik]]

Additionally, I've created a support ticket with MSI.

Let's move forward and investigate blocking the DLL.

Comment 5

[Greg Stoll :gstoll]

Will do.

Comment 6

[Greg Stoll :gstoll]

Windows DLLBlocklist request form

1. How were we aware of the problem?
   Notified via crash reports linked in this bug.

2. What is a suspicious product causing the problem?
   MSI Nahimic audio driver

3. Is the product downloadable? If so, do we have a local repro?
   It is downloadable, but installing the linked driver has no effect because I don't have the appropriate hardware, so it doesn't try to load in Firefox.

4. Which OS versions does the problem occur on?
   All the reports I've seen are from Windows 11.

5. Which process types does the problem occur on?
   Just the GPU process; unfortunately we don't have a way to block DLLs just in the GPU process right now. (and given that this is an audio driver it doesn't seem to try to load in other processes)

6. What is the maximum version of the module in the crash reports?
   2.2.25.0

7. Is the issue fixed by a newer version of the product?
   Possibly - we have third-party-module pings from 2.2.27.0 but no crash reports for this version.

8. Do we have data about the module in the third-party-module ping?
   Yes, but unfortunately only in the browser process, as we don't get pings from the GPU process.

9. Do we know how the module is loaded?
   I'm a little concerned about this - the call stack for a load in the browser process is

0 firefox.exe!mozilla::freestanding::patched_LdrLoadDll(wchar_t*, unsigned long*, _UNICODE_STRING*, void**)+0x1ef
1 KERNELBASE.dll!LoadLibraryExW+0x172
2 user32.dll!_ClientLoadLibrary+0xa7
3 ntdll.dll!KiUserCallbackDispatcherContinue+0x0
4 win32u.dll!NtUserCallNextHookEx+0x14
5 user32.dll!CallNextHookEx+0x53
6 user32.dll!DispatchHookW(int, uint64_t, int64_t, int64_t (*)(int, uint64_t, int64_t))+0x74
7 user32.dll!_fnHkINLPCBTCREATESTRUCT+0xb8
8 ntdll.dll!KiUserCallbackDispatcherContinue+0x0
9 win32u.dll!NtUserCreateWindowEx+0x14
10 user32.dll!VerNtUserCreateWindowEx(unsigned long, _LARGE_STRING*, _LARGE_STRING*, unsigned long, int, int, int, int, HWND__*, HMENU__*, void*, void*, ZBID, unsigned long, unsigned long)+0x210
11 user32.dll!CreateWindowInternal+0x1a8
12 user32.dll!CreateWindowExW+0x82
13 combase.dll!InitMainThreadWnd()+0x57
14 combase.dll!ThreadFirstInitialize(COleTls&, unsigned long)+0x181
15 combase.dll!_CoInitializeEx(unsigned long)+0x1bd
16 combase.dll!CoInitializeEx(void*, unsigned long)+0x37
17 mozglue.dll!mozilla::mscom::ProcessRuntime::ProcessRuntime(const mozilla::mscom::ProcessRuntime::ProcessCategory)+0x5b
18 mozglue.dll!mozilla::CreateAndStorePreXULSkeletonUI(HINSTANCE__*, int, char**)+0x5b
19 firefox.exe!wmain(int, wchar_t**)+0x4d6
20 firefox.exe!__scrt_common_main_seh()+0x10c
21 kernel32.dll!BaseThreadInitThunk+0x1d
22 ntdll.dll!RtlUserThreadStart+0x28

I think it would be safer to blocklist this only in the GPU process.

10. Describe your conclusion.
    We should add the capability to blocklist modules only in the GPU process, and use that to block nahimicosd.dll versions 2.2.25.0 and earlier.

Other notes:

• As far as I can tell all of these crashes are in the GPU process, when initializing Direct3D11 (DeviceManagerDx::LoadD3D11()).
• This WPF GitHub issue suggests that users may be able to add Firefox.exe to a blocklist to stop it from loading in Firefox.

Comment 7

[Greg Stoll :gstoll]

Attachment: Bug 1804023 - Part 1: make blocklisting initialization take process type instead of a series of booleans r=gsvelto

This is just refactoring in preparation for the next patches

Comment 8

[Greg Stoll :gstoll]

Attachment: Bug 1804023 - Part 2: add ability to blocklist DLLs in just the GPU process r=gsvelto

As with the socket process, we can't automated test that the block works in the GPU process, but I manually verified this. I did add an automated test that ensures blocking something in the GPU process doesn't block it in other processes.

Depends on D167398

Comment 9

[Greg Stoll :gstoll]

Attachment: Bug 1804023 - Part 3: block nahimicosd.dll in just the GPU process r=gsvelto

Depends on D167399

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b5b4be5f7502
Part 1: make blocklisting initialization take process type instead of a series of booleans r=gsvelto
https://hg.mozilla.org/integration/autoland/rev/9d2766a3d506
Part 2: add ability to blocklist DLLs in just the GPU process r=gsvelto
https://hg.mozilla.org/integration/autoland/rev/8ac7bb069839
Part 3: block nahimicosd.dll in just the GPU process r=gsvelto

Comment 11

[Cosmin Sabou [:CosminS]]

Backed out for causing DllBlocklistInit.cpp related bustages.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=8ac7bb069839aab7717ec50ce16fe631bca44f37&selectedTaskRun=aZjp2HGhQeueiKB1JLh0Ww.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=403048656&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/f2fbf518572b21b732795f6edeb10a3209799757

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4d423cd40d19
Part 1: make blocklisting initialization take process type instead of a series of booleans r=gsvelto
https://hg.mozilla.org/integration/autoland/rev/20a89b89aced
Part 2: add ability to blocklist DLLs in just the GPU process r=gsvelto
https://hg.mozilla.org/integration/autoland/rev/d9154943e83f
Part 3: block nahimicosd.dll in just the GPU process r=gsvelto

Comment 13

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/4d423cd40d19
https://hg.mozilla.org/mozilla-central/rev/20a89b89aced
https://hg.mozilla.org/mozilla-central/rev/d9154943e83f

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

There have been 5 startup crashes from one installation of yesterday's Nightly. Could you investigate?

Comment 15

[Greg Stoll :gstoll]

The reports I looked at all have launcherProcessState: 0 in the telemetry environment, which means the Launcher process isn't active, which means we fall back to the mozglue blocklist, which isn't as effective.

Looking at the stack trace, I'm a little surprised that this doesn't get blocked, but since it's just one user/installation I don't think it's worth looking into right now unless we see more instances of this.

Comment 16

[Gabriele Svelto [:gsvelto]]

There's a few more crashes this time involving version 2.2.27.0 and the GPU process and sometimes the parent process. They almost all seem to involve people who've turned off the launcher process.