1766029 - Crash in [@ LdrShutdownThread] caused by Waterwall Systems Co. Safaweb.dll

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect)

Description

[Bob Owen (:bobowen)]

Relatively small number of clients (17) extracted from crash pings for experiment.
515 crashes in the crash ping data I have extracted, so far.

safaweb.dll (for 32-bit) and safaweb64.dll from Waterwall Systems Co., Ltd seem to be the culprits.
Looks like it might be a South Korean company.

Crash report: https://crash-stats.mozilla.org/report/index/3b5e5f4e-b8d0-4e31-aa40-918890220421

Reason: EXCEPTION_ACCESS_VIOLATION_EXEC

Top 10 frames of crashing thread:

0 None @0x598800a4 
1 ntdll.dll LdrShutdownThread 
2 ntdll.dll RtlExitUserThread 
3 kernelbase.dll FreeLibraryAndExitThread 
4 kernel32.dll FreeLibraryAndExitThreadStub 
5 safaweb.dll safaweb.dll@0x00047289 
6 safaweb.dll safaweb.dll@0x0004736e 
7 safaweb.dll safaweb.dll@0x000471dc 
8 kernel32.dll BaseThreadInitThunk 
9 ntdll.dll _RtlUserThreadStart 

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1759168

Comment 2

[Bob Owen (:bobowen)]

I've emailed [e-mail] that I found on their website, to alert them to the issue.
I don't see a free/demo version, so I can't reproduce and I don't want to try and block the DLLs without testing.

Comment 3

[Barret Rennie [:barret] (they/them)]

:bobowen Is there anything left to do here for 100 or are we just waiting on the 3rd party?

Comment 4

[Bob Owen (:bobowen)]

(In reply to Barret Rennie [:barret] (they/them) from comment #3)

:bobowen Is there anything left to do here for 100 or are we just waiting on the 3rd party?

Yeah, just waiting on the 3rd party I think.

Comment 5

[Bob Owen (:bobowen)]

This is the biggest crash at the moment connected to win32k lockdown in Fx100.
Relatively small number of clients in crash pings, no more than 20-30.
Extracting and analysing the crashes is slow, so I'm doing it in parallel per day, which makes determining the true client number tricky.
Quite a large number of crashes, but that is presumably because it hits lots of content processes per client.

Using the instructions to get the stacks from the third party module pings gives the following.
The loads were from March. Obviously difficult to be sure that blocking will work from this.

0 firefox.exe!mozilla::freestanding::patched_LdrLoadDll(wchar_t*, unsigned long*, _UNICODE_STRING*, void**)+0x1ec
1 <unknown>+0xffffffff

Comment 6

[Bob Owen (:bobowen)]

Attachment: Bug 1766029: Block safaweb* DLLs in child processes due to win32k lockdown crash. r=gcp!

The version is the last one for which we have seen crashes.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0e94ec1aa0ec
Block safaweb* DLLs in child processes due to win32k lockdown crash. r=gcp

Comment 8

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/0e94ec1aa0ec

Comment 9

[Bob Owen (:bobowen)]

Comment on attachment 9275740 [details]
Bug 1766029: Block safaweb* DLLs in child processes due to win32k lockdown crash. r=gcp!

Beta/Release Uplift Approval Request

• User impact if declined: We don't see this crash in the crash pings on Nightly, so we need it on Beta to know if it is helping or not.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: Bug 1768800
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): There is a risk that blocking the DLL could cause more issues, but as we are not seeing this crash on Nightly anyway the chance is we won't be mitigating that risk by waiting for it to roll naturally to Beta.
  I will monitor the crash pings, so we should pick up any issues the next day when they come in.
• String changes made/needed: None
• Is Android affected?: No

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9275740 [details]
Bug 1766029: Block safaweb* DLLs in child processes due to win32k lockdown crash. r=gcp!

Approved for 101.0b6.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/db49ad3c446f

Comment 12

[Bob Owen (:bobowen)]

Comment on attachment 9275740 [details]
Bug 1766029: Block safaweb* DLLs in child processes due to win32k lockdown crash. r=gcp!

Beta/Release Uplift Approval Request

• User impact if declined: Users with safaweb*.dlls injected into the content processes will still see a high number of crashes.
  (Possibly for all content processes.)
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: Bug 1768800
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): I'm actually dropping this to low (I put medium for Beta uplift) because it looks like it might be hitting all content processes for the users that are seeing the problem.
  In which case blocking is unlikely to make things worse.
• String changes made/needed: None
• Is Android affected?: No

Comment 13

[Dianna Smith [:diannaS]]

Comment on attachment 9275740 [details]
Bug 1766029: Block safaweb* DLLs in child processes due to win32k lockdown crash. r=gcp!

Approved for 100.0.1

Comment 14

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-release/rev/78da43015486