1644240 - Crash in [@ mciwindow] with PROCOMP INDUSTRIA ELETRONICA (Diebold Brazil)

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect, P1)

Description

[[:philipp]]

This bug is for crash report bp-4bfad432-82b8-4f0e-923a-df4ec0200608.

Top 4 frames of crashing thread:

0  @0x7ffbbba00ffa 
1 winmm.dll mciwindow 
2 kernel32.dll BaseThreadInitThunk 
3 ntdll.dll RtlUserThreadStart 

this crash signature started appearing in late april and s constantly increasing in volume since then - correlations show the presence of a dll by kaspersky hooking into the browser process:

(100.0% in signature vs 02.29% overall) Module "klsihk64.dll" = true

judging by locale and user comments this seems to primarily hit users in brazil.

Comment 1

[Toshihito Kikuchi [:toshi]]

This is very similar to bug 1438562. This crash is EXCEPTION_ACCESS_VIOLATION_EXEC on an address jumped from winmm!mciwindow via a hook for user32!GetMessageA. On the other hand, the crash of bug 1438562 is EXCEPTION_ACCESS_VIOLATION_READ at the start address of user32!GetMessageA after jumped from winmm!mciwindow.

The version of klsihk64.dll in the crash instances is 20.1.16.0 (and a few instances of 16.0.11.0). I can see version 20.2.24.0 in the loading events of the third-party module ping. This issue might have fixed in that version.

Comment 2

[Toshihito Kikuchi [:toshi]]

Hi Alexey, I heard we can get help from you about Kaspersky's product. We're getting the crash reports from the hook of user32!GetMessageA that I think applied by klsihk64.dll. Can you please take a look?

Comment 3

[Alexey Totmakov]

Hi Toshihito,

we will look at the crash and come back.

Comment 4

[alexey.ryaskin]

(In reply to Toshihito Kikuchi [:toshi] from comment #2)

Hi Alexey, I heard we can get help from you about Kaspersky's product. We're getting the crash reports from the hook of user32!GetMessageA that I think applied by klsihk64.dll. Can you please take a look?

Hi Toshihito,
My name is Alexey Ryaskin, I'm one of engineers from Kaspersky responsible for the klsihk64.dll. Could you please help my to get the crash dumps?

Comment 5

[Toshihito Kikuchi [:toshi]]

I'm afraid our policy does not allow sharing minidumps because it may contain private user data. I can share non-personal data like stack traces instead. Sorry for inconvenience.

Below is my debugger log analyzing https://crash-stats.mozilla.org/report/index/847e38da-de46-45a3-a354-fb7870200619. What happened was winmm!mciwindow made a jump to GetMessageA through winmm's IAT, ending up executing a freed address. The paths of loaded modules told us Kaspersky Total Security 20.0 was installed. I believe one of its modules hooked user32!GetMessage, and a trampoline page of the hook was freed too early. There is a thread running klsihk64's code, waiting on WaitForSingleObjectEx. Maybe that thread freed the trampoline.

Please let me know if there is any other info you need and I can pull out from the minidump.

0:098> .excr
rax=00007ff80c4975c0 rbx=0000000000000001 rcx=0000001a591bf700
rdx=0000000000000000 rsi=00007ff805389000 rdi=00000000000020d8
rip=00007ff80d440ffa rsp=0000001a591bf698 rbp=0000000000000000
 r8=0000000000000000  r9=0000000000000000 r10=0000000000000000
r11=0000001a591bf510 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
00007ff8`0d440ffa ??              ???
0:098> !address 00007ff8`0d440ffa

Usage:                  Free
Base Address:           00007ff8`0d43c000
End Address:            00007ff8`0d460000
Region Size:            00000000`00024000 ( 144.000 kB)
State:                  00010000          MEM_FREE
Protect:                00000001          PAGE_NOACCESS
Type:                   <info not present at the target>

Content source: 0 (invalid), length: 35f006
0:098> knL
  *** Stack trace for last set context - .thread/.cxr resets it
 # Child-SP          RetAddr           Call Site
00 0000001a`591bf698 00007ff8`05373301 0x00007ff8`0d440ffa
01 0000001a`591bf6a0 00007ff8`0d887bd4 winmm!mciwindow+0xf1
02 0000001a`591bf740 00007ff8`0e08ce51 kernel32!BaseThreadInitThunk+0x14
03 0000001a`591bf770 00000000`00000000 ntdll!RtlUserThreadStart+0x21
0:098> ub 00007ff8`05373301
winmm!mciwindow+0xd0:
00007ff8`053732e0 0f1f440000      nop     dword ptr [rax+rax]
00007ff8`053732e5 85db            test    ebx,ebx
00007ff8`053732e7 0f8402320000    je      winmm!mciwindow+0x32df (00007ff8`053764ef)
00007ff8`053732ed 4533c9          xor     r9d,r9d
00007ff8`053732f0 488d4c2460      lea     rcx,[rsp+60h]
00007ff8`053732f5 4533c0          xor     r8d,r8d
00007ff8`053732f8 33d2            xor     edx,edx
00007ff8`053732fa 48ff15078e0100  call    qword ptr [winmm!_imp_GetMessageA (00007ff8`0538c108)]

Comment 6

[Semion Shilkov]

(In reply to Toshihito Kikuchi [:toshi] from comment #1)

This is very similar to bug 1438562. This crash is EXCEPTION_ACCESS_VIOLATION_EXEC on an address jumped from winmm!mciwindow via a hook for user32!GetMessageA. On the other hand, the crash of bug 1438562 is EXCEPTION_ACCESS_VIOLATION_READ at the start address of user32!GetMessageA after jumped from winmm!mciwindow.

The version of klsihk64.dll in the crash instances is 20.1.16.0 (and a few instances of 16.0.11.0). I can see version 20.2.24.0 in the loading events of the third-party module ping. This issue might have fixed in that version.

Hi, Toshihito

Have info about how many dumps did you received and how many users affected by this problem?

Comment 7

[alexey.ryaskin]

(In reply to Toshihito Kikuchi [:toshi] from comment #5)

I'm afraid our policy does not allow sharing minidumps because it may contain private user data. I can share non-personal data like stack traces instead. Sorry for inconvenience.

Below is my debugger log analyzing https://crash-stats.mozilla.org/report/index/847e38da-de46-45a3-a354-fb7870200619. What happened was winmm!mciwindow made a jump to GetMessageA through winmm's IAT, ending up executing a freed address. The paths of loaded modules told us Kaspersky Total Security 20.0 was installed. I believe one of its modules hooked user32!GetMessage, and a trampoline page of the hook was freed too early. There is a thread running klsihk64's code, waiting on WaitForSingleObjectEx. Maybe that thread freed the trampoline.

Unfortunately lack of access to the memory dump extremely decreases our chances to determinate root cause of the issue.

Klsihk doesn't hook GetMessage. Thus I think It could be somebody else.
Could you please check, if there is in the memory dump any other third party module that might hook the GetMessage.

Comment 8

[Toshihito Kikuchi [:toshi]]

(In reply to Semion Shilkov from comment #6)

Have info about how many dumps did you received and how many users affected by this problem?

Yes, we received 1000+ crash reports in the last seven days. You can see the details at https://crash-stats.mozilla.org/signature/?product=Firefox&signature=mciwindow.

(In reply to alexey.ryaskin from comment #7)

Klsihk doesn't hook GetMessage. Thus I think It could be somebody else.
Could you please check, if there is in the memory dump any other third party module that might hook the GetMessage.

The correlations info in our report indicates all crash instances had klsihk64.dll and inproc_agent.dll loaded. Those are only the non-Microsoft modules with 100% correlation. It's surprising Klsihk does not hook GetMessageA. How about inproc_agent.dll then?

Comment 9

[alexey.ryaskin]

(In reply to Toshihito Kikuchi [:toshi] from comment #8)

The correlations info in our report indicates all crash instances had klsihk64.dll and inproc_agent.dll loaded. Those are only the non-Microsoft modules with 100% correlation. It's surprising Klsihk does not hook GetMessageA. How about inproc_agent.dll then?

Nether inproc_agent.dll nor klsihk.dll/klsihk64.dll hooks 'GetMessageA'.

Are there some specific conditions that provoke the issue? For instance it's reproducible ONLY on some specific Windows or Firefox versions.
Do you know how to reproduce the issue?
Perhaps you have in-house repro (virtual machine) and can share it to us?

Comment 10

[Toshihito Kikuchi [:toshi]]

(In reply to alexey.ryaskin from comment #9)

Nether inproc_agent.dll nor klsihk.dll/klsihk64.dll hooks 'GetMessageA'.

Are there some specific conditions that provoke the issue? For instance it's reproducible ONLY on some specific Windows or Firefox versions.
Do you know how to reproduce the issue?
Perhaps you have in-house repro (virtual machine) and can share it to us?

I downloaded kav20.0.14.1085abcdefghijen_es_fr_19077.exe and installed it, but an injected module was antimalware_provider.dll, not inproc_agent.dll nor klsihk64.dll.

Unfortunately we don't have a local repro or repro steps. Given that the uptime of the crash varies and the crash happened just after the thread was started from winmm!mciwindow as shown in comment 5, it might be triggered by a media content on a page, but we're not sure.

It's clear that the crash was caused by a hook on user32!GetMessageA. Although there is no suspicious DLL in the dumps, you're right, it's still possible that another kernel driver modified our code region.

Some crash reporters kindly shared their email address in their report. Let us reach out them and ask for their consent to hand over a crash dump to us and Kaspersky. I'll keep this bug updated. Thank you your help!

Comment 11

[alexey.ryaskin]

(In reply to Toshihito Kikuchi [:toshi] from comment #10)

Some crash reporters kindly shared their email address in their report. Let us reach out them and ask for their consent to hand over a crash dump to us and Kaspersky. I'll keep this bug updated. Thank you your help!

Ok, thanks. Waiting for your update.

Comment 12

[Gian-Carlo Pascutto [:gcp]]

We need to keep a close eye on this as the crash rate is already in the thousands and raising.

Comment 13

[Toshihito Kikuchi [:toshi]]

I emailed several reporters with the steps to collect a full dump, but no data yet.

Alexey, I have a quick question. As in comment 10, I cannot find neither inproc_agent.dll nor klsihk64.dll in the trial version of Kaspersky product. Which product contains these DLLs? Is there any chance I can get and try it on my end?

Comment 14

[alexey.ryaskin]

(In reply to Toshihito Kikuchi [:toshi] from comment #13)

I emailed several reporters with the steps to collect a full dump, but no data yet.

Alexey, I have a quick question. As in comment 10, I cannot find neither inproc_agent.dll nor klsihk64.dll in the trial version of Kaspersky product. Which product contains these DLLs? Is there any chance I can get and try it on my end?

The product that contains klsihk.dll/klsihk64.dll is KIS2020.
You can find it here: https://www.kaspersky.com/downloads/thank-you/internet-security-free-trial#download
After installation dlls are located here: C:\ProgramData\Kaspersky Lab\AVP20.0\Bases\

Comment 15

[alexey.ryaskin]

(In reply to alexey.ryaskin from comment #14)

(In reply to Toshihito Kikuchi [:toshi] from comment #13)

I emailed several reporters with the steps to collect a full dump, but no data yet.

Alexey, I have a quick question. As in comment 10, I cannot find neither inproc_agent.dll nor klsihk64.dll in the trial version of Kaspersky product. Which product contains these DLLs? Is there any chance I can get and try it on my end?

The product that contains klsihk.dll/klsihk64.dll is KIS2020.
You can find it here: https://www.kaspersky.com/downloads/thank-you/internet-security-free-trial#download
After installation dlls are located here: C:\ProgramData\Kaspersky Lab\AVP20.0\Bases\

I have just checked it with kis20.0.14.1085abcdefghijen_22595.exe (accessible using the link I provided)

Comment 16

[Toshihito Kikuchi [:toshi]]

(In reply to alexey.ryaskin from comment #15)

The product that contains klsihk.dll/klsihk64.dll is KIS2020.
You can find it here: https://www.kaspersky.com/downloads/thank-you/internet-security-free-trial#download
After installation dlls are located here: C:\ProgramData\Kaspersky Lab\AVP20.0\Bases\

I have just checked it with kis20.0.14.1085abcdefghijen_22595.exe (accessible using the link I provided)

Is it included in the free trial? I installed it from kis20.0.14.1085abcdefghijen_22595.exe with the default options. The following folders were created under the "Kaspersky Lab", but no AVP20.0.

• Kaspersky Internet Security 20.0
• Kaspersky Password Manager 9.0.2
• Kaspersky Secure Connection 4.0

Comment 17

[alexey.ryaskin]

(In reply to Toshihito Kikuchi [:toshi] from comment #16)

Is it included in the free trial? I installed it from kis20.0.14.1085abcdefghijen_22595.exe with the default options.

Yes it's included in freetrial. I tried now onemore time to download it from link provided and install on Windows 10 (yesterday on Windows 7) - everything is as I described.

The following folders were created under the "Kaspersky Lab", but no AVP20.0.

• Kaspersky Internet Security 20.0
• Kaspersky Password Manager 9.0.2
• Kaspersky Secure Connection 4.0

You need to download and install - "Kaspersky Internet Security for Windows" (kis20.0.14.1085abcdefghijen_22595.exe).
Have you managed to download this file kis20.0.14.1085abcdefghijen_22595.exe?

Comment 18

[Toshihito Kikuchi [:toshi]]

(In reply to alexey.ryaskin from comment #17)

You need to download and install - "Kaspersky Internet Security for Windows" (kis20.0.14.1085abcdefghijen_22595.exe).
Have you managed to download this file kis20.0.14.1085abcdefghijen_22595.exe?

Yes, I downloaded kis20.0.14.1085abcdefghijen_22595.exe (sha1: 5630FFFEA07D289F2976090B3768CC3F7B2E5A11). I uploaded screenshot images here while installing KIS on Win10. Can you check I was doing something different?

Comment 19

[alexey.ryaskin]

(In reply to Toshihito Kikuchi [:toshi] from comment #18)

(In reply to alexey.ryaskin from comment #17)

You need to download and install - "Kaspersky Internet Security for Windows" (kis20.0.14.1085abcdefghijen_22595.exe).
Have you managed to download this file kis20.0.14.1085abcdefghijen_22595.exe?

Yes, I downloaded kis20.0.14.1085abcdefghijen_22595.exe (sha1: 5630FFFEA07D289F2976090B3768CC3F7B2E5A11). I uploaded screenshot images here while installing KIS on Win10. Can you check I was doing something different?

Everything is correct except one thing: Please find dlls here: C:\ProgramData\Kaspersky Lab\AVP20.0\Bases
NOT "C:\Program Files (x86)..."

Comment 20

[Toshihito Kikuchi [:toshi]]

(In reply to alexey.ryaskin from comment #19)

Everything is correct except one thing: Please find dlls here: C:\ProgramData\Kaspersky Lab\AVP20.0\Bases
NOT "C:\Program Files (x86)..."

Oh, you have mentioned ProgramData! My bad. I found the files. Thank you for your patience.

I didn't get any fulldumps from the reporters. I'll ask more reporters for their consent to share a minidump with Kaspersky developers.

Comment 21

[alexey.ryaskin]

(In reply to Toshihito Kikuchi [:toshi] from comment #20)

(In reply to alexey.ryaskin from comment #19)

Everything is correct except one thing: Please find dlls here: C:\ProgramData\Kaspersky Lab\AVP20.0\Bases
NOT "C:\Program Files (x86)..."

Oh, you have mentioned ProgramData! My bad. I found the files. Thank you for your patience.

I didn't get any fulldumps from the reporters. I'll ask more reporters for their consent to share a minidump with Kaspersky developers.

Good. Looking forward to the fulldumps.

Comment 22

[Gian-Carlo Pascutto [:gcp]]

Good. Looking forward to the fulldumps.

FYI we had some back and forth over this, including with legal/privacy, and the conclusion was that we can't actually do this. Sorry.

The good news is that meanwhile Toshihito was able to confirm what you said earlier, that Kaspersky indeed isn't hooking those functions. We initially suspected this because of the 100% perfect correlation to Kaspersky DLLs, but knowing the above, I have some other theories. Given the very high correlation to a specific Brazil locale, and the near-perfect correlation of the crash graph to working days (vs weekends), that might indicate that this is a standardized, company wide install, that just happens to enforce a Kaspersky install on every machine, but also some other software, and it's the other software that is crashing.

So looking further, we find:

(74.19% in signature vs 00.73% overall) Module "wslbdhm64.dll" = true [100.0% vs 02.36% if platform_version = 10.0.19041]
wslbdhm64.dll 1.0.3.1116 AB1493ABA47041449F3C19D6307117A21 hook_module.pdb PROCOMP INDUSTRIA ELETRONICA LTDA

And this is indeed a Brazilian company.

Comment 23

[Gian-Carlo Pascutto [:gcp]]

Marco, do you still have a working contact for Diebold? Looks like they re-introduced bugs.

Comment 24

[Toshihito Kikuchi [:toshi]]

Thanks, gcp. I should've noticed that!

I downloaded Warsaw from http://www.dieboldnixdorf.com.br/warsaw, and installed it. wslbdhm64.dll was injected into firefox.exe, but no hook on USER32!GetMessageA. Maybe a specific operation is needed to trigger it..

Comment 25

[Marco Castelluccio [:marco]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #23)

Marco, do you still have a working contact for Diebold? Looks like they re-introduced bugs.

I have the email address of a person at Diebold, though I'm not sure if they are still active. I sent the contact to you and Philipp.

Comment 26

[Toshihito Kikuchi [:toshi]]

I sent an email on August 4 to the contact at Diebold Marco shared, asking their module hooks GetMessageA or not. I got a response and they said the right team was informed.

Comment 27

[Marcelo Ghelman]

Hi there,

I'm a Mozilla volunteer localizer for pt-BR, and software developer since 1984.
For years, we are struggling here in Brazil against that Warsaw software.
The installation is mandatory by many Brazilian banks to allow online transactions.
This software invades the system and causes many troubles.
A lot of clients of these banks send complaints, and even open lawsuits, but nothing changes.
Be sure that the contact at Diebold you send the email calls "right team" his trash can.
Perhaps you guys from Mozilla, Kaspersky, and more can achieve better results, or at least expose this problem widely to the public.

Best regards,
Marcelo

Comment 28

[Toshihito Kikuchi [:toshi]]

Hi Marcelo, thank you for your comment. Do you know what the Warsaw software technically does in the browser and how it can be mandatory for online transactions? If we can identify their module injected in our process, we can block it from being loaded, but our concern is that blocking a bad module may break online transactions even on users not hitting this problem.

We are not sure that an application causing this crash is really Warsaw. The first step we can do would be to land an instrumental patch just to identify who is hooking GetMessageA.

Comment 29

[Francisco B]

Hello people. I'm here because a Firefox expert introduced me to this thread. I started my search because of so many crashes in Firefox, and according to him, they are connected to Kaspersky, but reading the thread, I saw that there is a possibility of the connection being with Warsaw, because I really use it (unfortunately!) I will uninstall of it, and try it for a few days, and if it works, I will inform you here. Thanks.

In case anyone wants to take a quick look at my firefox crashes, they are listed here:

https://crash-stats.mozilla.org/report/index/da0feceb-6c54-47d8-a1f3-b9eaf0201119
https://crash-stats.mozilla.org/report/index/7bc9e00b-8c06-45a6-89b0-d06210201118
https://crash-stats.mozilla.org/report/index/7b61d4c3-1b4a-4132-b70e-8519a0201118
https://crash-stats.mozilla.org/report/index/8fb0e154-7df6-43a6-a9a7-429e80201118
https://crash-stats.mozilla.org/report/index/6e739bb4-5084-4337-a754-11cd20201117

Thanks.

Comment 30

[Toshihito Kikuchi [:toshi]]

Thank you for reporting the issue, Francisco! I hope uninstalling Warsaw prevents the crash.

Upon analyzing the dumps, I have an idea to bypass this crash on our side though it's not a bug in our code. Because you're the first person who reported this issue directly (not via a crash report), could you test a private version of Nightly with Warsaw installed and see the crash is gone or still persists? If you're ok, please download target.zip from that link, unzip it, and run firefox.exe inside (no installation needed).

A patch I wrote can be found here.

Below is the technical details and my theory.

First, the start address of the crashing thread is always winmm!mciwindow. I found out when we call winmm's API PlaySound for the first time in the process, winmm starts winmm!mciwindow in a new thread and enters into a message loop. After that, every time we call PlaySound, it sends several window messages to that thread.

The crash happens when winmm!mciwindow calls to GetMessageA because that function call was redirected to an address which was already freed. So probably GetMessageA was hooked by someone (could be Warsaw), and somehow the trampoline region was freed but it forgot to revert the hook.

I also noticed all dumps I checked still had C:\Program Files\Diebold\Warsaw\wslbdhm64.dll loaded in the process, but the module named wslbdhm64, that looks like another Warsaw module, was unloaded. My guess is when wslbdhm64 was unloaded, the trampoline region was freed.

Thus, the patch is to suppress PlaySound when wslbdhm64.dll is loaded but wslbdhm64 is not loaded on Windows x64.

Comment 31

[Francisco B]

(In reply to Toshihito Kikuchi [:toshi] from comment #30)

Thank you for reporting the issue, Francisco! I hope uninstalling Warsaw prevents the crash.

Upon analyzing the dumps, I have an idea to bypass this crash on our side though it's not a bug in our code. Because you're the first person who reported this issue directly (not via a crash report), could you test a private version of Nightly with Warsaw installed and see the crash is gone or still persists? If you're ok, please download target.zip from that link, unzip it, and run firefox.exe inside (no installation needed).

A patch I wrote can be found here.

Below is the technical details and my theory.

First, the start address of the crashing thread is always winmm!mciwindow. I found out when we call winmm's API PlaySound for the first time in the process, winmm starts winmm!mciwindow in a new thread and enters into a message loop. After that, every time we call PlaySound, it sends several window messages to that thread.

The crash happens when winmm!mciwindow calls to GetMessageA because that function call was redirected to an address which was already freed. So probably GetMessageA was hooked by someone (could be Warsaw), and somehow the trampoline region was freed but it forgot to revert the hook.

I also noticed all dumps I checked still had C:\Program Files\Diebold\Warsaw\wslbdhm64.dll loaded in the process, but the module named wslbdhm64, that looks like another Warsaw module, was unloaded. My guess is when wslbdhm64 was unloaded, the trampoline region was freed.

Thus, the patch is to suppress PlaySound when wslbdhm64.dll is loaded but wslbdhm64 is not loaded on Windows x64.

Hello Toshihito Kikuchi!

First, sorry for the delay in responding.

The current situation is as follows: When you answered me, I had already uninstalled Warsaw, and to my surprise, the problem stopped! Firefox never crashed again! However, to fulfill your request, I would need to install Warsaw again, so I will do the following, I will create a virtual machine with Windows 10 of the same version and patch that I have on the host machine, install Warsaw, and use your version of firefox . Can be? I return in a few days.

Thanks!

Comment 32

[Toshihito Kikuchi [:toshi]]

(In reply to Francisco B from comment #31)

Hello Toshihito Kikuchi!

First, sorry for the delay in responding.

The current situation is as follows: When you answered me, I had already uninstalled Warsaw, and to my surprise, the problem stopped! Firefox never crashed again! However, to fulfill your request, I would need to install Warsaw again, so I will do the following, I will create a virtual machine with Windows 10 of the same version and patch that I have on the host machine, install Warsaw, and use your version of firefox . Can be? I return in a few days.

Thanks!

Thank you for the update! That's a great news that the problem stopped after Warsaw was installed. Yes, using a VM is perfect. Thank you for agreeing to test the patch, too.

Comment 33

[Francisco B]

(In reply to Toshihito Kikuchi [:toshi] from comment #32)

(In reply to Francisco B from comment #31)

Hello Toshihito Kikuchi!

First, sorry for the delay in responding.

The current situation is as follows: When you answered me, I had already uninstalled Warsaw, and to my surprise, the problem stopped! Firefox never crashed again! However, to fulfill your request, I would need to install Warsaw again, so I will do the following, I will create a virtual machine with Windows 10 of the same version and patch that I have on the host machine, install Warsaw, and use your version of firefox . Can be? I return in a few days.

Thanks!

Thank you for the update! That's a great news that the problem stopped after Warsaw was installed. Yes, using a VM is perfect. Thank you for agreeing to test the patch, too.

Hello Toshihito Kikuchi!

Since that day I have performed numerous tests inside the virtual machine that I proposed, and I must already inform you that unfortunately (or fortunately) there was no crash in any of the situations.
For knowledge, everything I tested followed the script below:

• I installed Windows Windows 10 Education x64, version 20H2, which is the most updated patch.
• I installed the standard firefox version 83.0 (64-bits).
• Installed Warsaw from the same downloaded from the same place where I had installed it on my machine, that is, through my bank's website (Caixa Federal - https://www.caixa.gov.br).
  I can't say if the version of Warsaw that the bank distributes is the same version as the official website that Diebold distributes (https://www.dieboldnixdorf.com.br/warsaw/).
  My bank file has a different name (and when installing colors and custom bank layout), however, during the days of trouble on my main machine, weeks ago, I had also removed Warsaw from my bank and installed the Warsaw downloaded from the official website (Diebold), and the crashes continued, that is, it happened with both installers.

Continuing on the virtual machine, after these downloads and installations, I left firefox running with many static and interactive tabs, like youtube, twitch, news sites, etc. I left videos running for more than 18 hours these days, interacting a few times, clicking, closing, even killing the firefox process through the "taskkill" command, I entered my internet banking and had no errors.
After the first 6 hours of use, and no errors, I installed the Kaspersky Total Security suite, just like I have on my main machine, I also installed the browser plugin, and all the other extensions I use on the main Firefox, and no errors .

• After these 18 hours, I installed the official version of firefox nightly as you requested. After 3 hours of playing videos and browsing, no error.
• After these 3 hours, I started the test with its version (.zip) of firefox nightly, which has been running for 3 hours, on several sites, and no errors so far.

Note: I checked in Windows services that Warsaw has always been running, even if it wasn't, I wouldn't have been able to access my bank.

I don't know what else I can do to try to reproduce the error. I'll try for a few more days, but if I can help in any other way, I'm happy to help!

Thanks!

Comment 34

[Toshihito Kikuchi [:toshi]]

Thank you for verifying the build thoroughly! This is really helpful. I understand this result did not prove the crash was really mitigated by my patch, but I'm glad that no critical regression was observed in those scenarios. I think we're good to go ahead and land this patch.

Comment 35

[Toshihito Kikuchi [:toshi]]

Attachment: Bug 1644240 - Suppress playing a sound if Diebold Warsaw's modules are partially unloaded. r=cmartin

This patch is to mitigate the crash which was probably caused by Diebold Warsaw.
We couldn't reproduce the problem, but our crash reports indicate the crash happened
when winmm!mciwindow called USER32!GetMessageA but it was redirected to a freed
buffer. This happens Firefox calls to PlaySound e.g. showing the menu bar by
pressing Alt key, or showing a dialogbox.

Most of (not 100%) the crash instances have wslbdhm64 loaded but wslbscrwh64.dll
was unloaded. The proposed mitigation is to suppress playing a sound under
such a condition.

Comment 36

[Ryan VanderMeulen [:RyanVM]]

This is showing up in the early Fx84 crash signatures too. We may want to take this in a dot release if the workaround is low-risk.

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/95c5bdd6c712
Suppress playing a sound if Diebold Warsaw's modules are partially unloaded.  r=cmartin

Comment 38

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/95c5bdd6c712

Comment 39

[Ryan VanderMeulen [:RyanVM]]

Please nominate this for Beta/Release/ESR78 approval when you get a chance.

Comment 40

[Chris Martin [:cmartin]]

Comment on attachment 9193101 [details]
Bug 1644240 - Suppress playing a sound if Diebold Warsaw's modules are partially unloaded. r=cmartin

Beta/Release Uplift Approval Request

• User impact if declined: Users with Diebold Warsaw installed on their machines may continue to experience crashes
• Is this code covered by automated tests?: Unknown
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's a small, easy-to-reason-about change, and "not playing a sound" is a lot less disruptive than crashing + potentially creating a security exploit.
• String changes made/needed:

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: It has caused many crashes
• User impact if declined: Users with Diebold Warsaw installed on their machines may continue to experience crashes
• Fix Landed on Version: Nightly
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's a small, easy-to-reason-about change, and "not playing a sound" is a lot less disruptive than crashing + potentially creating a security exploit.
• String or UUID changes made by this patch:

Comment 41

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9193101 [details]
Bug 1644240 - Suppress playing a sound if Diebold Warsaw's modules are partially unloaded. r=cmartin

Approved for 85.0b4, thanks!

Comment 42

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/6e7c522ae344

Comment 43

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9193101 [details]
Bug 1644240 - Suppress playing a sound if Diebold Warsaw's modules are partially unloaded. r=cmartin

Fix looks good on Nightly and Beta, thanks for the easy fix! Approved for 84.0.1.

Comment 44

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/21ddaecbbe4e

Comment 45

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9193101 [details]
Bug 1644240 - Suppress playing a sound if Diebold Warsaw's modules are partially unloaded. r=cmartin

Approved for 78.7esr.

Comment 46

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/f5259220ccf8