1833793 - Crash in [@ RtlFreeHeap | eoppbrowser.dll | <unknown in ole32.dll> | eoppbrowser.dll | RtlLockHeap | GlobalSize]

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect, P2)

Description

[Ryan VanderMeulen [:RyanVM]]

All on Windows 7.

Crash report: https://crash-stats.mozilla.org/report/index/6140244f-d0a8-4d4b-b966-46e020230518

Reason: EXCEPTION_ACCESS_VIOLATION_READ

Top 10 frames of crashing thread:

0  ntdll.dll  RtlFreeHeap  
1  eOppBrowser.dll  eOppBrowser.dll@0x5fb5  
2  ole32.dll  <unknown in ole32.dll>  
3  eOppBrowser.dll  eOppBrowser.dll@0xe654  
4  ntdll.dll  RtlLockHeap  
5  kernel32.dll  GlobalSize  
6  eOppBrowser.dll  eOppBrowser.dll@0xc5ec  
7  eOppBrowser.dll  eOppBrowser.dll@0x1d71f  
8  eOppBrowser.dll  eOppBrowser.dll@0x1d737  
9  xul.dll  _tailMerge_d3dcompiler_47.dll  

Comment 1

[Ryan VanderMeulen [:RyanVM]]

eOppBrowser.dll belongs to ESET Endpoint Security

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Adding a few more related-looking signatures, but there's also a pretty long tail of others in this DLL... :-\

Comment 3

[Greg Stoll :gstoll]

This seems pretty bad.

Lots of the comments indicate that pasting into Firefox triggers this. (possibly copying text also, possibly specific to the address bar?)

Comment 4

[Greg Stoll :gstoll]

I can't repro this on Windows 11 (not a surprise given that 100% of these crashes are on Windows 7) - going to try to get a Windows 7 machine.

Comment 5

[Greg Stoll :gstoll]

FWIW the versions of eoppbrowser.dll in this bucket are 1.0.87.0 and 1.0.88.0. When I installed ESET today 1.0.88.0 is the current version.

Comment 6

[Greg Stoll :gstoll]

I pinged our ESET contact, hopefully we will hear back. If not we should consider blocking eoppbrowser.dll on Windows 7 only.

Comment 7

[Greg Stoll :gstoll]

We have not heard back from ESET, and while the crash rate has been going down it's still relatively high. I've confirmed that 1.0.88.0 is still the newest version of ESET (based on telemetry), so I'm going to block this version and older on Windows 7 only.

Comment 8

[Greg Stoll :gstoll]

Attachment: Bug 1833793 - block ESET security DLL on Windows 7 r=gsvelto

Comment 9

[Greg Stoll :gstoll]

Windows DLLBlocklist request form

1. How were we aware of the problem?
   topcrasher

2. What is a suspicious product causing the problem?
   ESET Security

3. Is the product downloadable? If so, do we have a local repro?
   It is downloadable. On my Windows 11 machine, eplgBrowser.dll loads into Firefox, but not eoppbrowser.dll. On a Windows 7 VM, no ESET dll loads into Firefox, so in neither case can I reproduce the issue.

4. Which OS versions does the problem occur on?
   Windows 7

5. Which process types does the problem occur on?
   Only in the browser process.

6. What is the maximum version of the module in the crash reports?
   1.0.88.0.

7. Is the issue fixed by a newer version of the product?
   According to telemetry, 1.0.88.0 is the newest version of eoppbrowser.dll.

8. Do we have data about the module in the third-party-module ping?
   Yes

9. Do we know how the module is loaded?
   From a sampling it looks like it's loaded directly via LdrpInitializeProcess - presumably this is being injected somehow. I think this should be OK to block.

10. Describe your conclusion.
    Ideally, we would block versions 1.0.88.0 and earlier of eoppbrowser.dll in the browser process on Windows 7 only. However, we can't do this because of bug 1835926 - we can only limit it to just Windows 7 or just the browser process. So instead we will block versions 1.0.88.0 and earlier of eoppbrowser.dll on Windows 7 only.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bbe30bdf2ac8
block ESET security DLL on Windows 7 r=gsvelto

Comment 11

[Greg Stoll :gstoll]

My plan is to leave this bug open until this fix makes it to the Beta train next week. Since we have a reasonable number of crashes in Beta from this, we should be able to feel confident that this fixes the crash.

Comment 12

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/bbe30bdf2ac8

Comment 13

[Greg Stoll :gstoll]

The data shows that this crash is greatly reduced in 115 beta. I'm going to request uplift to release.

Comment 14

[Greg Stoll :gstoll]

Attachment: Bug 1833793 - block ESET security DLL on Windows 7

Original Revision: https://phabricator.services.mozilla.com/D179484

Comment 15

[Phabricator Automation]

Uplift Approval Request

• User impact if declined: crashes when pasting from clipboard on Win 7 with ESET
• Fix verified in Nightly: yes
• Risk associated with taking this patch: low
• String changes made/needed: no
• Is Android affected?: no
• Needs manual QE test: no
• Code covered by automated testing: no
• Explanation of risk level: already been in nightly and beta
• Steps to reproduce for manual QE testing: n/a

Comment 16

[Pascal Chevrel:pascalc]

https://hg.mozilla.org/releases/mozilla-release/rev/4ec7224506b1abe0bb004cba2b0d8512274e2e7a

Comment 17

[Greg Stoll :gstoll]

The block doesn't seem to be working, as we're still getting a bunch of these crashes. A contact at ESET has reached out to me, and they haven't been able to reproduce this but have some potential fixes in version 1.0.90.0 of eoppbrowser.dll, which should release next week. I'll be keeping an eye on crash telemetry to see if that's the case.

Comment 18

[Greg Stoll :gstoll]

FWIW I haven't been able to figure out why the block isn't working. eoppbrowser.dll doesn't attempt to load in my dev build, and I can't seem to debug an installed version to figure out why it's able to load there. It is noteworthy that when eoppbrowser.dll loads it loads very early. (right after ntdll.dll and before kernel32.dll)

Comment 19

[Greg Stoll :gstoll]

So far we haven't seen any crashes with version 1.0.90.0 of eoppbrowser.dll. Going to keep this bug open a bit longer to monitor.

Comment 20

[Greg Stoll :gstoll]

Telemetry still shows no crashes with version 1.0.90.0, even though a decent-sized chunk of Windows 7 users are running with that version now. So I'm pretty convinced that version fixes this crash, and I'm going to close this bug.

Comment 21

[Yannis Juglaret [:yannis]]

This is still high volume in release with eOppBrowser.dll version 1.0.90.0. The crashes are listed under a new signature.

Comment 22

[Yannis Juglaret [:yannis]]

Example stack in WinDbg with eOppBrowser version 1.0.90.0 loaded at 0x7fef7ca0000:

 # Child-SP          RetAddr               Call Site
00 00000000`009bcb20 000007fe`f7ca6136     ntdll!RtlFreeHeap+0x1a5
01 00000000`009bcba0 000007fe`f7caea65     eOppBrowser+0x6136
02 00000000`009bcbd0 000007fe`f7cae73a     eOppBrowser!DebugBreak+0x7175
03 00000000`009bcc20 000007fe`f7ca2928     eOppBrowser!DebugBreak+0x6e4a
04 00000000`009bcc60 000007fe`f7caca08     eOppBrowser+0x2928
05 00000000`009bcc90 000007fe`fda689fb     eOppBrowser!DebugBreak+0x5118
06 00000000`009bde50 000007fe`fda9904e     ole32!CreateWrapperClipDataObjectFromFormatsArray+0x5b [d:\w7rtm\com\ole32\ole232\clipbrd\clipapi.cpp @ 2908] 
07 00000000`009bde90 000007fe`c72110b9     ole32!OleGetClipboard+0x9e [d:\w7rtm\com\ole32\ole232\clipbrd\clipapi.cpp @ 2581] 
08 (Inline Function) --------`--------     xul!RepeatedlyTry+0x6 [/builds/worker/checkouts/gecko/widget/windows/nsClipboard.cpp @ 460] 
09 (Inline Function) --------`--------     xul!RepeatedlyTryOleGetClipboard+0x6 [/builds/worker/checkouts/gecko/widget/windows/nsClipboard.cpp @ 1285] 
0a 00000000`009bdec0 000007fe`c71acf4e     xul!nsClipboard::GetNativeClipboardData+0xa9 [/builds/worker/checkouts/gecko/widget/windows/nsClipboard.cpp @ 1333] 
0b 00000000`009bdff0 000007fe`c6f52c06     xul!nsBaseClipboard::GetData+0x7e [/builds/worker/checkouts/gecko/widget/nsBaseClipboard.cpp @ 214] 
0c 00000000`009be050 000007fe`c43183f6     xul!mozilla::dom::ContentParent::RecvGetClipboard+0xd6 [/builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp @ 3518] 
0d 00000000`009be0d0 000007fe`c3e142d2     xul!mozilla::dom::PContentParent::OnMessageReceived+0x1a56 [/builds/worker/workspace/obj-build/ipc/ipdl/PContentParent.cpp @ 16935] 
0e 00000000`009be310 000007fe`c4cf1c72     xul!mozilla::ipc::MessageChannel::DispatchSyncMessage+0xa2 [/builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp @ 1778] 
0f (Inline Function) --------`--------     xul!mozilla::ipc::MessageChannel::DispatchMessage+0x909 [/builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp @ 1734] 
10 (Inline Function) --------`--------     xul!mozilla::ipc::MessageChannel::RunMessage+0x9cf [/builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp @ 1536] 
11 00000000`009be3c0 000007fe`c4f0491f     xul!mozilla::ipc::MessageChannel::MessageTask::Run+0xa82 [/builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp @ 1643] 
12 (Inline Function) --------`--------     xul!mozilla::RunnableTask::Run+0x11 [/builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp @ 555] 
13 00000000`009be740 000007fe`c4f03e14     xul!mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal+0x78f [/builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp @ 879] 
14 (Inline Function) --------`--------     xul!mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal+0xb [/builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp @ 702] 

A variation (same version, loaded at 0x7fef84b0000):

 # Child-SP          RetAddr               Call Site
00 00000000`008cc7f0 000007fe`f84b6136     ntdll!RtlFreeHeap+0x1a5
01 00000000`008cc870 000007fe`f84bea65     eOppBrowser+0x6136
02 00000000`008cc8a0 000007fe`f84be73a     eOppBrowser!DebugBreak+0x7175
03 00000000`008cc8f0 000007fe`f84b2928     eOppBrowser!DebugBreak+0x6e4a
04 00000000`008cc930 000007fe`f84bca08     eOppBrowser+0x2928
05 00000000`008cc960 000007fe`fec6e060     eOppBrowser!DebugBreak+0x5118
06 00000000`008cdb20 000007fe`fec6fa08     ole32!CClipDataObject::GetRealDataObjPtr+0x50 [d:\w7rtm\com\ole32\ole232\clipbrd\clipdata.cpp @ 258] 
07 00000000`008cdb50 000007fe`f15cf4ba     ole32!CClipDataObject::GetData+0x78 [d:\w7rtm\com\ole32\ole232\clipbrd\clipdata.cpp @ 1397] 
08 (Inline Function) --------`--------     xul!RepeatedlyTryGetData::<lambda_11>::operator()+0x16 [/builds/worker/checkouts/gecko/widget/windows/nsClipboard.cpp @ 633] 
09 (Inline Function) --------`--------     xul!RepeatedlyTry+0x16 [/builds/worker/checkouts/gecko/widget/windows/nsClipboard.cpp @ 460] 
0a (Inline Function) --------`--------     xul!RepeatedlyTryGetData+0x3e [/builds/worker/checkouts/gecko/widget/windows/nsClipboard.cpp @ 632] 
0b 00000000`008cdb90 000007fe`f15cf746     xul!nsClipboard::FillSTGMedium+0xda [/builds/worker/checkouts/gecko/widget/windows/nsClipboard.cpp @ 653] 
0c 00000000`008cdc20 000007fe`f15d03e2     xul!nsClipboard::GetNativeDataOffClipboard+0xb6 [/builds/worker/checkouts/gecko/widget/windows/nsClipboard.cpp @ 687] 
0d 00000000`008cdda0 000007fe`f15d140b     xul!nsClipboard::GetDataFromDataObject+0x232 [/builds/worker/checkouts/gecko/widget/windows/nsClipboard.cpp @ 951] 
0e 00000000`008cde70 000007fe`f156d2de     xul!nsClipboard::GetNativeClipboardData+0x13b [/builds/worker/checkouts/gecko/widget/windows/nsClipboard.cpp @ 1337] 
0f 00000000`008cdfa0 000007fe`f13130e6     xul!nsBaseClipboard::GetData+0x7e [/builds/worker/checkouts/gecko/widget/nsBaseClipboard.cpp @ 214] 
10 00000000`008ce000 000007fe`ee6db286     xul!mozilla::dom::ContentParent::RecvGetClipboard+0xd6 [/builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp @ 3518] 

We seem to crash in a hook set by ESET on GetClipboardData.

Comment 23

[Yannis Juglaret [:yannis]]

(In reply to Greg Stoll from comment #18)

FWIW I haven't been able to figure out why the block isn't working. eoppbrowser.dll doesn't attempt to load in my dev build, and I can't seem to debug an installed version to figure out why it's able to load there. It is noteworthy that when eoppbrowser.dll loads it loads very early. (right after ntdll.dll and before kernel32.dll)

This would probably have been caused by the broken DLL blocklist on Win 7, fixed in bug 1837242 (unless it is ESET themselves bypassing the blocklist?). Anyway, the volume is concerningly high again now that 1.0.90.0 is widespread. I think we should definitely try to block 1.0.90.0 for 116 release and 115.0.4esr.

Comment 24

[Greg Stoll :gstoll]

Attachment: Bug 1833793 - block newer versions of ESET DLL on Windows 7 r=gsvelto

Comment 25

[Yannis Juglaret [:yannis]]

We have investigated how Firefox reacts to blocking eoppbrowser.dll when ESET is installed with [:gstoll], and it does not seem to be a good idea to do that. It seems like it would result in failure to start with a launcher process, followed by a proper start without a launcher process but with eoppbrowser.dll still present.

On another front, this issue seems tied to a specific update level for Windows 7 x64 (KB4474419), since all these crashes seem to have version 6.1.7601.24384 of ntdll.dll. That is the second most common version for ntdll.dll in Windows 7 x64 crashes in general, on par with 6.1.7601.17514, whereas the most common would be 6.1.7601.24545. I have forwarded this extra piece of information to ESET which may help them reproduce or identify the issue.

Comment 26

[Yannis Juglaret [:yannis]]

We are still receiving crashes with eOppBrowser version 1.0.91.0 (e.g. this one).

Comment 27

[Yannis Juglaret [:yannis]]

We are in touch with ESET about this bug and with their help, I have been able to install ESET on a Windows 7 x64 6.1.7601.24384 VM and to reproduce the crash somewhat consistently and analyze what happens. The root cause seems to be a bug in the NT kernel ntoskrnl.exe in version 6.1.7601.24384.

NtQueryVirtualMemory can be used with a non-documented MemoryInformationClass of 2 to get the section file name associated with a virtual memory address. We use that call, and ESET too. To get the length of the section file name, we can pass in a pointer to a ReturnLength that will receive it. When that occurs, the return value for the system call will be STATUS_INFO_LENGTH_MISMATCH (0xc0000004), unless we are also providing a pointer to a MemoryInformationBuffer and its size MemoryInformationLength such that the size of the buffer is big enough to contain the name (in this case we get a NT_SUCCESS value). Despite its name, STATUS_INFO_LENGTH_MISMATCH is a NT_ERROR and not a NT_INFORMATION.

In ntoskrnl.exe in version 6.1.7601.24384, NtQueryVirtualMemory is implemented such that the ReturnLength is only filled when the system call returns a NT_SUCCESS return value (or a NT_INFORMATION). When a NT_ERROR is returned, the ReturnLength is not filled.

In a normal kernel, you would find a block like this one in NtQueryVirtualMemory:

CALL       ObQueryNameStringMode
MOV        EBX, EAX                    # ebx = ntStatus from ObQueryNameStringMode
MOV        RCX, R13
CALL       ObfDereferenceObject
MOV        RDX, qword ptr [RSP + 0x60] # rdx = ReturnLength
TEST       RDX, RDX                    # early exit if ReturnLength == NULL
JZ         exit
MOV        ECX, dword ptr [RSP + 0x48] # ecx = SectionNameLength
MOV        qword ptr [RDX],RCX         # *ReturnLength = SectionNameLength
JMP        exit

But in ntoskrnl.exe version 6.1.7601.24384 you will find the following instead:

CALL       ObQueryNameString
MOV        EBX, EAX                    # ebx = ntStatus from ObQueryNameString
MOV        RCX, R12
CALL       ObfDereferenceObject
TEST       EBX, EBX                    # early exit if !(STATUS_SUCCESS(ntStatus))
JS         exit
TEST       R14, R14                    # early exit if ReturnLength == NULL
JZ         exit
MOV        ECX, dword ptr [RSP + 0x50] # ecx = SectionNameLength
MOV        qword ptr [R14], RCX        # *ReturnLength = SectionNameLength
JMP        exit

ESET's hook for NtMapViewOfSection uses that system call and expects that the ReturnLength is always set when calling NtQueryVirtualMemory with a MemoryInformationClass of 2, when the returned value is the NT_ERROR of value STATUS_INFO_LENGTH_MISMATCH. This is true with most kernels, but not in version 6.1.7601.24384, thus causing their code to use an uninitialized value to compute the allocation size for the section name buffer, and then perform an out-of-bounds write if the computed size is strictly less than 16 bytes.

I have forwarded the details to ESET and I hope that this will allow them to fix the crash soon.

Comment 28

[Greg Stoll :gstoll]

ESET believes that this is fixed in eoppbrowser.dll version 1.0.93.0, which is rolling out now. I will keep an eye on telemetry to verify this.

Comment 29

[Greg Stoll :gstoll]

We haven't seen any crashes with eoppbrowser.dll version 1.0.93.0 with any suspicious-looking signature. Closing this bug.