1841751 - Startup crashes in dynamic code, correlated with the presence of seemingly malicious DLLs (Quoteex?)

Status: VERIFIED FIXED

(External Software Affecting Firefox :: Other, defect)

Description

[Yannis Juglaret [:yannis]]

Note for users: If you experience this crash, it is likely because you have malware installed.

Among the (early) top crashers for 115 release we have signatures that have no symbols because we crash in dynamic code. In all the crash reports, there is a suspicious loaded module, likely malware. The DLL name and path vary, but possibly not the version (version 1.0.0.29915) nor the timestamp (Sun Nov 19 11:31:29 2017 (5A115D81)) looking at a few examples. Here are some paths: C:\ProgramData\Voyasollam\Flex-Find.dll, C:\ProgramData\Quoteex\Biofan.dll, C:\ProgramData\Quoteex\Kontop.dll. Quoteex seems to be a known adware.

The crashes possibly occur because the dynamic code triggered from the malicious DLL would be reading xul.dll entirely in memory to search for patterns of interest, and consequently trying to read the .retplne section, which has PAGE_NOACCESS protection.

I am not sure what changed in 115 that explains that this crash was not occurring before.

Comment 1

[Yannis Juglaret [:yannis]]

(In reply to Yannis Juglaret [:yannis] from comment #0)

I am not sure what changed in 115 that explains that this crash was not occurring before.

In fact, xul.dll did not have a .retplne section in 114, hence no crash. Did we change of Windows SDK version with 115?

To summarize the problem, some malicious injected DLLs read xul.dll entirely in memory to search for patterns in it. They will crash when they hit the .retplne section because of its access rights, as it is not allowed to read these pages. This is the same root cause as bug 1837242.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

(In reply to Yannis Juglaret [:yannis] from comment #1)

In fact, xul.dll did not have a .retplne section in 114, hence no crash. Did we change of Windows SDK version with 115?

Bug 1832467 bumped us to VS2019 & WinSDK 10.0.19041

Comment 3

[Greg Stoll :gstoll]

Well, we could block this based on the timestamp, version, and path (everything starts with C:\ProgramData), but that doesn't seem ideal.

Comment 4

[Yannis Juglaret [:yannis]]

Iterating through the reports, I found that these are actually from 5 different versions of the same (renamed) DLL. Here is the info shown by WinDbg for each variant:

    Timestamp:        Mon Apr 11 13:29:20 2016 (570B8A90)
    CheckSum:         00000000
    ImageSize:        00062000
    File version:     1.0.0.26638

    Timestamp:        Thu May  5 15:38:44 2016 (572B4CE4)
    CheckSum:         00000000
    ImageSize:        00062000
    File version:     1.0.0.26793

    Timestamp:        Sun Aug  7 14:12:28 2016 (57A725AC)
    CheckSum:         00000000
    ImageSize:        00061000
    File version:     1.0.0.27567

    Timestamp:        Sun Nov 19 11:31:29 2017 (5A115D81)
    CheckSum:         00000000
    ImageSize:        0005D000
    File version:     1.0.0.29915

    Timestamp:        Tue Jun 11 12:55:52 2019 (5CFF88B8)
    CheckSum:         00000000
    ImageSize:        0005D000
    File version:     1.0.0.31122

I believe the combination of these four pieces of information (Timestamp, CheckSum=0, ImageSize, File version) should make collisions unlikely, so I prepared a patch to block the DLLs based on that and would be curious to have your opinion.

I also prepared a DLL that matches with the latest variation of the malicious DLL, which we can use for QA testing. Attached is the result in a custom build (right) compared to regular Nightly (left). Below is how WinDbg parses the info from the DLL:

start             end                 module name
00007fff`52280000 00007fff`522dd000   Delicious   (deferred)
    Image path: C:\repos\HookTests\x64\Release\Delicious.dll
    Image name: Delicious.dll
    Browse all global symbols  functions  data
    Timestamp:        Tue Jun 11 12:55:52 2019 (5CFF88B8)
    CheckSum:         00000000
    ImageSize:        0005D000
    File version:     1.0.0.31122
    Product version:  1.0.0.1
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    Information from resource tables:
        ProductVersion:   1.0.0.1
        FileVersion:      1.0.0.31122

Comment 5

[Yannis Juglaret [:yannis]]

Attachment: Blocking of a custom DLL that exposes the same information as the malicious one

Comment 6

[Yannis Juglaret [:yannis]]

Attachment: Bug 1841751 - Block instances of a malicious injected DLL causing startup crashes. r=gstoll,handyman,mhowell

With bug 1832467 we have updated our Windows SDK version to 10.0.19041.
As a result, we now have a .retplne section in xul.dll, starting with
Firefox 115. This is a section with PAGE_NOACCESS protection, so
accessing it crashes the process.

Some injected DLLs read the whole memory space dedicated to the xul.dll
image to search for patterns in it. When they hit the .retplne section,
we will crash. This happened both for a legit product, in bug 1837242,
but also for a malicious DLL, in bug 1841751.

This change blocks the variants of this malicious DLL, to eliminate the
associated crash spike. Because the DLL does not use a fixed name, we
block by matching on the combination of version number + timestamp +
image size, based on the values found in crash reports. We additionnally
check for a checksum of 0, which is rather uncommon for legit DLLs and
thus helps further reduce the chances of collision.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e31033710cfe
Block instances of a malicious injected DLL causing startup crashes. r=gstoll

Comment 8

[Yannis Juglaret [:yannis]]

Comment on attachment 9342955 [details]
Bug 1841751 - Block instances of a malicious injected DLL causing startup crashes. r=gstoll,handyman,mhowell

Beta/Release Uplift Approval Request

• User impact if declined: Inability to use Firefox 115 (constant startup crashes) for users who have the Quoteex malware installed.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Please reach me so that I can provide the full details. Below is a summary for people reading the bug.

Expected behavior:

• With custom executable TestHooks.exe (please ask)
• Launch TestHooks.exe
• Launch Firefox
• Open about:third-party
• Delicious.dll should be listed as blocked

• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This change is similar to adding entries in the DLL blocklist, except that because we do not have a name for the DLL, we do this with code instead of a regular blocklist entry, and we match some specific fields that characterize the DLL. We carefully chose the fields to avoid collision with other DLLs.
• String changes made/needed:
• Is Android affected?: No

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: The root cause for this crash also applies to ESR (building with new SDK).
• User impact if declined: Inability to use Firefox 115 ESR (constant startup crashes) for users who have the Quoteex malware installed.
• Fix Landed on Version: 115.0.2
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This change is similar to adding entries in the DLL blocklist, except that because we do not have a name for the DLL, we do this with code instead of a regular blocklist entry, and we match some specific fields that characterize the DLL. We carefully chose the fields to avoid collision with other DLLs.

Comment 9

[Donal Meehan [:dmeehan]]

Comment on attachment 9342955 [details]
Bug 1841751 - Block instances of a malicious injected DLL causing startup crashes. r=gstoll,handyman,mhowell

Approved for 115.0.2
Approved for 115.0.2esr

Comment 10

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/f208fe07dfb2

Comment 11

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/872b5aae170f

Comment 12

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/f1551ef129b1

Comment 13

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/e31033710cfe

Comment 14

[Bogdan Maris, Desktop Test Engineering]

I reproduced the crash with the correct signature using 115.0.1 (https://crash-stats.mozilla.org/report/index/60b583e0-f888-4b62-94d2-460c10230711) using the information I received from Yannis in private, thanks for the detailed info.
I can confirm that Firefox 115.0.2, 115.0.2esr and Latest Nightly 117.0a1 (with the fix) does not crash and the Delicious.dll is correctly shown as blocked in about:third-party without the possibility of users unblocking it from about:thirdparty.

Comment 15

[Bogdan Maris, Desktop Test Engineering]

Oh I was a bit fast on closing this bug as verified and removing qa-verify+, I'll make sure to verify the fix in Firefox 116 beta as well once it is uplifted.

Comment 16

[Dianna Smith [:diannaS]]

Comment on attachment 9342955 [details]
Bug 1841751 - Block instances of a malicious injected DLL causing startup crashes. r=gstoll,handyman,mhowell

Approved for 116.0b4

Comment 17

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/de5ae9e8d7dc

Comment 18

[Yannis Juglaret [:yannis]]

I'm marking this bug as Windows 10, so that users reading the 115.0.2 release notes can more easily know if they experienced this crash (where reports are from Windows 10 and 11) or the other crash from bug 1842368 (Windows 7). It's important that they easily find out, to help them figure out that they have malware installed if they do.

Comment 19

[Bogdan Maris, Desktop Test Engineering]

(In reply to Bogdan Maris, Desktop QA from comment #15)

Oh I was a bit fast on closing this bug as verified and removing qa-verify+, I'll make sure to verify the fix in Firefox 116 beta as well once it is uplifted.

Verified that this is fixed on Firefox 116.0b4 Windows 10 64bit.

Comment 20

[Yannis Juglaret [:yannis]]

The early reports after publishing 115.0.2 suggest that we have successfully fixed this issue for the vast majority of affected users. However, there will likely be a few reports coming from a small portion of these users (currently, 1).

This would be users for whom the launcher process fails to start. The current fix blocks the malicious DLL from within the launcher process blocklist code, so, if the launcher process fails to start, the DLL won't be blocked: we will be using the fallback mozglue blocklist code instead. I can confirm this by testing with pref browser.launcherProcess.enabled set to false.

I'm reopening this bug as a reminder that we are not yet completely done with this crash, although the volume should be very significantly diminished now. To fix this completely, we could port the patch to the fallback mozglue blocklist.

Comment 21

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 20 desktop browser crashes on release

For more information, please visit BugBot documentation.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

This bug had patches landed and uplifted. For the sake of better tracking, let's take any follow-up work to a new bug.