Closed Bug 1758641 Opened 2 years ago Closed 1 year ago

Crash in [@ RtlpFreeHeapInternal | RtlFreeHeap | wslbdhm64.dll | RtlFreeHeap] linked to Topaz OFD Warsaw, previously Diebold Warsaw

Categories

(External Software Affecting Firefox :: Other, defect)

Unspecified
Windows
defect

Tracking

(firefox108 wontfix, firefox109 wontfix, firefox110 affected)

RESOLVED FIXED
Tracking Status
firefox108 --- wontfix
firefox109 --- wontfix
firefox110 --- affected

People

(Reporter: gsvelto, Unassigned)

Details

(Keywords: crash, Whiteboard: [win:stability])

Crash Data

Crash report: https://crash-stats.mozilla.org/report/index/8a3733d0-a77c-4abd-bb9a-67b5d0220309

Reason: STATUS_HEAP_CORRUPTION

Top 10 frames of crashing thread:

0 ntdll.dll RtlReportFatalFailure 
1 ntdll.dll RtlReportCriticalFailure 
2 ntdll.dll RtlpHeapHandleError 
3 ntdll.dll RtlpHpHeapHandleError 
4 ntdll.dll RtlpLogHeapFailure 
5 ntdll.dll RtlpFreeHeapInternal 
6 ntdll.dll RtlFreeHeap 
7 wslbdhm64.dll wslbdhm64.dll@0x00000000000350cb 
8 wslbdhm64.dll wslbdhm64.dll@0x000000000005dbbb 
9 wslbdhm64.dll wslbdhm64.dll@0x000000000005ee74 

This is caused by a an injected module which could be related to Brazilian government institutions. It could be an authentication module, smartcard reader plug-in or something similar; we've already encountered several of this. One of the comments mentions trying to log in into the web page of a Brasilian bank. Unfortunately I couldn't track down the installer yet. I am adding the 10 topmost signatures as the volume is non-trivial.

To find all the crashes caused by this module this query can be used.

This is still a problem, adding a relatively high volume signature.

Crash Signature: RtlpAllocateHeapInternal | CreateStringCopy_AnsiToUnicode] [@ RtlFreeHeap | wslbdhm64.dll] [@ RtlpLowFragHeapAllocFromContext | RtlpAllocateHeapInternal | wslbdhm64.dll] → RtlpAllocateHeapInternal | CreateStringCopy_AnsiToUnicode] [@ RtlFreeHeap | wslbdhm64.dll] [@ RtlpLowFragHeapAllocFromContext | RtlpAllocateHeapInternal | wslbdhm64.dll] [@ RtlpFreeHeapInternal | RtlFreeHeap | wslbdhm64.dll | RtlpFreeHeapInternal | ws…

Gabriele, someone posted about this on https://www.reddit.com/r/firefox/comments/101y3t1/firefox_daily_crashingfreezing_at_least_one_time/ in regards to crash bp-9009408c-f6cd-4b10-99fb-9fb530230103 and guessed that wslbdhm64.dll is related to Diebold Warsaw, which apparently is required by Brazilian banks for customers.

Bug 1644240 also references this dll and software. Hopefully this gives you a lead on how to diagnose these issues further.

Adding another signature, I'll see if someone in my team has some spare cycles to look into this.

Crash Signature: wslbdhm64.dll | RtlFreeHeap] → wslbdhm64.dll | RtlFreeHeap] [@ RtlpLowFragHeapAllocFromContext | RtlpAllocateHeapInternal | wslbdhm64.dll | RtlpFreeUserBlock]

I was able to get a copy of wslbdhm64.dll version 1.1.1.24 by installing the latest version of Guardião 30 horas in a VM, this is the version of the module found in most (all?) crashes.

I have analyzed the most recurrent crash thanks to the DLL. The heap corruption is reported as a double free. I have identified that the last called functions in the stack belong to the statically-linked JsonCpp library. I have also restored the Firefox part of the stack. The crash occurs in third-party code that gets called through PR_GetAddrInfoByName, they are probably hooking getaddrinfo from ws2_32.dll or something like this. The full call stack looks like this:

00 000000cd`343ce370 00007ffb`951cf673     ntdll!RtlReportFatalFailure+0x9
01 000000cd`343ce3c0 00007ffb`951d83f2     ntdll!RtlReportCriticalFailure+0x97
02 000000cd`343ce4b0 00007ffb`951d86da     ntdll!RtlpHeapHandleError+0x12
03 000000cd`343ce4e0 00007ffb`951de361     ntdll!RtlpHpHeapHandleError+0x7a
04 000000cd`343ce510 00007ffb`950f5bf0     ntdll!RtlpLogHeapFailure+0x45
05 000000cd`343ce540 00007ffb`950f47b1     ntdll!RtlpFreeHeapInternal+0x4e0
06 000000cd`343ce600 00007ffb`542850cc     ntdll!RtlFreeHeap+0x51
 wslbdhm64+0x350cc _free_base
 wslbdhm64+0x5eeb8 Json::Value::~Value
 wslbdhm64+0x5ef84 Json::Value::"DestructiveMoveAssignment"
 wslbdhm64+0x657a6 DoStuff
 wslbdhm64+0x65e8b
 wslbdhm64+0x1de43
 wslbdhm64+0x1d9d8
 wslbdhm64+0x160db
 wslbdhm64+0x6829e
 wslbdhm64+0x64faf
 [ dynamic code ]
00 00000016`760dede0 00007ffb`49b4f32a     nss3!PR_GetAddrInfoByName+0xc3 [/builds/worker/checkouts/gecko/nsprpub/pr/src/misc/prnetdb.c @ 2171]
01 (Inline Function) --------`--------     xul!mozilla::net::_GetAddrInfo_Portable+0x5f [/builds/worker/checkouts/gecko/netwerk/dns/GetAddrInfo.cpp @ 241]
02 00000016`760dee60 00007ffb`4a2a284b     xul!mozilla::net::GetAddrInfo+0x15a [/builds/worker/checkouts/gecko/netwerk/dns/GetAddrInfo.cpp @ 374]
03 00000016`760df0a0 00007ffb`4a1e4471     xul!nsHostResolver::ThreadFunc+0x43b [/builds/worker/checkouts/gecko/netwerk/dns/nsHostResolver.cpp @ 1727]
04 (Inline Function) --------`--------     xul!mozilla::detail::RunnableMethodArguments<>::applyImpl+0x9 [/builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h @ 1147]
05 (Inline Function) --------`--------     xul!mozilla::detail::RunnableMethodArguments<>::apply+0x9 [/builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h @ 1153]
06 00000016`760df200 00007ffb`4a211f3e     xul!mozilla::detail::RunnableMethodImpl<RefPtr<nsObserverService>,void (nsObserverService::*)(),1,0>::Run+0x21 [/builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h @ 1203]
07 00000016`760df230 00007ffb`4b1c1aa8     xul!nsThreadPool::Run+0x29e [/builds/worker/checkouts/gecko/xpcom/threads/nsThreadPool.cpp @ 311]
08 (Inline Function) --------`--------     xul!nsThread::ProcessNextEvent+0x17b0 [/builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp @ 1198]
09 00000016`760df3d0 00007ffb`4b1ad48f     xul!NS_ProcessNextEvent+0x1808 [/builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp @ 466]
0a 00000016`760df760 00007ffb`4a3aba5f     xul!mozilla::ipc::MessagePumpForNonMainThreads::Run+0xcf [/builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp @ 301]
0b (Inline Function) --------`--------     xul!MessageLoop::RunInternal+0x16 [/builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc @ 381]
0c 00000016`760df800 00007ffb`499ef54e     xul!MessageLoop::RunHandler+0x2f [/builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc @ 375]
0d 00000016`760df850 00007ffb`4a20d757     xul!MessageLoop::Run+0x4e [/builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc @ 357]
0e 00000016`760df8b0 00007ffb`63d244bc     xul!nsThread::ThreadFunc+0xd7 [/builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp @ 385]
0f 00000016`760dfa60 00007ffb`63d39241     nss3!_PR_NativeRunThread+0x13c [/builds/worker/checkouts/gecko/nsprpub/pr/src/threads/combined/pruthr.c @ 421]
10 00000016`760dfae0 00007ffb`c2701bb2     nss3!pr_root+0x11 [/builds/worker/checkouts/gecko/nsprpub/pr/src/md/windows/w95thred.c @ 140]
11 00000016`760dfb10 00007ffb`c3657614     ucrtbase!thread_start<unsigned int (__cdecl*)(void *),1>+0x42
12 00000016`760dfb40 00007ffb`9782e8a8     kernel32!BaseThreadInitThunk+0x14
13 (Inline Function) --------`--------     mozglue!mozilla::interceptor::FuncHook<mozilla::interceptor::WindowsDllInterceptor<mozilla::interceptor::VMSharingPolicyShared>,void (*)(int, void *, void *)>::operator()+0x15 [/builds/worker/checkouts/gecko/toolkit/xre/dllservices/mozglue/nsWindowsDllInterceptor.h @ 150]
14 00000016`760dfb70 00007ffb`c4b426a1     mozglue!patched_BaseThreadInitThunk+0x28 [/builds/worker/checkouts/gecko/toolkit/xre/dllservices/mozglue/WindowsDllBlocklist.cpp @ 587]
15 00000016`760dfbe0 00000000`00000000     ntdll!RtlUserThreadStart+0x21

From analyzing the code in the DLL, I believe the code is doing a move assignment to a subvalue of a Json::Value stored as a global or static variable (that's what I called DestructiveMoveAssignment in the call stack), like the assignment in the ThreadFunc below:

#include "json/json.h"
#include <windows.h>

constexpr auto nThreads = 2;
constexpr auto nIterations = 0x1000;

Json::Value gValue;

DWORD ThreadFunc(void* aParam)
{
    for (size_t i = 0; i < nIterations; ++i) {
        gValue[Json::String("")] = Json::Value("hello");
    }
    return 0;
}

int main() {
    auto bResult = HeapSetInformation(NULL, HeapEnableTerminationOnCorruption, NULL, 0);
    if (!bResult) {
        return 1;
    }
    HANDLE threads[nThreads]{};
    for (auto& thread : threads) {
        DWORD threadId = 0;
        thread = CreateThread(nullptr, 0, ThreadFunc, nullptr, 0, &threadId);
        if (!thread) {
            return 1;
        }
    }
    WaitForMultipleObjects(nThreads, threads, TRUE, INFINITE);
    return 0;
}

Such a move assignment is not thread-safe, as the example program illustrates; the example program forces the race condition and crashes with a double free heap corruption with a similar call stack (the call to releasePayload and releasePrefixedStringValue are inlined but present in the original crash):

00 00000038`b59ff3e0 00007fff`5989c213     ntdll!RtlReportFatalFailure+0x9
01 00000038`b59ff430 00007fff`598a52aa     ntdll!RtlReportCriticalFailure+0x97
02 00000038`b59ff520 00007fff`598a558a     ntdll!RtlpHeapHandleError+0x12
03 00000038`b59ff550 00007fff`598b1585     ntdll!RtlpHpHeapHandleError+0x7a
04 00000038`b59ff580 00007fff`597cc59c     ntdll!RtlpLogHeapFailure+0x45
05 00000038`b59ff5b0 00007fff`597cb1e1     ntdll!RtlpFreeHeapInternal+0x84c
06 00000038`b59ff670 00007fff`56c637eb     ntdll!RtlFreeHeap+0x51
07 00000038`b59ff6b0 00007ff7`49f34604     ucrtbase!_free_base+0x1b
08 00000038`b59ff6e0 00007ff7`49f354be     MyJson!Json::releasePrefixedStringValue+0x14 [C:\repos\MyJson\packages\JsonCpp-source.1.0.2\build\native\src\json\json_value.cpp @ 182] 
09 00000038`b59ff710 00007ff7`49f350d3     MyJson!Json::Value::releasePayload+0x4e [C:\repos\MyJson\packages\JsonCpp-source.1.0.2\build\native\src\json\json_value.cpp @ 1018] 
0a 00000038`b59ff760 00007ff7`49f315e1     MyJson!Json::Value::~Value+0x13 [C:\repos\MyJson\packages\JsonCpp-source.1.0.2\build\native\src\json\json_value.cpp @ 443] 
0b 00000038`b59ff790 00007fff`58f726bd     MyJson!ThreadFunc+0xb1 [C:\repos\MyJson\MyJson\MyJson.cpp @ 13] 
0c 00000038`b59ff840 00007fff`597edfb8     KERNEL32!BaseThreadInitThunk+0x1d
0d 00000038`b59ff870 00000000`00000000     ntdll!RtlUserThreadStart+0x28

I will try to contact the vendor to confirm if they do have code that looks like this and if they could make their code thread-safe.

Can anyone confirm that PR_GetAddrInfoByName can indeed potentially get called simultaneously by different threads? (Well, the call stack suggests that this is exactly what we are doing here -- using a specific thread each time we need to resolve a hostname.)

Thanks!

(I haven't analyzed the other crashes for the moment.)

Summary: Crash in [@ RtlpFreeHeapInternal | RtlFreeHeap | wslbdhm64.dll | RtlFreeHeap] → Crash in [@ RtlpFreeHeapInternal | RtlFreeHeap | wslbdhm64.dll | RtlFreeHeap] linked to Topaz OFD Warsaw, previously Diebold Warsaw

From talking to users in Brazil, our understanding is that this software is essentially mandatory for home banking there, which means blocking the DLL, despite the huge crash rate, is very unattractive.

There was no response from the vendor at the general support contact, so we're trying to reach out to engineers working on this product directly.

Some preliminary tests with detecting the buggy DLL and enforcing it to lock on DNS lookups didn't look too promising from the performance perspective, so we have no good options here for now.

I got a response from one of the engineers and they told me they are working on this, so fingers crossed!

Sent another follow-up here asking if they have any status they can share.

This URL can be used to monitor which versions of wslbdhm64.dll and wslbdhm32.dll lead to crashes and the associated volume. We currently have two versions of wslbdhm64.dll there (and two of wslbdhm32.dll): version 1.1.1.24 (wslbdhm64.dll/45c698ff000441d082acd4938181e1961) and version 1.1.1.32 (wslbdhm64.dll/fb391fef58ae42248c6b16b6f762fa4f1). The first crashes for version 1.1.1.32 date back to 2023-01-13, so it was released before comment 7. Once a new version gets released and we know it has been propagated to (some) users, we can use that URL to double check if the crash was properly mitigated.

Whiteboard: [win:stability]

Warsaw comes in multiple variants that are rebranded with the colors and logo (and maybe more) of the customer bank. At the moment I identified that:

  • Banco do Brasil still uses version 1.1.1.32 of the DLL - Can be downloaded from Seg.BB - Diagnóstico do Módulo de Segurança by clicking Concordo near Módulo de Segurança.
  • Itau still uses version 1.1.1.24 of the DLL - Can be downloaded from Guardião 30 horas by clicking iniciar instalação under instale o Guardião Itaú no seu computador.
  • Caixa and Santander seem to require an account to download the product, so I do not know what version they are using.
Crash Signature: wslbdhm64.dll | RtlFreeHeap] [@ RtlpLowFragHeapAllocFromContext | RtlpAllocateHeapInternal | wslbdhm64.dll | RtlpFreeUserBlock] → wslbdhm64.dll | RtlFreeHeap] [@ RtlpLowFragHeapAllocFromContext | RtlpAllocateHeapInternal | wslbdhm64.dll | RtlpFreeUserBlock] [@ RtlpLowFragHeapAllocFromContext | RtlpAllocateHeapInternal | wslbdhm64.dll | RtlFreeHeap | wslbdhm64.dll] [@ RtlpFreeHea…

The crash rate was suggesting something good might have happened, so I checked this again, and the news is good indeed! Around the time of comment 9, Topaz OFD had warned us that they had fixed the issue thanks to the info from comment 5, but that propagation to clients would however take some time, because their clients each have their own release schedules. I can now confirm that the setup tool from Banco do Brasil installs wslbdhm64.dll version 1.1.1.34, a version for which we have so far received no crashes. This suggests that the issue was correctly identified in comment 5, and that Topaz OFD has successfully patched it. So I'm closing this bug. Note that the setup tool from Itau still installs version 1.1.1.24 which will continue to crash.

If as a user you are impacted by this crash, you should be able to block wslbdhm64.dll by following this guide to prevent crashes. However, doing this should prevent you from engaging in online banking activities with Firefox while the DLL is blocked, so please make sure to unblock the DLL before doing things related to banking. At any point, you can check your version of the DLL by navigating to about:third-party. The problem should be fixed once your bank uses version 1.1.1.34 or higher of the DLL.

Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Crash Signature: RtlpFreeHeapInternal] [@ RtlpReportHeapFailure | RtlpFreeHeapInternal | RtlFreeHeap | wslbdhm32.dll | MultiByteToWideChar] → RtlpFreeHeapInternal] [@ RtlpReportHeapFailure | RtlpFreeHeapInternal | RtlFreeHeap | wslbdhm32.dll | MultiByteToWideChar] [@ RtlpLowFragHeapAllocFromContext | RtlpAllocateHeapInternal | wslbdhm64.dll | RtlpAllocateHeapInternal | CreateStringCopy_AnsiT…
You need to log in before you can comment on or make changes to this bug.