1772899 - Intermittent crash while fuzzing in ntdll.dll

Status: NEW

(Core :: DLL Services, defect)

Description

[Tyson Smith [:tsmith]]

This is hit by fuzzers a few times a week. Reports go back as far as 20220425-9cb38db713cc but could earlier. Potentially media related?

None of the test cases are reproducible. Not sure if there is anything that can be done but logging just in case.

==6244==ERROR: AddressSanitizer: access-violation on unknown address 0x0238a184d000 (pc 0x7ff88bb8c5c0 bp 0x003e10dffde0 sp 0x003e10dffc08 T16)
==6244==The signal is caused by a WRITE memory access.
    #0 0x7ff88bb8c5bf  (C:\Windows\SYSTEM32\ntdll.dll+0x1800ac5bf)
    #1 0x7ff87a139ab6 in __asan::AsanThread::ClearShadowForThreadStackAndTLS /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_thread.cpp:323
    #2 0x7ff87a139ab6 in __asan::AsanThread::Init(struct __asan::AsanThread::InitOptions const *) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_thread.cpp:240
    #3 0x7ff87a139d23 in __asan::AsanThread::ThreadStart(unsigned __int64) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_thread.cpp:264
    #4 0x7ff889d884d3  (C:\Windows\System32\KERNEL32.DLL+0x1800084d3)
    #5 0x7ff87b2e8b2c in mozilla::interceptor::FuncHook<mozilla::interceptor::WindowsDllInterceptor<mozilla::interceptor::VMSharingPolicyShared>,void (*)(int, void *, void *)>::operator() src/toolkit/xre/dllservices/mozglue/nsWindowsDllInterceptor.h:150
    #6 0x7ff87b2e8b2c in patched_BaseThreadInitThunk src/toolkit/xre/dllservices/mozglue/WindowsDllBlocklist.cpp:572
    #7 0x7ff88bb31790  (C:\Windows\SYSTEM32\ntdll.dll+0x180051790)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: access-violation (C:\Windows\SYSTEM32\ntdll.dll+0x1800ac5bf) 
Thread T16 created by T1 here:
    #0 0x7ff87a13af32 in __asan_wrap_CreateThread /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_win.cpp:146
    #1 0x7ff888bffa76  (C:\Windows\System32\ucrtbase.dll+0x18001fa76)
    #2 0x7ff879d3186d in _PR_MD_CREATE_THREAD src/nsprpub/pr/src/md/windows/w95thred.c:153
    #3 0x7ff879d5a46a in _PR_NativeCreateThread src/nsprpub/pr/src/threads/combined/pruthr.c:1058
    #4 0x7ff879d5ac03 in _PR_CreateThread src/nsprpub/pr/src/threads/combined/pruthr.c:1184
    #5 0x7ff879d50aff in PR_CreateThread src/nsprpub/pr/src/threads/combined/pruthr.c:1404
    #6 0x7ff851acd561 in nsThread::Init(class nsTSubstring<char> const &) src/xpcom/threads/nsThread.cpp:604
    #7 0x7ff851adff88 in nsThreadManager::NewNamedThread(class nsTSubstring<char> const &, unsigned int, class nsIThread **) src/xpcom/threads/nsThreadManager.cpp:533
    #8 0x7ff851aecddc in NS_NewNamedThread(class nsTSubstring<char> const &, class nsIThread **, struct already_AddRefed<class nsIRunnable>, unsigned int) src/xpcom/threads/nsThreadUtils.cpp:161
    #9 0x7ff851ae58ff in NS_NewNamedThread src/xpcom/threads/nsThreadUtils.cpp:154
    #10 0x7ff851ae58ff in nsThreadPool::PutEvent(struct already_AddRefed<class nsIRunnable>, unsigned int) src/xpcom/threads/nsThreadPool.cpp:123
    #11 0x7ff851ae87a1 in nsThreadPool::Dispatch(struct already_AddRefed<class nsIRunnable>, unsigned int) src/xpcom/threads/nsThreadPool.cpp:362
    #12 0x7ff851a9790a in mozilla::SharedThreadPool::Dispatch(struct already_AddRefed<class nsIRunnable>, unsigned int) /builds/worker/workspace/obj-build/dist/include/mozilla/SharedThreadPool.h:72
    #13 0x7ff851aabd8b in mozilla::TaskQueue::DispatchLocked(class nsCOMPtr<class nsIRunnable> &, unsigned int, enum mozilla::AbstractThread::DispatchReason) src/xpcom/threads/TaskQueue.cpp:122
    #14 0x7ff851af056b in mozilla::TaskQueue::Dispatch(struct already_AddRefed<class nsIRunnable>, unsigned int) /builds/worker/workspace/obj-build/dist/include/mozilla/TaskQueue.h:73
    #15 0x7ff8531106bb in mozilla::ipc::MessageChannel::PostErrorNotifyTask(void) src/ipc/glue/MessageChannel.cpp:2073
    #16 0x7ff853115c45 in mozilla::ipc::PortLink::OnPortStatusChanged(void) src/ipc/glue/MessageLink.cpp:182
    #17 0x7ff85313b69c in mozilla::ipc::PortLink::PortObserverThunk::OnPortStatusChanged(void) src/ipc/glue/MessageLink.cpp:47
    #18 0x7ff853122a86 in mozilla::ipc::NodeController::PortStatusChanged(class mojo::core::ports::PortRef const &) src/ipc/glue/NodeController.cpp:413
    #19 0x7ff853064ca8 in mojo::core::ports::Node::DestroyAllPortsWithPeer(struct mojo::core::ports::NodeName const &, struct mojo::core::ports::PortName const &) src/ipc/chromium/src/mojo/core/ports/node.cc:1602
    #20 0x7ff85306392f in mojo::core::ports::Node::LostConnectionToNode(struct mojo::core::ports::NodeName const &) src/ipc/chromium/src/mojo/core/ports/node.cc:486
    #21 0x7ff85311e969 in mozilla::ipc::NodeController::DropPeer(struct mojo::core::ports::NodeName) src/ipc/glue/NodeController.cpp:306
    #22 0x7ff853128546 in mozilla::ipc::NodeController::OnChannelError(struct mojo::core::ports::NodeName const &) src/ipc/glue/NodeController.cpp:743
    #23 0x7ff853001b73 in base::MessagePumpForIO::WaitForIOCompletion(unsigned long, class base::MessagePumpForIO::IOHandler *) src/ipc/chromium/src/base/message_pump_win.cc:471
    #24 0x7ff85300155a in base::MessagePumpForIO::WaitForWork src/ipc/chromium/src/base/message_pump_win.cc:453
    #25 0x7ff85300155a in base::MessagePumpForIO::DoRunLoop(void) src/ipc/chromium/src/base/message_pump_win.cc:438
    #26 0x7ff8530023a8 in base::MessagePumpWin::RunWithDispatcher src/ipc/chromium/src/base/message_pump_win.cc:61
    #27 0x7ff8530023a8 in base::MessagePumpWin::Run(class base::MessagePump::Delegate *) src/ipc/chromium/src/base/message_pump_win.h:79
    #28 0x7ff85302df25 in MessageLoop::RunInternal src/ipc/chromium/src/base/message_loop.cc:380
    #29 0x7ff85302df25 in MessageLoop::RunHandler(void) src/ipc/chromium/src/base/message_loop.cc:373
    #30 0x7ff85303cf08 in MessageLoop::Run src/ipc/chromium/src/base/message_loop.cc:356
    #31 0x7ff85303cf08 in base::Thread::ThreadMain(void) src/ipc/chromium/src/base/thread.cc:187
    #32 0x7ff853003aa6 in `anonymous namespace'::ThreadFunc src/ipc/chromium/src/base/platform_thread_win.cc:19
    #33 0x7ff87a139d93 in __asan::AsanThread::ThreadStart(unsigned __int64) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_thread.cpp:277
    #34 0x7ff889d884d3  (C:\Windows\System32\KERNEL32.DLL+0x1800084d3)
    #35 0x7ff87b2e8b2c in mozilla::interceptor::FuncHook<mozilla::interceptor::WindowsDllInterceptor<mozilla::interceptor::VMSharingPolicyShared>,void (*)(int, void *, void *)>::operator() src/toolkit/xre/dllservices/mozglue/nsWindowsDllInterceptor.h:150
    #36 0x7ff87b2e8b2c in patched_BaseThreadInitThunk src/toolkit/xre/dllservices/mozglue/WindowsDllBlocklist.cpp:572
    #37 0x7ff88bb31790  (C:\Windows\SYSTEM32\ntdll.dll+0x180051790)

Thread T1 created by T0 here:
    #0 0x7ff87a13af32 in __asan_wrap_CreateThread /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_win.cpp:146
    #1 0x7ff853003a3c in PlatformThread::Create(unsigned __int64, class PlatformThread::Delegate *, void **) src/ipc/chromium/src/base/platform_thread_win.cc:57
    #2 0x7ff85303c4dc in base::Thread::StartWithOptions(struct base::Thread::Options const &) src/ipc/chromium/src/base/thread.cc:93
    #3 0x7ff85303daed in ChildProcess::ChildProcess(class ChildThread *) src/ipc/chromium/src/chrome/common/child_process.cc:20
    #4 0x7ff85312baf5 in mozilla::ipc::ProcessChild::ProcessChild(unsigned long) src/ipc/glue/ProcessChild.cpp:33
    #5 0x7ff858d3e0b4 in mozilla::RDDProcessImpl::RDDProcessImpl(unsigned long) src/dom/media/ipc/RDDProcessImpl.cpp:25
    #6 0x7ff85fb29251 in mozilla::MakeUnique /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:609
    #7 0x7ff85fb29251 in XRE_InitChildProcess(int, char **const, struct XREChildData const *) src/toolkit/xre/nsEmbedFunctions.cpp:673
    #8 0x7ff68cd32578 in content_process_main src/browser/app/../../ipc/contentproc/plugin-container.cpp:58
    #9 0x7ff68cd32578 in NS_internal_main(int, char **, char **) src/browser/app/nsBrowserApp.cpp:338
    #10 0x7ff68cd317bf in wmain src/toolkit/xre/nsWindowsWMain.cpp:167
    #11 0x7ff68ce2c427 in invoke_main d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:90
    #12 0x7ff68ce2c427 in __scrt_common_main_seh d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #13 0x7ff889d884d3  (C:\Windows\System32\KERNEL32.DLL+0x1800084d3)
    #14 0x7ff88bb31790  (C:\Windows\SYSTEM32\ntdll.dll+0x180051790)

Comment 1

[David Parks [:handyman]]

No idea why this is happening but I can add a little more detail:

From the stacks in comment 0, this starts with a process' connection to the RDD process being lost. RDD is connected to content and main and we don't know which was lost. The lost peer causes RDD to spawn a new thread (from a thread pool, but here that pool creates a new thread) to handle the error -- in MessageChannel::PostErrorNotifyTask. None of this seems to be related to the actual crash though.

The new thread has runs our hooked BaseThreadInitThunk, which checks the blocklist before proceeding. This is only interesting because it suggests that the new thread's stack is operational.

From there, it runs the win32 BaseThreadInitThunk, which fails in ASAN at a spot that suggests that some part of the stack is not valid memory. CheckShadowForThreadStackAndTLS calls PoisonShadow. Since we have a WRITE failure at address 0x0238a184d000, we can assume that the FastPoisonShadow part (that actually does the poisoning) is responsible. The failure address is 16-byte aligned, so, while we don't know that it is the start of the block it is clearing, we know that could be the start of the stack. Additionally, it is 4K aligned, so it's a page boundary... but maybe not an entire VirtualAlloc region (which I think are 64K aligned)? If so, that would suggest a VirtualAlloc region that was later split by permissions.

ASAN gets the stack like this. I have some doubts about how they are interpreting the MEMORY_BASIC_INFORMATION structure (like not accounting for mbi itself) but its docs are highly ambiguous and you can find many theories about how to find the current stack space. And if this only failing with RDD then there may be more to it.

There is a better API for Windows 8+, discussed here but, as the MSDN page suggests, it also has some issues.