Closed
Bug 1769798
Opened 2 years ago
Closed 2 years ago
ASan allocator does not return NULL on Windows
Categories
(Core :: Sanitizers, defect)
Tracking
()
RESOLVED
FIXED
105 Branch
| Tracking | Status | |
|---|---|---|
| firefox105 | --- | fixed |
People
(Reporter: tsmith, Assigned: truber)
References
Details
(Keywords: testcase, Whiteboard: [fuzzblocker])
Attachments
(2 files)
Instead of returning NULL when a fallible allocation does not succeed, an error/crash is encountered ==3236==ERROR: AddressSanitizer failed to allocate 0x7710003000 (511369555968) bytes of LargeMmapAllocator (error code: 1455).
Truber pointed out that error code: 1455 might not be handled properly.
ERROR_COMMITMENT_LIMIT
1455 (0x5AF)
The paging file is too small for this operation to complete.
Decoder mentioned there may be other scenarios where this is happening.
We currently set ASAN_OPTIONS=allocator_may_return_null=1 in the build and it is also set in the automation environment.
==3236==ERROR: AddressSanitizer failed to allocate 0x7710003000 (511369555968) bytes of LargeMmapAllocator (error code: 1455)
==3236==Dumping process modules:
0x121429f90000-0x12142a114000 C:\Windows\System32\gdi32full.dll
0x7ff7da3e0000-0x7ff7da55c000 C:\Users\task_165278910375246\builds\asan\firefox.exe
0x7ffe2daf0000-0x7ffe44fe6000 C:\Users\task_165278910375246\builds\asan\xul.dll
0x7ffe574a0000-0x7ffe57a15000 C:\Users\task_165278910375246\builds\asan\mozavcodec.dll
0x7ffe57a20000-0x7ffe581a4000 C:\Users\task_165278910375246\builds\asan\nss3.dll
0x7ffe581b0000-0x7ffe58c07000 C:\Users\task_165278910375246\builds\asan\clang_rt.asan_dynamic-x86_64.dll
0x7ffe5daa0000-0x7ffe5db3e000 C:\Users\task_165278910375246\builds\asan\mozavutil.dll
0x7ffe5e720000-0x7ffe5e736000 C:\Windows\system32\napinsp.dll
0x7ffe5e740000-0x7ffe5e74e000 C:\Windows\System32\winrnr.dll
0x7ffe603f0000-0x7ffe603fa000 C:\Windows\SYSTEM32\VERSION.dll
0x7ffe61990000-0x7ffe61b2e000 C:\Users\task_165278910375246\builds\asan\freebl3.dll
0x7ffe61b30000-0x7ffe61bf9000 C:\Users\task_165278910375246\builds\asan\softokn3.dll
0x7ffe61c00000-0x7ffe61e69000 C:\Windows\system32\dwrite.dll
0x7ffe61e70000-0x7ffe61efd000 C:\Windows\SYSTEM32\MSVCP140.dll
0x7ffe61fa0000-0x7ffe621bf000 C:\Users\task_165278910375246\builds\asan\mozglue.dll
0x7ffe644e0000-0x7ffe644eb000 C:\Windows\SYSTEM32\AVRT.dll
0x7ffe64bc0000-0x7ffe64bd9000 C:\Users\task_165278910375246\builds\asan\lgpllibs.dll
0x7ffe65200000-0x7ffe65209000 C:\Windows\SYSTEM32\WSOCK32.dll
0x7ffe65210000-0x7ffe6521c000 C:\Windows\SYSTEM32\VCRUNTIME140_1.dll
0x7ffe657a0000-0x7ffe657bb000 C:\Windows\SYSTEM32\VCRUNTIME140.dll
0x7ffe65d40000-0x7ffe65d58000 C:\Windows\system32\NLAapi.dll
0x7ffe66280000-0x7ffe66405000 C:\Windows\SYSTEM32\PROPSYS.dll
0x7ffe67a00000-0x7ffe67a2b000 C:\Windows\SYSTEM32\WINMMBASE.dll
0x7ffe67b40000-0x7ffe67b63000 C:\Windows\SYSTEM32\WINMM.dll
0x7ffe683d0000-0x7ffe683f8000 C:\Windows\SYSTEM32\DEVOBJ.dll
0x7ffe68440000-0x7ffe68472000 C:\Windows\SYSTEM32\ntmarta.dll
0x7ffe68d60000-0x7ffe68d98000 C:\Windows\SYSTEM32\IPHLPAPI.DLL
0x7ffe68da0000-0x7ffe68e42000 C:\Windows\SYSTEM32\DNSAPI.dll
0x7ffe69030000-0x7ffe6908d000 C:\Windows\System32\mswsock.dll
0x7ffe69690000-0x7ffe6969b000 C:\Windows\SYSTEM32\cryptbase.dll
0x7ffe696b0000-0x7ffe696db000 C:\Windows\SYSTEM32\bcrypt.dll
0x7ffe697d0000-0x7ffe697e0000 C:\Windows\System32\MSASN1.dll
0x7ffe697e0000-0x7ffe697ef000 C:\Windows\System32\kernel.appcore.dll
0x7ffe697f0000-0x7ffe6988c000 C:\Windows\System32\msvcp_win.dll
0x7ffe69890000-0x7ffe69a83000 C:\Windows\System32\CRYPT32.dll
0x7ffe69a90000-0x7ffe69afc000 C:\Windows\System32\bcryptPrimitives.dll
0x7ffe69bb0000-0x7ffe69c58000 C:\Windows\System32\shcore.dll
0x7ffe6a340000-0x7ffe6a434000 C:\Windows\System32\ucrtbase.dll
0x7ffe6a5d0000-0x7ffe6a625000 C:\Windows\System32\WINTRUST.dll
0x7ffe6a630000-0x7ffe6a84e000 C:\Windows\System32\KERNELBASE.dll
0x7ffe6a850000-0x7ffe6a892000 C:\Windows\System32\cfgmgr32.dll
0x7ffe6a8a0000-0x7ffe6a8be000 C:\Windows\System32\win32u.dll
0x7ffe6a990000-0x7ffe6aa3d000 C:\Windows\System32\KERNEL32.DLL
0x7ffe6aa40000-0x7ffe6aba6000 C:\Windows\System32\user32.dll
0x7ffe6abb0000-0x7ffe6abe4000 C:\Windows\System32\GDI32.dll
0x7ffe6abf0000-0x7ffe6abf8000 C:\Windows\System32\NSI.dll
0x7ffe6ac00000-0x7ffe6ac6a000 C:\Windows\System32\WS2_32.dll
0x7ffe6ace0000-0x7ffe6ad3b000 C:\Windows\System32\sechost.dll
0x7ffe6ad40000-0x7ffe6ae00000 C:\Windows\System32\OLEAUT32.dll
0x7ffe6ae00000-0x7ffe6ae9e000 C:\Windows\System32\msvcrt.dll
0x7ffe6aed0000-0x7ffe6b2f9000 C:\Windows\System32\SETUPAPI.dll
0x7ffe6b600000-0x7ffe6b6a6000 C:\Windows\System32\advapi32.dll
0x7ffe6b6b0000-0x7ffe6b976000 C:\Windows\System32\combase.dll
0x7ffe6cfc0000-0x7ffe6d0dd000 C:\Windows\System32\RPCRT4.dll
0x7ffe6d0e0000-0x7ffe6d219000 C:\Windows\System32\ole32.dll
0x7ffe6d320000-0x7ffe6d4ef000 C:\Windows\SYSTEM32\ntdll.dll
AddressSanitizer: CHECK failed: sanitizer_common.cpp:53 "((0 && "unable to mmap")) != (0)" (0x0, 0x0) (tid=8136)
#0 0x7ffe581f6fe7 in __asan::CheckUnwind /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_rtl.cpp:67
#1 0x7ffe581c5635 in __sanitizer::CheckFailed(char const *, int, char const *, unsigned __int64, unsigned __int64) /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:86
#2 0x7ffe581b4bae in __sanitizer::ReportMmapFailureAndDie(unsigned __int64, char const *, char const *, unsigned int, bool) /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common.cpp:53
#3 0x7ffe581c24ab in __sanitizer::ReturnNullptrOnOOMOrDie /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_win.cpp:162
#4 0x7ffe581c24ab in __sanitizer::MmapOrDieOnFatalError(unsigned __int64, char const *) /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_win.cpp:170
#5 0x7ffe581db039 in __sanitizer::LargeMmapAllocator<struct __asan::AsanMapUnmapCallback, class __sanitizer::LargeMmapAllocatorPtrArrayDynamic, struct __sanitizer::LocalAddressSpaceView>::Allocate(class __sanitizer::AllocatorStats *, unsigned __int64, unsigned __int64) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_allocator_secondary.h:97
#6 0x7ffe581dae33 in __sanitizer::CombinedAllocator<class __sanitizer::SizeClassAllocator64<struct __asan::AP64<struct __sanitizer::LocalAddressSpaceView>>, class __sanitizer::LargeMmapAllocatorPtrArrayDynamic>::Allocate(struct __sanitizer::SizeClassAllocator64LocalCache<class __sanitizer::SizeClassAllocator64<struct __asan::AP64<struct __sanitizer::LocalAddressSpaceView>>> *, unsigned __int64, unsigned __int64) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_allocator_combined.h:71
#7 0x7ffe581d6645 in __asan::Allocator::Allocate(unsigned __int64, unsigned __int64, struct __sanitizer::BufferedStackTrace *, enum __asan::AllocType, bool) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_allocator.cpp:526
#8 0x7ffe581d6409 in __asan::asan_malloc(unsigned __int64, struct __sanitizer::BufferedStackTrace *) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_allocator.cpp:953
#9 0x7ffe581ee040 in malloc /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_malloc_win.cpp:99
#10 0x7ffe3db46191 in alloc::alloc::alloc /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c\library\alloc\src\alloc.rs:87
#11 0x7ffe3db46191 in fallible_collections::vec::vec_try_extend /builds/worker/checkouts/gecko/third_party/rust/fallible_collections/src/vec.rs:488
#12 0x7ffe3db46191 in fallible_collections::vec::vec_try_reserve /builds/worker/checkouts/gecko/third_party/rust/fallible_collections/src/vec.rs:445
#13 0x7ffe3db46191 in fallible_collections::vec::impl$19::try_reserve /builds/worker/checkouts/gecko/third_party/rust/fallible_collections/src/vec.rs:509
#14 0x7ffe3db46191 in fallible_collections::vec::impl$19::try_with_capacity /builds/worker/checkouts/gecko/third_party/rust/fallible_collections/src/vec.rs:533
#15 0x7ffe3db46191 in fallible_collections::vec::TryVec<mp4parse::unstable::Indice>::with_capacity /builds/worker/checkouts/gecko/third_party/rust/fallible_collections/src/vec.rs:97
#16 0x7ffe3db46191 in mp4parse::unstable::create_sample_table /builds/worker/checkouts/gecko/third_party/rust/mp4parse/src/unstable.rs:166
#17 0x7ffe3db46191 in mp4parse_capi::get_indice_table /builds/worker/checkouts/gecko/third_party/rust/mp4parse_capi/src/lib.rs:1150
#18 0x7ffe3db46191 in mp4parse_get_indice_table /builds/worker/checkouts/gecko/third_party/rust/mp4parse_capi/src/lib.rs:1117
#19 0x7ffe364c8a94 in mozilla::MP4Metadata::GetTrackIndice(unsigned int) /builds/worker/checkouts/gecko/dom/media/mp4/MP4Metadata.cpp:448
#20 0x7ffe364c0ca9 in mozilla::MP4Demuxer::Init(void) /builds/worker/checkouts/gecko/dom/media/mp4/MP4Demuxer.cpp:216
#21 0x7ffe355c169b in mozilla::MediaFormatReader::DemuxerProxy::Init::<lambda_86>::operator() /builds/worker/checkouts/gecko/dom/media/MediaFormatReader.cpp:777
#22 0x7ffe355c169b in mozilla::detail::ProxyFunctionRunnable<`lambda at /builds/worker/checkouts/gecko/dom/media/MediaFormatReader.cpp:775:22',mozilla::MozPromise<mozilla::MediaResult,mozilla::MediaResult,1> >::Run /builds/worker/workspace/obj-build/dist/include/mozilla/MozPromise.h:1645
#23 0x7ffe2e71ac9e in mozilla::TaskQueue::Runner::Run(void) /builds/worker/checkouts/gecko/xpcom/threads/TaskQueue.cpp:259
#24 0x7ffe2e75344c in nsThreadPool::Run(void) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadPool.cpp:310
#25 0x7ffe2e740805 in nsThread::ProcessNextEvent(bool, bool *) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1174
#26 0x7ffe2e74ee1c in NS_ProcessNextEvent(class nsIThread *, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:465
#27 0x7ffe2fd7354e in mozilla::ipc::MessagePumpForNonMainThreads::Run(class base::MessagePump::Delegate *) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:300
#28 0x7ffe2fc8b145 in MessageLoop::RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:380
#29 0x7ffe2fc8b145 in MessageLoop::RunHandler(void) /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:373
#30 0x7ffe2fc8af15 in MessageLoop::Run(void) /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:355
#31 0x7ffe2e73663f in nsThread::ThreadFunc(void *) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:378
#32 0x7ffe57e1946d in _PR_NativeRunThread /builds/worker/checkouts/gecko/nsprpub/pr/src/threads/combined/pruthr.c:399
#33 0x7ffe57df1a3b in pr_root /builds/worker/checkouts/gecko/nsprpub/pr/src/md/windows/w95thred.c:139
#34 0x7ffe6a35fb7f (C:\Windows\System32\ucrtbase.dll+0x18001fb7f)
#35 0x7ffe581f9d93 in __asan::AsanThread::ThreadStart(unsigned __int64) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_thread.cpp:277
#36 0x7ffe6a9984d3 (C:\Windows\System32\KERNEL32.DLL+0x1800084d3)
#37 0x7ffe620b8b2c in mozilla::interceptor::FuncHook<mozilla::interceptor::WindowsDllInterceptor<mozilla::interceptor::VMSharingPolicyShared>,void (*)(int, void *, void *)>::operator() /builds/worker/checkouts/gecko/toolkit/xre/dllservices/mozglue/nsWindowsDllInterceptor.h:150
#38 0x7ffe620b8b2c in patched_BaseThreadInitThunk /builds/worker/checkouts/gecko/toolkit/xre/dllservices/mozglue/WindowsDllBlocklist.cpp:572
#39 0x7ffe6d371790 (C:\Windows\SYSTEM32\ntdll.dll+0x180051790)
|
Assignee | |
Comment 1•2 years ago
|
||
This error code should probably be included here: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.3/compiler-rt/lib/sanitizer_common/sanitizer_win.cpp#L160
|
|
Assignee | |
Comment 2•2 years ago
|
||
There's an interesting comment here too: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.3/compiler-rt/lib/sanitizer_common/sanitizer_win.cpp#L232-L234
Maybe MEM_COMMIT should be excluded in more cases for win64?
|
||
Comment 3•2 years ago
|
||
The severity field is not set for this bug.
:decoder, could you have a look please?
For more information, please visit auto_nag documentation.
Flags: needinfo?(choller)
|
|
Assignee | |
Comment 4•2 years ago
|
||
|
||
Updated•2 years ago
|
Assignee: nobody → jschwartzentruber
Status: NEW → ASSIGNED
Pushed by jdschwa@gmail.com: https://hg.mozilla.org/integration/autoland/rev/7a6fa71eb328 Return nullptr on ASan allocator commitment error. r=firefox-build-system-reviewers,ahochheiden
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox105:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 105 Branch
|
||
Comment 7•2 years ago
|
||
Jesse, can you check https://phabricator.services.mozilla.com/D152934#5016781 ?
Flags: needinfo?(jschwartzentruber)
|
|
Assignee | |
Comment 8•2 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #7)
Jesse, can you check https://phabricator.services.mozilla.com/D152934#5016781 ?
I opened https://reviews.llvm.org/D130781 for this. I'll make a bug for applying to clang-trunk as well.
Flags: needinfo?(jschwartzentruber)
|
||
Updated•2 months ago
|
Severity: -- → S3
Flags: needinfo?(choller)
You need to log in
before you can comment on or make changes to this bug.
Description
•