Closed Bug 1318262 Opened 9 years ago Closed 9 years ago

Memory leak and OOM crash in ASAN Nightly while idling on drudgereport.com

Categories

(Core :: Memory Allocator, defect)

Product:
Core
Component:
Memory Allocator
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED INCOMPLETE
Tracking Flags:
Tracking Status
firefox53 --- affected

People

(Reporter: geeknik, Unassigned)

Details

(Keywords: crash, Whiteboard: [MemShrink])

Attachments

(1 file, 2 obsolete files)

about:memory measure&save
61.62 KB, application/x-gzip
Details
ASAN Nightly Build ID 20161117010803 STR: 1) Launch ASAN Nightly with a clean profile 2) Visit http://drudgereport.com 3) Walk away for approx 90 minutes and come back to a crash Drudge Report refreshes the page every X seconds and over time Nightly continues using more and more memory with each successive page refresh until... ==49744==ERROR: AddressSanitizer failed to allocate 0x10000 (65536) bytes of memory at address 0x6250018e0 (error code: 12) ERROR: Failed to mmap [Child 49783] ###!!! ABORT: Aborting on channel error.: file /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp, line 2148 [Child 49783] ###!!! ABORT: Aborting on channel error.: file /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp, line 2148 ASAN:DEADLYSIGNAL ================================================================= ==49783==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0000004e114b bp 0x7f1cda979090 sp 0x7f1cda979080 T2) ###!!! [Child][MessageChannel] Error: (msgtype=0x2,name=PAPZCTreeManager::Msg_ContentReceivedInputBlock) Channel error: cannot send/recv #0 0x4e114a in mozalloc_abort(char const*) /home/worker/workspace/build/src/memory/mozalloc/mozalloc_abort.cpp:33:5 #1 0x7f1cdd575df5 in Abort(char const*) /home/worker/workspace/build/src/xpcom/base/nsDebugImpl.cpp:449:3 #2 0x7f1cdd575b9c in NS_DebugBreak /home/worker/workspace/build/src/xpcom/base/nsDebugImpl.cpp:405:7 #3 0x7f1cde4da6ef in mozilla::ipc::MessageChannel::OnChannelErrorFromLink() /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:2148:13 #4 0x7f1cde4df993 in OnChannelError /home/worker/workspace/build/src/ipc/glue/MessageLink.cpp:367:5 #5 0x7f1cde4df993 in non-virtual thunk to mozilla::ipc::ProcessLink::OnChannelError() /home/worker/workspace/build/src/ipc/glue/MessageLink.cpp:359 #6 0x7f1cde495c3b in event_process_active_single_queue /home/worker/workspace/build/src/ipc/chromium/src/third_party/libevent/event.c:1350:4 #7 0x7f1cde495c3b in event_process_active /home/worker/workspace/build/src/ipc/chromium/src/third_party/libevent/event.c:1420 #8 0x7f1cde495c3b in event_base_loop /home/worker/workspace/build/src/ipc/chromium/src/third_party/libevent/event.c:1621 #9 0x7f1cde4550d1 in base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) /home/worker/workspace/build/src/ipc/chromium/src/base/message_pump_libevent.cc:364:7 #10 0x7f1cde44f538 in RunInternal /home/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:232:3 #11 0x7f1cde44f538 in RunHandler /home/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:225 #12 0x7f1cde44f538 in MessageLoop::Run() /home/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:205 #13 0x7f1cde46f6e1 in base::Thread::ThreadMain() /home/worker/workspace/build/src/ipc/chromium/src/base/thread.cc:180:3 #14 0x7f1cde47023c in ThreadFunc(void*) /home/worker/workspace/build/src/ipc/chromium/src/base/platform_thread_posix.cc:38:3 #15 0x7f1cf814b0a3 in start_thread /build/glibc-daoqzt/glibc-2.19/nptl/pthread_create.c:309 #16 0x7f1cf725262c in clone /build/glibc-daoqzt/glibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111 AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV /home/worker/workspace/build/src/memory/mozalloc/mozalloc_abort.cpp:33:5 in mozalloc_abort(char const*) Thread T2 (Chrome_ChildThr) created by T0 (Web Content) here: #0 0x49a869 in __interceptor_pthread_create /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:238:3 #1 0x7f1cde46f2fb in CreateThread /home/worker/workspace/build/src/ipc/chromium/src/base/platform_thread_posix.cc:137:14 #2 0x7f1cde46f2fb in Create /home/worker/workspace/build/src/ipc/chromium/src/base/platform_thread_posix.cc:148 #3 0x7f1cde46f2fb in base::Thread::StartWithOptions(base::Thread::Options const&) /home/worker/workspace/build/src/ipc/chromium/src/base/thread.cc:98 #4 0x7f1cde4e1b77 in mozilla::ipc::ProcessChild::ProcessChild(int) /home/worker/workspace/build/src/ipc/glue/ProcessChild.cpp:24:5 #5 0x7f1ce5c81d39 in ContentProcess /home/worker/workspace/build/src/obj-firefox/dist/include/mozilla/dom/ContentProcess.h:31:7 #6 0x7f1ce5c81d39 in XRE_InitChildProcess /home/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:619 #7 0x4dfb5b in content_process_main /home/worker/workspace/build/src/browser/app/../../ipc/contentproc/plugin-container.cpp:197:19 #8 0x4dfb5b in main /home/worker/workspace/build/src/browser/app/nsBrowserApp.cpp:438 #9 0x7f1cf718bb44 in __libc_start_main /build/glibc-daoqzt/glibc-2.19/csu/libc-start.c:287 ==49783==ABORTING real 79m22.088s user 2m29.460s sys 5m0.448s Happens pretty consistently.
Component: Untriaged → Memory Allocator
Product: Firefox → Core
I'll see if I can repro. I'd guess we're going to see a ton of orphaned nodes. cc'ing mcrr8 in case CC comes into play.
Whiteboard: [MemShrink]
Can you reproduce this outside of ASan? What does the memory report saved from about:memory look like?
Attached file cc log (obsolete) —
Attached file gc log (obsolete) —
I hope these are what you were looking for.
Disregard these logs, I attached the wrong logs to the wrong bug. ): I'll get back to you on your request.
On a Windows 8.1 machine, I left the 16 November 2016 build of Nightly x64 running overnight and came back to it today and it still has Drudge Report loaded and the browser was a little sluggish, but responsive and definitely did not crash.
Attachment #8811922 - Attachment is obsolete: true
Attachment #8811923 - Attachment is obsolete: true
I did a 20 minute run and didn't see anything of notice in the memory reports, I did run into a crash about 10 minutes later that has since been fixed (bug 1317415). I'll do another test run once I get a fresh build up.
In this case, I haven't seen the OOM crash happen quicker than 60 or 70 minutes. I haven't run into that other crash on this website or any others (just while fuzzing Nightly).
Okay I ran overnight, but couldn't repro an oom (or high memory). I'm on Linux64 debug, I suppose it's possible there's some poor interaction with ASAN.
Interesting enough, while going through 10000 testcases generated with BF3 overnight, loading 1 per second, the browser OOM'd and crashed after 79m 23.406s (compared to 79m 22.088s with just leaving DrudgeReport open in the browser). If it matters, I'm running these tests in a Debian 8 x64 VMware instance with 4GB RAM, 3vCPU, 20GB disk allocated. I kind of want to call this the 79 minute bug now. (; ==12536==ERROR: AddressSanitizer failed to allocate 0x8000 (32768) bytes of SetAlternateSignalStack (error code: 12) ERROR: Failed to mmap [Child 12620] ###!!! ABORT: Aborting on channel error.: file /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp, line 2148 [Child 12620] ###!!! ABORT: Aborting on channel error.: file /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp, line 2148 ASAN:DEADLYSIGNAL ================================================================= ==12620==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0000004e114b bp 0x7f89eb398090 sp 0x7f89eb398080 T2) #0 0x4e114a in mozalloc_abort(char const*) /home/worker/workspace/build/src/memory/mozalloc/mozalloc_abort.cpp:33:5 #1 0x7f89edf58455 in Abort(char const*) /home/worker/workspace/build/src/xpcom/base/nsDebugImpl.cpp:449:3 #2 0x7f89edf581fc in NS_DebugBreak /home/worker/workspace/build/src/xpcom/base/nsDebugImpl.cpp:405:7 #3 0x7f89eeebce2f in mozilla::ipc::MessageChannel::OnChannelErrorFromLink() /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:2148:13 #4 0x7f89eeec20d3 in OnChannelError /home/worker/workspace/build/src/ipc/glue/MessageLink.cpp:367:5 #5 0x7f89eeec20d3 in non-virtual thunk to mozilla::ipc::ProcessLink::OnChannelError() /home/worker/workspace/build/src/ipc/glue/MessageLink.cpp:359 #6 0x7f89eee7837b in event_process_active_single_queue /home/worker/workspace/build/src/ipc/chromium/src/third_party/libevent/event.c:1350:4 #7 0x7f89eee7837b in event_process_active /home/worker/workspace/build/src/ipc/chromium/src/third_party/libevent/event.c:1420 #8 0x7f89eee7837b in event_base_loop /home/worker/workspace/build/src/ipc/chromium/src/third_party/libevent/event.c:1621 #9 0x7f89eee37811 in base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) /home/worker/workspace/build/src/ipc/chromium/src/base/message_pump_libevent.cc:364:7 #10 0x7f89eee31c78 in RunInternal /home/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:232:3 #11 0x7f89eee31c78 in RunHandler /home/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:225 #12 0x7f89eee31c78 in MessageLoop::Run() /home/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:205 #13 0x7f89eee51e21 in base::Thread::ThreadMain() /home/worker/workspace/build/src/ipc/chromium/src/base/thread.cc:180:3 #14 0x7f89eee5297c in ThreadFunc(void*) /home/worker/workspace/build/src/ipc/chromium/src/base/platform_thread_posix.cc:38:3 #15 0x7f8a088c30a3 in start_thread /build/glibc-daoqzt/glibc-2.19/nptl/pthread_create.c:309 #16 0x7f8a079ca62c in clone /build/glibc-daoqzt/glibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111 AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV /home/worker/workspace/build/src/memory/mozalloc/mozalloc_abort.cpp:33:5 in mozalloc_abort(char const*) Thread T2 (Chrome_ChildThr) created by T0 (Web Content) here: #0 0x49a869 in __interceptor_pthread_create /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:238:3 #1 0x7f89eee51a3b in CreateThread /home/worker/workspace/build/src/ipc/chromium/src/base/platform_thread_posix.cc:137:14 #2 0x7f89eee51a3b in Create /home/worker/workspace/build/src/ipc/chromium/src/base/platform_thread_posix.cc:148 #3 0x7f89eee51a3b in base::Thread::StartWithOptions(base::Thread::Options const&) /home/worker/workspace/build/src/ipc/chromium/src/base/thread.cc:98 #4 0x7f89eeec42b7 in mozilla::ipc::ProcessChild::ProcessChild(int) /home/worker/workspace/build/src/ipc/glue/ProcessChild.cpp:24:5 #5 0x7f89f6624b69 in ContentProcess /home/worker/workspace/build/src/obj-firefox/dist/include/mozilla/dom/ContentProcess.h:31:7 #6 0x7f89f6624b69 in XRE_InitChildProcess /home/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:619 #7 0x4dfb5b in content_process_main /home/worker/workspace/build/src/browser/app/../../ipc/contentproc/plugin-container.cpp:197:19 #8 0x4dfb5b in main /home/worker/workspace/build/src/browser/app/nsBrowserApp.cpp:438 #9 0x7f8a07903b44 in __libc_start_main /build/glibc-daoqzt/glibc-2.19/csu/libc-start.c:287 ==12620==ABORTING real 79m23.406s user 10m44.976s sys 5m27.472s
Why do you think this is an OOM? This looks like an IPC issue, maybe we're leaking shared memory?
Flags: needinfo?(geeknik)
Well, when I'm awake and running the tests, I'm steadily watching free memory on the VM drop from 3GB free to 0 while swap increases from 0 to 1GB and that's about when the crash normally happens. Here is an about:memory measurement as we creep closer to the 79 minute mark.
Flags: needinfo?(geeknik)
We think this is isolated to interaction with an ASAN build, I'm not sure there's anything we can do.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → INCOMPLETE
I don't think this is limited to just ASAN builds. While running 32-bit Windows Firefox 50.1.0 under WinDbg with ApplicationVerifier, I've managed to generate `A heap allocation was leaked` messages after 60-90 minutes of having one tab open to drudgereport.com. Working on generating more useful information.
Here's what I've got so far, looks like it might be related to D3D9, DXVA and/or how we interface with igdumd32.dll: ======================================= VERIFIER STOP 00000900: pid 0xE44: A heap allocation was leaked. D1E4EF18 : Address of the leaked allocation. Run !heap -p -a <address> to get additional information about the allocation. FE367F84 : Address to the allocation stack trace. Run dps <address> to view the allocation stack. E7240FE0 : Address of the owner dll name. Run du <address> to read the dll name. 6F770000 : Base of the owner dll. Run .reload <dll_name> = <address> to reload the owner dll. Use 'lm' to get more information about the loaded and unloaded modules. ======================================= This verifier stop is continuable. After debugging it use `go' to continue. ======================================= (e44.1f84): Break instruction exception - code 80000003 (first chance) eax=ebb03fc0 ebx=00000000 ecx=000001a1 edx=000000e6 esi=66a52e70 edi=d1e4ef18 eip=66af37f9 esp=0b16f160 ebp=0b16f374 iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202 vrfcore!VerifierStopMessageEx+0x599: 66af37f9 cc int 3 1:072> !heap -p -a D1E4EF18 address d1e4ef18 found in _DPH_HEAP_ROOT @ fe111000 in busy allocation ( DPH_HEAP_BLOCK: UserAddr UserSize - VirtAddr VirtSize) d1d008bc: d1e4ef18 e4 - d1e4e000 2000 61979c2c verifier!AVrfDebugPageHeapAllocate+0x0000023c 7727f093 ntdll!RtlDebugAllocateHeap+0x0000003c 771e81c6 ntdll!RtlpAllocateHeap+0x00001676 771e5c95 ntdll!RtlpAllocateHeapInternal+0x00000155 771e5b32 ntdll!RtlAllocateHeap+0x00000032 66afaeaa vrfcore!VfCoreRtlAllocateHeap+0x0000002a 66a4572c vfbasics!AVrfpRtlAllocateHeap+0x000000dc 7515d122 KERNELBASE!LocalAlloc+0x00000082 66a46316 vfbasics!AVrfpLocalAlloc+0x000000a6 *** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\SysWOW64\igdumd32.dll - 6f771201 igdumd32!OpenAdapter+0x00000051 6b6b822a d3d9!CreateDeviceLHDDI+0x00000459 6b6b8ee4 d3d9!D3D9CreateDirectDrawObject+0x00000182 6b6b6480 d3d9!FetchDirectDrawData+0x00000103 6b6b91b9 d3d9!InternalDirectDrawCreate+0x00000180 6b6a7274 d3d9!CEnum::CEnum+0x00000245 6b6a1c6a d3d9!Direct3DCreate9Impl+0x00000090 6b6a1bd0 d3d9!Direct3DCreate9Ex+0x00000020 5d1b6b78 xul!mozilla::D3D9DXVA2Manager::Init+0x000000aa [c:\builds\moz2_slave\m-rel-w32-00000000000000000000\build\src\dom\media\platforms\wmf\dxva2manager.cpp @ 285] 5d1b5e4d xul!mozilla::DXVA2Manager::CreateD3D9DXVA+0x0000004b [c:\builds\moz2_slave\m-rel-w32-00000000000000000000\build\src\dom\media\platforms\wmf\dxva2manager.cpp @ 500] 5c0cd737 xul!mozilla::ipc::MessagePump::Run+0x00000072 [c:\builds\moz2_slave\m-rel-w32-00000000000000000000\build\src\ipc\glue\messagepump.cpp @ 100] 1:072> dps FE367F84 fe367f84 fe27b80c fe367f88 0000a004 fe367f8c 00140000 fe367f90 61979c2c verifier!AVrfDebugPageHeapAllocate+0x23c fe367f94 7727f093 ntdll!RtlDebugAllocateHeap+0x3c fe367f98 771e81c6 ntdll!RtlpAllocateHeap+0x1676 fe367f9c 771e5c95 ntdll!RtlpAllocateHeapInternal+0x155 fe367fa0 771e5b32 ntdll!RtlAllocateHeap+0x32 fe367fa4 66afaeaa vrfcore!VfCoreRtlAllocateHeap+0x2a fe367fa8 66a4572c vfbasics!AVrfpRtlAllocateHeap+0xdc fe367fac 7515d122 KERNELBASE!LocalAlloc+0x82 fe367fb0 66a46316 vfbasics!AVrfpLocalAlloc+0xa6 fe367fb4 6f771201 igdumd32!OpenAdapter+0x51 fe367fb8 6b6b822a d3d9!CreateDeviceLHDDI+0x459 fe367fbc 6b6b8ee4 d3d9!D3D9CreateDirectDrawObject+0x182 fe367fc0 6b6b6480 d3d9!FetchDirectDrawData+0x103 fe367fc4 6b6b91b9 d3d9!InternalDirectDrawCreate+0x180 fe367fc8 6b6a7274 d3d9!CEnum::CEnum+0x245 fe367fcc 6b6a1c6a d3d9!Direct3DCreate9Impl+0x90 fe367fd0 6b6a1bd0 d3d9!Direct3DCreate9Ex+0x20 fe367fd4 5d1b6b78 xul!mozilla::D3D9DXVA2Manager::Init+0xaa [c:\builds\moz2_slave\m-rel-w32-00000000000000000000\build\src\dom\media\platforms\wmf\dxva2manager.cpp @ 285] fe367fd8 5d1b5e4d xul!mozilla::DXVA2Manager::CreateD3D9DXVA+0x4b [c:\builds\moz2_slave\m-rel-w32-00000000000000000000\build\src\dom\media\platforms\wmf\dxva2manager.cpp @ 500] fe367fdc 5c0cd737 xul!mozilla::ipc::MessagePump::Run+0x72 [c:\builds\moz2_slave\m-rel-w32-00000000000000000000\build\src\ipc\glue\messagepump.cpp @ 100] fe367fe0 00000000 fe367fe4 00000000 fe367fe8 0000a001 fe367fec 00150000 fe367ff0 61979e42 verifier!AVrfDebugPageHeapFree+0xc2 fe367ff4 7727f89e ntdll!RtlDebugFreeHeap+0x3c fe367ff8 771e995d ntdll!RtlpFreeHeap+0xd3d fe367ffc 771e8bd8 ntdll!RtlFreeHeap+0x758 fe368000 66afaf7b vrfcore!VfCoreRtlFreeHeap+0x2b 1:072> du E7240FE0 e7240fe0 "igdumd32.dll"
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: