1537692 - Intermittent SUMMARY: AddressSanitizer: heap-use-after-free z:\build\build\src\gfx\vr\gfxVR.cpp:51 in mozilla::gfx::VRSystemManager::NotifyVSync(void)

Status: RESOLVED FIXED

(Core :: WebVR, defect)

Description

#[markdown(off)]
Filed by: rgurzau [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=235118100&repo=autoland

https://queue.taskcluster.net/v1/task/bzLFXB1ORs2EA2H6wLglfA/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/bzLFXB1ORs2EA2H6wLglfA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

04:00:27 INFO - [Parent 10656, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
04:00:28 INFO - 1553140828145 Marionette TRACE Received observer notification xpcom-will-shutdown
04:00:28 INFO - 1553140828145 Marionette INFO Stopped listening on port 2828
04:00:28 INFO - 1553140828145 Marionette DEBUG Remote service is inactive
04:00:28 INFO - [VR 1756, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
04:00:28 INFO - [GPU 860, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
04:00:28 INFO - =================================================================
04:00:28 ERROR - ==860==ERROR: AddressSanitizer: heap-use-after-free on address 0x11c956780808 at pc 0x7fff7c7f0bb0 bp 0x005f325fe8e0 sp 0x005f325fe928
04:00:28 INFO - WRITE of size 8 at 0x11c956780808 thread T2
04:00:28 INFO - ###!!! [Child][MessageChannel] Error: (msgtype=0x9B0002,name=PVRGPU::Msg_StopVRService) Closed channel: cannot send/recv
04:00:28 INFO - [GPU 860, Chrome_ChildThread] WARNING: pipe er
04:00:28 INFO - ###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
04:00:28 INFO - ror: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
04:00:28 INFO - [GPU 860, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
04:00:28 INFO - [Parent 10656, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
04:00:28 INFO - [GPU 860, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
04:00:28 INFO - #0 0x7fff7c7f0baf in mozilla::gfx::VRSystemManager::NotifyVSync(void) z:\build\build\src\gfx\vr\gfxVR.cpp:51
04:00:28 INFO - #1 0x7fff7c7daa57 in mozilla::gfx::VRSystemManagerPuppet::NotifyVSync(void) z:\build\build\src\gfx\vr\gfxVRPuppet.cpp:669
04:00:28 INFO - #2 0x7fff7c7e88e5 in mozilla::gfx::VRManager::NotifyVsync(class mozilla::TimeStamp const &) z:\build\build\src\gfx\vr\VRManager.cpp:208
04:00:28 INFO - #3 0x7fff7c7da9c8 in mozilla::gfx::VRSystemManagerPuppet::Run10msTasks(void) z:\build\build\src\gfx\vr\gfxVRPuppet.cpp:665
04:00:28 INFO - #4 0x7fff7c7e9f04 in mozilla::gfx::VRManager::Run10msTasks(void) z:\build\build\src\gfx\vr\VRManager.cpp:335
04:00:28 INFO - #5 0x7fff7c7e90a1 in mozilla::gfx::VRManager::RunTasks(void) z:\build\build\src\gfx\vr\VRManager.cpp:274
04:00:28 INFO - #6 0x7fff793033e4 in nsTimerImpl::Fire(int) z:\build\build\src\xpcom\threads\nsTimerImpl.cpp:559
04:00:28 INFO - #7 0x7fff79302975 in nsTimerEvent::Run(void) z:\build\build\src\xpcom\threads\TimerThread.cpp:260
04:00:28 INFO - #8 0x7fff7a3519e3 in ?DeferOrRunPendingTask@MessageLoop@@IEAA_N$$QEAUPendingTask@1@@Z z:\build\build\src\ipc\chromium\src\base\message_loop.cc:450
04:00:28 INFO - #9 0x7fff7a3533de in MessageLoop::DoWork(void) z:\build\build\src\ipc\chromium\src\base\message_loop.cc:523
04:00:28 INFO - #10 0x7fff7a3239b1 in base::MessagePumpForUI::DoRunLoop(void) z:\build\build\src\ipc\chromium\src\base\message_pump_win.cc:203
04:00:28 INFO - #11 0x7fff7a325fd9 in base::MessagePumpWin::Run(class base::MessagePump::Delegate *) z:\build\build\src\ipc\chromium\src\base\message_pump_win.h:79
04:00:28 INFO - #12 0x7fff7a35075e in MessageLoop::RunHandler(void) z:\build\build\src\ipc\chromium\src\base\message_loop.cc:308
04:00:28 INFO - #13 0x7fff7a362682 in base::Thread::ThreadMain(void) z:\build\build\src\ipc\chromium\src\base\thread.cc:192
04:00:28 INFO - #14 0x7fff7a3277ef in `anonymous namespace'::ThreadFunc z:\build\build\src\ipc\chromium\src\base\platform_thread_win.cc:19
04:00:28 INFO - #15 0x7fffbe23e888 in __asan::AsanThread::ThreadStart(unsigned __int64,struct __sanitizer::atomic_uintptr_t *) Z:\task_1553018212\build\src\build\build-clang\build-clang\src\llvm\projects\compiler-rt\lib\asan\asan_thread.cc:264
04:00:28 INFO - #16 0x7fffcfc53033 (C:\Windows\System32\KERNEL32.DLL+0x180013033)
04:00:28 INFO - #17 0x7fffca7adf21 in patched_BaseThreadInitThunk z:\build\build\src\mozglue\build\WindowsDllBlocklist.cpp:712
04:00:28 INFO - #18 0x7fffd1801460 (C:\Windows\SYSTEM32\ntdll.dll+0x180071460)
04:00:28 INFO - 0x11c956780808 is located 8 bytes inside of 73760-byte region [0x11c956780800,0x11c956792820)

Comment 2

[:kip (Kearwood Gilbert)]

This may be the same core cause as Bug 1536847.

In VRManager.h:

typedef nsRefPtrHashtable<nsUint32HashKey, gfx::VRDisplayHost>
VRDisplayHostHashMap;
VRDisplayHostHashMap mVRDisplays;

mVRDisplays is a weak pointer. This pointer is cleared in VRManager::Shutdown() and at the end of running scheduled tasks run by VRManager's nsTimer. The problem arises when the VRDisplayHost is removed by a VRSystemManager and is accessed by VRManager before VRManager's mVRDisplays is updated.

Proposed solution, to be applied in Bug 1536847:

• Remove VRManager::mVRDisplays
• Ask every VRSystemManager for the list of VRDisplayHost it knows about when we need to access them, but don't retain the VRDisplayHost inside VRManager
• Make an nsTArray<uint32_t> mVRDisplayIDs for VRManager
• VRManager::RefreshVRDisplays uses mVRDisplayIDs to compare the new list of VRDisplayHost's returned by the VRSystemManager's to what it saw last time. When different, it can set displaySetChanged to true

This is expected to fix this bug also. I'll take this bug and verify the fix corrects both before closing this one.

Comment 3

[Julien Cristau [:jcristau]]

Do we know what other branches are affected?

Comment 5

[Julien Cristau [:jcristau]]

Kip, any update here?

Comment 6

[:kip (Kearwood Gilbert)]

Bug 1522358 has landed, which corrects a range of shutdown race bugs, including this one.

Comment 7

[:kip (Kearwood Gilbert)]

(In reply to Julien Cristau [:jcristau] from comment #3)

Do we know what other branches are affected?

A conservative / inclusive range would start after landing Bug 1473397 (Implement haptic feedback support for gfxVRExternal and OpenVRSession).

This would be limited to 64-bit Windows and MacOS platforms.

Comment 8

[:kip (Kearwood Gilbert)]

I'll mark this as fixed. Please NI me if more action is needed.