1522358 - Intermittent Last test finished | application crashed [@ mozilla::gfx::VRDisplayHost::~VRDisplayHost()]

Status: RESOLVED FIXED

(Core :: WebVR, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=223636536&repo=autoland

https://queue.taskcluster.net/v1/task/EhW31s6ITWCqDCDtTsxCOA/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/EhW31s6ITWCqDCDtTsxCOA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

00:59:41 INFO - REFTEST PROCESS-CRASH | Last test finished | application crashed [@ mozilla::gfx::VRDisplayHost::~VRDisplayHost()]
00:59:41 INFO - Crash dump filename: c:\users\task_1548289714\appdata\local\temp\tmpaylsje.mozrunner\minidumps\086ca470-3896-43f0-8dac-cd418147ee2d.dmp
00:59:41 INFO - Operating system: Windows NT
00:59:41 INFO - 10.0.15063
00:59:41 INFO - CPU: amd64
00:59:41 INFO - family 6 model 79 stepping 1
00:59:41 INFO - 16 CPUs
00:59:41 INFO -
00:59:41 INFO - GPU: UNKNOWN
00:59:41 INFO -
00:59:41 INFO - Crash reason: EXCEPTION_ILLEGAL_INSTRUCTION
00:59:41 INFO - Crash address: 0x7ffca26e59d0
00:59:41 INFO - Assertion: Unknown assertion type 0x00000000
00:59:41 INFO - Process uptime: 8 seconds
00:59:41 INFO -
00:59:41 INFO - Thread 5 (crashed)
00:59:41 INFO - 0 xul.dll!void mozilla::gfx::VRDisplayHost::~VRDisplayHost() [VRDisplayHost.cpp:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 85 + 0x0]
00:59:41 INFO - rax = 0x00007ffca8955020 rdx = 0x0000000000000001
00:59:41 INFO - rcx = 0x000002375fc75000 rbx = 0x0000000000000008
00:59:41 INFO - rsi = 0x000002375fc75000 rdi = 0x0000000000000000
00:59:41 INFO - rbp = 0x0000000000000001 rsp = 0x000000f7c3dff138
00:59:41 INFO - r8 = 0x0000000000000017 r9 = 0x3978ff73726e4778
00:59:41 INFO - r10 = 0xfefefefefefefeff r11 = 0x8080808080808080
00:59:41 INFO - r12 = 0x0000023755aea630 r13 = 0x7fffffffffffffff
00:59:41 INFO - r14 = 0x000000f7c3dff1c0 r15 = 0x00007ffca8955078
00:59:41 INFO - rip = 0x00007ffca26e59d0
00:59:41 INFO - Found by: given as instruction pointer in context
00:59:41 INFO - 1 xul.dll!void nsTArray_Impl<RefPtr<mozilla::gfx::VRDisplayHost>,nsTArrayInfallibleAllocator>::~nsTArray_Impl() [nsTArray.h:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 876 + 0x81]
00:59:41 INFO - rsp = 0x000000f7c3dff140 rip = 0x00007ffca26f2d3b
00:59:41 INFO - Found by: stack scanning
00:59:41 INFO - 2 xul.dll!mozilla::gfx::VRSystemManager::NotifyVSync() [gfxVR.cpp:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 53 + 0xa]
00:59:41 INFO - rsp = 0x000000f7c3dff1a0 rip = 0x00007ffca26f49de
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 3 xul.dll!mozilla::gfx::VRSystemManagerPuppet::NotifyVSync() [gfxVRPuppet.cpp:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 669 + 0x5]
00:59:41 INFO - rsp = 0x000000f7c3dff1f0 rip = 0x00007ffca26eb81e
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 4 xul.dll!mozilla::gfx::VRManager::NotifyVsync(mozilla::TimeStamp const &) [VRManager.cpp:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 206 + 0x6]
00:59:41 INFO - rsp = 0x000000f7c3dff230 rip = 0x00007ffca26f11f0
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 5 xul.dll!mozilla::gfx::VRSystemManagerPuppet::Run10msTasks() [gfxVRPuppet.cpp:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 665 + 0xb]
00:59:41 INFO - rsp = 0x000000f7c3dff270 rip = 0x00007ffca26eb7c2
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 6 xul.dll!mozilla::gfx::VRManager::Run10msTasks() [VRManager.cpp:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 325 + 0x6]
00:59:41 INFO - rsp = 0x000000f7c3dff2d0 rip = 0x00007ffca26f1854
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 7 xul.dll!mozilla::gfx::VRManager::RunTasks() [VRManager.cpp:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 264 + 0x8]
00:59:41 INFO - rsp = 0x000000f7c3dff340 rip = 0x00007ffca26f15a1
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 8 xul.dll!nsTimerImpl::Fire(int) [nsTimerImpl.cpp:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 559 + 0xd]
00:59:41 INFO - rsp = 0x000000f7c3dff400 rip = 0x00007ffca110379f
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 9 xul.dll!nsTimerEvent::Run() [TimerThread.cpp:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 260 + 0x8]
00:59:41 INFO - rsp = 0x000000f7c3dff560 rip = 0x00007ffca110337c
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 10 xul.dll!MessageLoop::RunTask(already_AddRefed<nsIRunnable>) [message_loop.cc:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 442 + 0x6]
00:59:41 INFO - rsp = 0x000000f7c3dff5d0 rip = 0x00007ffca180a90f
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 11 xul.dll!MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask &&) [message_loop.cc:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 450 + 0x8]
00:59:41 INFO - rsp = 0x000000f7c3dff650 rip = 0x00007ffca180b43c
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 12 xul.dll!MessageLoop::DoWork() [message_loop.cc:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 523 + 0xb]
00:59:41 INFO - rsp = 0x000000f7c3dff6a0 rip = 0x00007ffca180b77b
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 13 xul.dll!base::MessagePumpForUI::DoRunLoop() [message_pump_win.cc:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 203 + 0x9]
00:59:41 INFO - rsp = 0x000000f7c3dff740 rip = 0x00007ffca17f4909
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 14 xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate *) [message_pump_win.h:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 79 + 0x57]
00:59:41 INFO - rsp = 0x000000f7c3dff7f0 rip = 0x00007ffca17f5637
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 15 xul.dll!MessageLoop::RunHandler() [message_loop.cc:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 308 + 0x8]
00:59:41 INFO - rsp = 0x000000f7c3dff850 rip = 0x00007ffca180a4d2
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 16 xul.dll!base::Thread::ThreadMain() [thread.cc:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 192 + 0x5a]
00:59:41 INFO - rsp = 0x000000f7c3dff8a0 rip = 0x00007ffca18146ae
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 17 xul.dll!static unsigned long `anonymous namespace'::ThreadFunc(void *) [platform_thread_win.cc:0d52389e664d34f9f7315e59cb7127c32b90b2e7 : 30 + 0x6]
00:59:41 INFO - rsp = 0x000000f7c3dffac0 rip = 0x00007ffca17f614a
00:59:41 INFO - Found by: call frame info
00:59:41 INFO - 18 kernel32.dll!LdrpLoadResourceFromAlternativeModule + 0x27c
00:59:41 INFO - rsp = 0x000000f7c3dffaf0 rip = 0x00007ffcd9152774
00:59:41 INFO - Found by: call frame info

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Daosheng, this looks like a regression from bug 1518895. Please take a look.

Comment 2

[Daosheng Mu[:daoshengmu]]

I agree. It is caused by calling VRManager::Destory() as I mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1522576#c0. We will fix it soon.

Comment 3

[Intermittent Failures Robot]

3 failures in 2383 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 1

Platform breakdown:

• windows10-64-ccov: 1
• windows10-64: 1
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1522358&startday=2019-01-21&endday=2019-01-27&tree=all

Comment 4

[Intermittent Failures Robot]

1 failures in 2628 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1522358&startday=2019-01-28&endday=2019-02-03&tree=all

Comment 5

[Intermittent Failures Robot]

2 failures in 2981 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1
• autoland: 1

Platform breakdown:

• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1522358&startday=2019-02-25&endday=2019-03-03&tree=all

Comment 6

[Daosheng Mu[:daoshengmu]]

I think we have resolved this in Bug 1523923. We will keep an eye on this.

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

After not occurring for 3 weeks, this has failed 4 times in automation since February 25th. Last fail on a production branch is https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=8fd4c49718305ec4d1f95c6d5c62f1ff798249e5&selectedJob=231073903

Comment 8

[Intermittent Failures Robot]

1 failures in 2818 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1522358&startday=2019-03-04&endday=2019-03-10&tree=all

Comment 9

[Intermittent Failures Robot]

25 failures in 2771 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 16
• mozilla-central: 7
• try: 1
• mozilla-inbound: 1

Platform breakdown:

• windows10-64: 7
• windows10-64-ccov: 4
• windows10-64-qr: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1522358&startday=2019-03-11&endday=2019-03-17&tree=all

Comment 10

[Andrei Ciure[:aciure]]

There are 35 failures associated to this bug in the last 7 days. These are occurring on windows10-64 and windows10-64-qr debug.

:daoshengmu can you take a look?

Comment 11

[Daosheng Mu[:daoshengmu]]

np. I am going to investigate this during this week.

Comment 12

[Daosheng Mu[:daoshengmu]]

I can't reproduce it by ./mach reftest dom/vr/test/reftest/ --verify --repeat=10. However, I think we might need to use a cancelable task for VRManager::TaskTimerCallback and cancel it when the VR service is shutdown.

Comment 13

[Intermittent Failures Robot]

16 failures in 475 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 2
• mozilla-inbound: 2

Platform breakdown:

• windows10-64-qr: 12
• windows10-64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1522358&startday=2019-03-21&endday=2019-03-21&tree=all

Comment 14

[Daosheng Mu[:daoshengmu]]

Attachment: Bug 1522358 - Requery VRDisplayHost from VRManagers instead of using weakptr from the hash table.

Comment 15

[Daosheng Mu[:daoshengmu]]

This patch will help fix other stability bugs as well. The root cause is we were using VRDisplayHostHashMap. This hash table stores weak ptr of VRDisplayHost and these VRDisplayHost ptr will be released when VRService calls VRManager::Shutdown(). So, we should use an array to keep VRDisplayId and check if they are available at runtime. If the id is changed, we need notify to update display info by trigger displaySetChanged.

In terms of VRControllerHostHashMap, we might don't need to change it because we always get length 0 from VRSystemManagerExternal because we handle it at VRDisplayClient. But, to be consistent, I still do adjustments to make it to be mVRControllerIDs array.

Comment 16

[Intermittent Failures Robot]

19 failures in 537 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-inbound: 7

Platform breakdown:

• windows10-64-qr: 14
• windows10-64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1522358&startday=2019-03-22&endday=2019-03-22&tree=all

Comment 17

[Intermittent Failures Robot]

60 failures in 2561 pushes (0.023 failures/push) were associated with this bug in the last 7 days.

This is the #40 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 41
• mozilla-central: 6
• mozilla-inbound: 13

Platform breakdown:

• windows10-64-qr: 44
• windows10-64: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1522358&startday=2019-03-18&endday=2019-03-24&tree=all

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/87be6cf9c45b
Requery VRDisplayHost from VRManagers instead of using weakptr from the hash table. r=kip

Comment 19

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/87be6cf9c45b

Comment 20

[Ryan VanderMeulen [:RyanVM]]

I don't see these failures on Beta67, but I believe the regressing change landed in that timeframe. Is there a user-facing issue demonstrated by this failure which would make us want to consider uplifting this patch or can it ride the trains?

Comment 21

[Daosheng Mu[:daoshengmu]]

If we don't see these failures on Beta 67, that means it only happens in Nightly when we turn on dom.vr.process.enabled. This change also causes an another bug like Bug 1539185. We will fix them in FF 68 and turn on dom.vr.process.enabled for release channel soon. Probably, it is needless to uplift this to cause risks.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

If it's still behind a pref in 67, I completely agree. Thanks!

Comment 23

[Intermittent Failures Robot]

14 failures in 3910 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 4
• mozilla-inbound: 2

Platform breakdown:

• windows10-64-qr: 8
• windows10-64: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1522358&startday=2019-03-25&endday=2019-03-31&tree=all