1485364 - Crash in vrclient_x64.dll@0xa0595

Status: RESOLVED FIXED

(Core :: WebVR, defect)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is
report bp-61d4789f-13f2-47d8-9298-81e7c0180822.
=============================================================

Seen while looking at nightly crash data: https://bit.ly/2MsWnE0. Fairly low volume Windows 10 crash. There are also similar signatures in 61/62, and one Mac crash.

Top 4 frames of crashing thread:

0 vrclient_x64.dll vrclient_x64.dll@0xa0595 
1 vrclient_x64.dll vrclient_x64.dll@0x2972ff 
2 vrclient_x64.dll vrclient_x64.dll@0x630fd 
3 vrclient_x64.dll vrclient_x64.dll@0x297327 

=============================================================

Comment 1

[(PTO June 19-July 7) Ryan VanderMeulen]

Kip, this sounds like something up your alley.

Comment 2

[:kip (Kearwood Gilbert)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #1)
> Kip, this sounds like something up your alley.

Thanks for the head's up.  This vrclient_x64.dll is part of the OpenVR runtimes, most often used with the HTC Vive.  I'll investigate.

Comment 3

[Daosheng Mu[:daoshengmu]]

https://crash-stats.mozilla.com/report/index/02cad6b7-6a0c-41b7-a4c9-b21200180901#tab-details.

I don't see there is any connection between vrclient_x64.dll and nsTArray::AppendElement(). Does it give us more information after moving VRManager::RemoveVRManagerParent to VRListener thread? Doing Add/RemoveVRManagerParent at VRListener makes sense to me.

Currently, I am not quite sure if the crash is because nsTArray::AppendElement() makes vrclient_x64.dll crash or they are separated issues. vrclient_x64.dll crash comes from some SteamVR function calls that we need their symbol to debug and also has another issue that is related to nsArray VRManagerParent.

Comment 4

[:kip (Kearwood Gilbert)]

(In reply to Daosheng Mu[:daoshengmu] from comment #3)
> https://crash-stats.mozilla.com/report/index/02cad6b7-6a0c-41b7-a4c9-
> b21200180901#tab-details.
> 
> I don't see there is any connection between vrclient_x64.dll and
> nsTArray::AppendElement(). Does it give us more information after moving
> VRManager::RemoveVRManagerParent to VRListener thread? Doing
> Add/RemoveVRManagerParent at VRListener makes sense to me.
> 
> Currently, I am not quite sure if the crash is because
> nsTArray::AppendElement() makes vrclient_x64.dll crash or they are separated
> issues. vrclient_x64.dll crash comes from some SteamVR function calls that
> we need their symbol to debug and also has another issue that is related to
> nsArray VRManagerParent.

I am experimenting with moving RemoveVRManagerParent to the VRListener thread and trying to replicate this crash locally by artificially inducing the case where the VRManagerParent is removed while other messages (eg, SubmitFrameInternal) are in mid-flight.  There is a possibility that there are multiple causes involved.

Comment 5

[Daosheng Mu[:daoshengmu]]

(In reply to :kip (Kearwood Gilbert) from comment #4)
> (In reply to Daosheng Mu[:daoshengmu] from comment #3)
> > https://crash-stats.mozilla.com/report/index/02cad6b7-6a0c-41b7-a4c9-
> > b21200180901#tab-details.
> > 
> > I don't see there is any connection between vrclient_x64.dll and
> > nsTArray::AppendElement(). Does it give us more information after moving
> > VRManager::RemoveVRManagerParent to VRListener thread? Doing
> > Add/RemoveVRManagerParent at VRListener makes sense to me.
> > 
> > Currently, I am not quite sure if the crash is because
> > nsTArray::AppendElement() makes vrclient_x64.dll crash or they are separated
> > issues. vrclient_x64.dll crash comes from some SteamVR function calls that
> > we need their symbol to debug and also has another issue that is related to
> > nsArray VRManagerParent.
> 
> I am experimenting with moving RemoveVRManagerParent to the VRListener
> thread and trying to replicate this crash locally by artificially inducing
> the case where the VRManagerParent is removed while other messages (eg,
> SubmitFrameInternal) are in mid-flight.  There is a possibility that there
> are multiple causes involved.

It sounds like we need a MutexLock when there is anyone trying to modify the VRManagerParent array.

Comment 6

[Daosheng Mu[:daoshengmu]]

Another interesting part is this crash starts after July 27th. We should consider to bisect it. It could be a new regression.

Comment 7

[Marcia Knous [:marcia]]

Adding a few more signatures from crash stats seen during nightly crash triage.

Comment 8

[:kip (Kearwood Gilbert)]

This may be related to some intermittent test failures I encountered while working on Bug 1473397 (Implement haptic feedback support for gfxVRExternal and OpenVRSession).  I have made a change as part of Bug 1473397's patchset to run VRManager::RemoveVRManagerParent in the VRListenerThread.

There is a chance this could help and/or fix this issue also.

Comment 9

[:kip (Kearwood Gilbert)]

Please note that we are removing the VRListenerThread entirelly in Bug 1494556 (Crash in TimerThread::RemoveTimer)

This is part of the overall refactoring needed for Bug 1362578 ([meta] Move VRService to its own process)

I'll keep this open until I can verify that these crashes are also solved with these changes.

Comment 10

[:kip (Kearwood Gilbert)]

This has not recurred since the patch for Bug 1494556 (Crash in TimerThread::RemoveTimer) has landed.  The 2 recent exceptions are from builds older than the fix.