Closed Bug 1485364 Opened 1 year ago Closed 1 year ago

Crash in vrclient_x64.dll@0xa0595

Categories

(Core :: WebVR, defect, critical)

Unspecified
Windows 10
defect
Not set
critical

Tracking

()

RESOLVED FIXED
mozilla64
Tracking Status
firefox-esr52 --- unaffected
firefox-esr60 --- unaffected
firefox61 --- wontfix
firefox62 --- wontfix
firefox63 --- wontfix
firefox64 --- fixed

People

(Reporter: marcia, Assigned: kip)

References

Details

(Keywords: crash, regression)

Crash Data

This bug was filed from the Socorro interface and is
report bp-61d4789f-13f2-47d8-9298-81e7c0180822.
=============================================================

Seen while looking at nightly crash data: https://bit.ly/2MsWnE0. Fairly low volume Windows 10 crash. There are also similar signatures in 61/62, and one Mac crash.

Top 4 frames of crashing thread:

0 vrclient_x64.dll vrclient_x64.dll@0xa0595 
1 vrclient_x64.dll vrclient_x64.dll@0x2972ff 
2 vrclient_x64.dll vrclient_x64.dll@0x630fd 
3 vrclient_x64.dll vrclient_x64.dll@0x297327 

=============================================================
Kip, this sounds like something up your alley.
(In reply to Ryan VanderMeulen [:RyanVM] from comment #1)
> Kip, this sounds like something up your alley.

Thanks for the head's up.  This vrclient_x64.dll is part of the OpenVR runtimes, most often used with the HTC Vive.  I'll investigate.
Assignee: nobody → kgilbert
Flags: needinfo?(kgilbert)
https://crash-stats.mozilla.com/report/index/02cad6b7-6a0c-41b7-a4c9-b21200180901#tab-details.

I don't see there is any connection between vrclient_x64.dll and nsTArray::AppendElement(). Does it give us more information after moving VRManager::RemoveVRManagerParent to VRListener thread? Doing Add/RemoveVRManagerParent at VRListener makes sense to me.

Currently, I am not quite sure if the crash is because nsTArray::AppendElement() makes vrclient_x64.dll crash or they are separated issues. vrclient_x64.dll crash comes from some SteamVR function calls that we need their symbol to debug and also has another issue that is related to nsArray VRManagerParent.
(In reply to Daosheng Mu[:daoshengmu] from comment #3)
> https://crash-stats.mozilla.com/report/index/02cad6b7-6a0c-41b7-a4c9-
> b21200180901#tab-details.
> 
> I don't see there is any connection between vrclient_x64.dll and
> nsTArray::AppendElement(). Does it give us more information after moving
> VRManager::RemoveVRManagerParent to VRListener thread? Doing
> Add/RemoveVRManagerParent at VRListener makes sense to me.
> 
> Currently, I am not quite sure if the crash is because
> nsTArray::AppendElement() makes vrclient_x64.dll crash or they are separated
> issues. vrclient_x64.dll crash comes from some SteamVR function calls that
> we need their symbol to debug and also has another issue that is related to
> nsArray VRManagerParent.

I am experimenting with moving RemoveVRManagerParent to the VRListener thread and trying to replicate this crash locally by artificially inducing the case where the VRManagerParent is removed while other messages (eg, SubmitFrameInternal) are in mid-flight.  There is a possibility that there are multiple causes involved.
(In reply to :kip (Kearwood Gilbert) from comment #4)
> (In reply to Daosheng Mu[:daoshengmu] from comment #3)
> > https://crash-stats.mozilla.com/report/index/02cad6b7-6a0c-41b7-a4c9-
> > b21200180901#tab-details.
> > 
> > I don't see there is any connection between vrclient_x64.dll and
> > nsTArray::AppendElement(). Does it give us more information after moving
> > VRManager::RemoveVRManagerParent to VRListener thread? Doing
> > Add/RemoveVRManagerParent at VRListener makes sense to me.
> > 
> > Currently, I am not quite sure if the crash is because
> > nsTArray::AppendElement() makes vrclient_x64.dll crash or they are separated
> > issues. vrclient_x64.dll crash comes from some SteamVR function calls that
> > we need their symbol to debug and also has another issue that is related to
> > nsArray VRManagerParent.
> 
> I am experimenting with moving RemoveVRManagerParent to the VRListener
> thread and trying to replicate this crash locally by artificially inducing
> the case where the VRManagerParent is removed while other messages (eg,
> SubmitFrameInternal) are in mid-flight.  There is a possibility that there
> are multiple causes involved.

It sounds like we need a MutexLock when there is anyone trying to modify the VRManagerParent array.
Another interesting part is this crash starts after July 27th. We should consider to bisect it. It could be a new regression.
Adding a few more signatures from crash stats seen during nightly crash triage.
Crash Signature: [@ vrclient_x64.dll@0xa0595] [@ vrclient.dylib@0x1df843] [@ vrclient_x64.dll@0x9ffa5] → [@ vrclient_x64.dll@0xa0595] [@ vrclient.dylib@0x1df843] [@ vrclient_x64.dll@0x9ffa5] [@ vrclient_x64.dll@0xa2865] [@ vrclient.dylib@0x1cb2cb]
This may be related to some intermittent test failures I encountered while working on Bug 1473397 (Implement haptic feedback support for gfxVRExternal and OpenVRSession).  I have made a change as part of Bug 1473397's patchset to run VRManager::RemoveVRManagerParent in the VRListenerThread.

There is a chance this could help and/or fix this issue also.
Crash Signature: [@ vrclient_x64.dll@0xa0595] [@ vrclient.dylib@0x1df843] [@ vrclient_x64.dll@0x9ffa5] [@ vrclient_x64.dll@0xa2865] [@ vrclient.dylib@0x1cb2cb] → [@ vrclient_x64.dll@0xa0595] [@ vrclient.dylib@0x1df843] [@ vrclient_x64.dll@0x9ffa5] [@ vrclient_x64.dll@0xa2865] [@ vrclient.dylib@0x1cb2cb] [@ vrclient.dylib@0x1cb2bb]
Please note that we are removing the VRListenerThread entirelly in Bug 1494556 (Crash in TimerThread::RemoveTimer)

This is part of the overall refactoring needed for Bug 1362578 ([meta] Move VRService to its own process)

I'll keep this open until I can verify that these crashes are also solved with these changes.
Crash Signature: [@ vrclient_x64.dll@0xa0595] [@ vrclient.dylib@0x1df843] [@ vrclient_x64.dll@0x9ffa5] [@ vrclient_x64.dll@0xa2865] [@ vrclient.dylib@0x1cb2cb] [@ vrclient.dylib@0x1cb2bb] → [@ vrclient_x64.dll@0xa0595] [@ vrclient.dylib@0x1df843] [@ vrclient_x64.dll@0x9ffa5] [@ vrclient_x64.dll@0xa2865] [@ vrclient.dylib@0x1cb2cb] [@ vrclient.dylib@0x1cb2bb] [@ vrclient.dylib@0x1cd4fb]
This has not recurred since the patch for Bug 1494556 (Crash in TimerThread::RemoveTimer) has landed.  The 2 recent exceptions are from builds older than the fix.
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Depends on: 1494556
Target Milestone: --- → mozilla64
You need to log in before you can comment on or make changes to this bug.