[webvr] Implement “Puppet VR Device” and DOM api to control it

RESOLVED FIXED in Firefox 54

Status

()

defect
RESOLVED FIXED
4 years ago
2 years ago

People

(Reporter: kip, Assigned: daoshengmu)

Tracking

(Blocks 1 bug)

Trunk
mozilla54
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox45 affected, firefox54 fixed)

Details

(Whiteboard: [webvr])

Attachments

(1 attachment, 1 obsolete attachment)

No description provided.
Blocks: 1229481, 1229484
No longer blocks: 1229478, 1229479
Depends on: 1229497
Depends on: 1229498
Depends on: 1229499
Assignee: nobody → kgilbert
Blocks: 1186656
Whiteboard: [webvr]
This could be implemented as a standalone program that implements the OpenVR api and communicates with scripts through WebSockets
You can simulate Oculus hardware without having actual hardware with these steps:

- Ensure you are running Windows 8 or later with a discrete AMD or NVidia GPU.
- Update your graphics driver to the latest supplied directly from AMD or Nvidia.  Oculus requires new API functions only available in the latest drivers and may refuse to install or run without them.
- Download and install the Oculus setup application: https://www3.oculus.com/en-us/setup/
- Download the “Oculus SDK for Windows”: https://developer3.oculus.com/downloads/
- Un-zip the SDK files to any convenient directory.
- In the Tools directory in the SDK, run “OculusDebugTool.exe”
- In the debug tool, change “Debug HMD Type” from “none” to “CV1”
- A window will appear showing the view from the simulated headset.
- Leave this window open while running your tests.

There are some limitations with the Debug HMD provided by Oculus:

- It only runs on Windows
- It may self-update itself at any time
- Requires specific GPU drivers and a discrete AMD or Nvidia GPU to be present
- Simulated headset orientation and position can not be changed

To solve these issues, we intend to implement our own “puppet” VR device that emulates the OpenVR API.  It will also have some features to enable testing with our existing Reftest and Mochitest frameworks:
- Communicate via a local WebSocket connection with Javascript to enable scripting
- Provide HMD output screen captures for use in Reftests
- Provide the ability to run scripted events at a scheduled high-resolution timestamp, such as HMD disconnection, HMD connection, and sensor input simulation.
- Provide the ability to simulate a variety of devices with different sensors, frame rates, and resolutions.
- Provide performance metrics such as frame latency and throughput.
The puppet device API should receive a timeline of events and play them back as if they occurred with real hardware.  There would be events that capture frames and the state of the hardware to verify that the browser responded accurately.  (ie. check for frame input-to-render uniformity issues)

The timing needs to be measured with a high resolution system timer and done asynchronously from the tests.  Need to play back events and read back state without about 0.1ms accuracy (ideally < 0.01ms).

The frame capture mechanism would be built into the puppet implementation of VRDevice.submitFrame.  It would blit the texture to an on-gpu buffer of frames for later download to the CPU for analysis after the test has run.

We need to avoid readback from the GPU during the test as this would create GPU driver stalls that would throw off our measurements.
In the WebVR implementers meetup, there was interest in sharing a conformance suite of tests, in the same spirit as the WebGL Conformance Tests.

For this conformance suite to work cross-browser, either an inter-operable API or script file format describing the events to be played back on the simulated VR hardware needs to be specced.  An OpenVR driver could be implemented to enable these tests to be run on both Firefox and Servo desktop browsers.  It is not feasible for all other implementers to use an OpenVR based simulation tool, so the formats and/or protocols used to direct the simulated VR hardware should be agnostic of the underlying API.

I'd recommend that we create such an OpenVR based simulation tool and work with the other browser vendors on an interoperable format for describing the simulated events.  I'll create a separate bug to track the work on the conformance suite and tools.

There will be some implementation-specific tests, such as verifying our performance optimizations in the WebVR rendering path.  I'd suggest that these tests be built on a separate tool, integrated within Gecko as a subclass on VRDisplayHost.  This Gecko integrated driver would be enabled with a pref and expose a set of privileged API's used by our implementation specific reftests and mochitests.  Additional functions may be added to support Firefox specific devtools features.

I'll keep this bug open to track Gecko / Quantum's "Puppet VR" implementation as described.
See Also: → webvr-conformance
See Also: → 1323328
I have discussed with Kip and let me take over this bug to do the following work.
Assignee: kgilbert → dmu
Blocks: 1343356
We still need to find a way to help us block frame until the next vblank. Currently, I don't have a good solution. glFinish() and GetFrameLatencyWaitableObject() might could give us some ideas.
(In reply to Daosheng Mu[:daoshengmu] from comment #9)
> We still need to find a way to help us block frame until the next vblank.
> Currently, I don't have a good solution. glFinish() and
> GetFrameLatencyWaitableObject() might could give us some ideas.

I think we can still land some useful tests even if submitFrame doesn't block yet.

I'd recommend opening a separate bug to make submitFrame not only block, but also capture frames for use in reftests.  Please note that it doesn't necessarily need to block to an actual vsync.  It would be desirable to emulate arbitrary refresh rates and even refresh rates that vary after the test display has been created, to simulate effects such as ATW (Async Time Warp).
Comment on attachment 8840315 [details]
Bug 1229480: Implement Puppet VR Device;

https://reviewboard.mozilla.org/r/114814/#review117954

r=me with the little nit about extra whitespace at the end of the file

Also, please open a separate bug to implement the submitFrame blocking at configurable frame-rates.

::: gfx/vr/gfxVRPuppet.cpp:460
(Diff revision 2)
> +VRSystemManagerPuppet::RemoveControllers()
> +{
> +  mPuppetController.Clear();
> +  mControllerCount = 0;
> +}
> +

nit: extra lines at end of file
Attachment #8840315 - Flags: review?(kgilbert) → review+
Attachment #8828963 - Attachment is obsolete: true
Depends on: 1343730
Pushed by dmu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0f9b03faff35
Implement Puppet VR Device; r=kip
(In reply to Daosheng Mu[:daoshengmu] from comment #12)
> Comment on attachment 8840315 [details]
> Bug 1229480: Implement Puppet VR Device;
> 
> Review request updated; see interdiff:
> https://reviewboard.mozilla.org/r/114814/diff/2-3/

According to Comment 11 to fix nits and file Bug 1343730 to support submitFrame.
https://hg.mozilla.org/mozilla-central/rev/0f9b03faff35
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla54
You need to log in before you can comment on or make changes to this bug.