1299930 - [webvr] Support HTC Vive controller trackpads through the Gamepad API

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[:kip (Kearwood Gilbert)]

Each HTC Vive Controllers include an independent, clickable trackpad.

Comment 1

[Daosheng Mu[:daoshengmu]]

I already have a prototype for supporting trackpads via Gamepad API's buttons and axes attributes.

Kip, do you think we should wait for the conclusion of https://github.com/w3c/gamepad/issues/34, or we can support it in my current way first?

Comment 2

[:kip (Kearwood Gilbert)]

(In reply to Daosheng Mu[:daoshengmu] from comment #1)
> I already have a prototype for supporting trackpads via Gamepad API's
> buttons and axes attributes.
> 
> Kip, do you think we should wait for the conclusion of
> https://github.com/w3c/gamepad/issues/34, or we can support it in my current
> way first?

I will bring this up with the rest of the WebVR team and in the implementers meeting.  (We meet every tuesday).

Ideally, we can make it easy for developers to find the trackpad axis on a wide range of controllers and support features such as multitouch through the same interface.

This may be either a new webidl interface in the Gamepad API or it could be some attributes to provide metadata for mapping the axis and buttons.

I'll keep the NI open until I can better answer your question.

Comment 3

[Daosheng Mu[:daoshengmu]]

In the VR controller sample from WebVR 1.1 in Chromium, https://github.com/toji/webvr-samples/blob/1cc580b81ad6afb268febbf1698624be6d7373b1/XX-vr-controllers.html#L277. They use buttons and axes attributes for trackpads as well. Probably, we can follow this way.

Comment 4

[Daosheng Mu[:daoshengmu]]

Attachment: Bug 1299930 Support OpenVR controller trackpads

This is close to review until we confirm some details about if we should define more attributes for trackpads.

Comment 5

[Daosheng Mu[:daoshengmu]]

Comment on attachment 8802416 [details] [diff] [review]
Bug 1299930 Support OpenVR controller trackpads

>From 56d4824d175f2211c2457b3509782d332891a1ab Mon Sep 17 00:00:00 2001
>From: Daosheng Mu <[e-mail]>
>Date: Wed, 19 Oct 2016 15:46:44 +0800
>Subject: [PATCH] Bug 1299930 - Support OpenVR controller trackpads via Gamepad
> API;
>
>MozReview-Commit-ID: 41C0rc7fZTi
>---
> gfx/vr/gfxVR.cpp       | 12 ++++++++++
> gfx/vr/gfxVR.h         |  3 +++
> gfx/vr/gfxVROpenVR.cpp | 60 +++++++++++++++++++++++++++++++++++++++++---------
> gfx/vr/gfxVROpenVR.h   |  2 ++
> 4 files changed, 66 insertions(+), 11 deletions(-)
>
>diff --git a/gfx/vr/gfxVR.cpp b/gfx/vr/gfxVR.cpp
>index 23eefbc..3a53bb9 100644
>--- a/gfx/vr/gfxVR.cpp
>+++ b/gfx/vr/gfxVR.cpp
>@@ -84,8 +84,20 @@ VRControllerManager::NewButtonEvent(uint32_t aIndex, uint32_t aButton,
> {
>   dom::GamepadButtonInformation a(aIndex, dom::GamepadServiceType::VR,
>                                   aButton, aPressed, aPressed ? 1.0L : 0.0L);
> 
>   VRManager* vm = VRManager::Get();
>   MOZ_ASSERT(vm);
>   vm->NotifyGamepadChange<dom::GamepadButtonInformation>(a);
> }
>+
>+void
>+VRControllerManager::NewAxisMove(uint32_t aIndex, uint32_t aAxis,
>+                                 double aValue)
>+{
>+  dom::GamepadAxisInformation a(aIndex, dom::GamepadServiceType::VR,
>+                                aAxis, aValue);
>+
>+  VRManager* vm = VRManager::Get();
>+  MOZ_ASSERT(vm);
>+  vm->NotifyGamepadChange<dom::GamepadAxisInformation>(a);
>+}
>\ No newline at end of file
>diff --git a/gfx/vr/gfxVR.h b/gfx/vr/gfxVR.h
>index 9f4d099..50c55d7 100644
>--- a/gfx/vr/gfxVR.h
>+++ b/gfx/vr/gfxVR.h
>@@ -247,28 +247,31 @@ public:
> 
>   static uint32_t AllocateControllerID();
>   virtual bool Init() = 0;
>   virtual void Destroy() = 0;
>   virtual void HandleInput() = 0;
>   virtual void GetControllers(nsTArray<RefPtr<VRControllerHost>>& aControllerResult) = 0;
>   virtual void ScanForDevices() = 0;
>   void NewButtonEvent(uint32_t aIndex, uint32_t aButton, bool aPressed);
>+  void NewAxisMove(uint32_t aIndex, uint32_t aAxis, double aValue);
>   void AddGamepad(const char* aID, dom::GamepadMappingType aMapping,
>                   uint32_t aNumButtons, uint32_t aNumAxes);
> 
> protected:
>   VRControllerManager() : mInstalled(false), mControllerCount(0) {}
>   virtual ~VRControllerManager() {}
> 
>   bool mInstalled;
>   uint32_t mControllerCount;
>   static Atomic<uint32_t> sControllerBase;
> 
> private:
>   virtual void HandleButtonPress(uint32_t aControllerIdx,
>                                  uint64_t aButtonPressed) = 0;
>+  virtual void HandleAxisMove(uint32_t aControllerIdx, uint32_t aAxis,
>+                              float aValue, int32_t aInvert) = 0;
> };
> 
> } // namespace gfx
> } // namespace mozilla
> 
> #endif /* GFX_VR_H */
>diff --git a/gfx/vr/gfxVROpenVR.cpp b/gfx/vr/gfxVROpenVR.cpp
>index 7bc80e1..fbfda79 100644
>--- a/gfx/vr/gfxVROpenVR.cpp
>+++ b/gfx/vr/gfxVROpenVR.cpp
>@@ -67,26 +67,40 @@ const uint64_t gOpenVRButtonMask[] = {
>   // vr::ButtonMaskFromId(vr::EVRButtonId::k_EButton_A),
>   vr::ButtonMaskFromId(vr::EVRButtonId::k_EButton_SteamVR_Touchpad),
>   vr::ButtonMaskFromId(vr::EVRButtonId::k_EButton_SteamVR_Trigger)
> };
> 
> const uint32_t gNumOpenVRButtonMask = sizeof(gOpenVRButtonMask) /
>                                       sizeof(uint64_t);
> 
>-const uint64_t gOpenVRAxisMask[] = {
>-  vr::ButtonMaskFromId(vr::EVRButtonId::k_EButton_Axis0),
>-  vr::ButtonMaskFromId(vr::EVRButtonId::k_EButton_Axis1),
>-  vr::ButtonMaskFromId(vr::EVRButtonId::k_EButton_Axis2),
>-  vr::ButtonMaskFromId(vr::EVRButtonId::k_EButton_Axis3),
>-  vr::ButtonMaskFromId(vr::EVRButtonId::k_EButton_Axis4)
>+enum class VRControllerAxisType : uint16_t {
>+  TrackpadXAxis,
>+  TrackpadYAxis,
>+  Trigger,
>+  NumVRControllerAxisType
> };
> 
>-const uint32_t gNumOpenVRAxisMask = sizeof(gOpenVRAxisMask) /
>-                                    sizeof(uint64_t);
>+struct VRControllerAxisInfo
>+{
>+  uint32_t mIdx;
>+  int32_t mInvert;
>+
>+  VRControllerAxisInfo(vr::EVRButtonId aId, uint32_t aInvert)
>+    : mIdx(aId - vr::EVRButtonId::k_EButton_Axis0), mInvert(aInvert) {}
>+};
>+
>+const VRControllerAxisInfo gOpenVRAxes[] = {
>+  VRControllerAxisInfo(vr::EVRButtonId::k_EButton_Axis0, 1),
>+  VRControllerAxisInfo(vr::EVRButtonId::k_EButton_Axis0, -1),

This is strange for trackpad axis, but we have a spec at GamepadAPI (https://w3c.github.io/gamepad/#remapping). It mentions negative is up, positive is down.


>+  VRControllerAxisInfo(vr::EVRButtonId::k_EButton_Axis1, 1)
>+};
>+
>+const uint32_t gNumOpenVRAxis = sizeof(gOpenVRAxes) /
>+                                sizeof(VRControllerAxisInfo);
> 
> bool
> LoadOpenVRRuntime()
> {
>   static PRLibrary *openvrLib = nullptr;
> 
>   nsAdoptingCString openvrPath = Preferences::GetCString("gfx.vr.openvr-runtime");
>   if (!openvrPath)
>@@ -474,17 +488,17 @@ VRDisplayManagerOpenVR::GetHMDs(nsTArray<RefPtr<VRDisplayHost>>& aHMDResult)
> 
> VRControllerOpenVR::VRControllerOpenVR()
>   : VRControllerHost(VRDeviceType::OpenVR)
> {
>   MOZ_COUNT_CTOR_INHERITED(VRControllerOpenVR, VRControllerHost);
>   mControllerInfo.mControllerName.AssignLiteral("OpenVR HMD");
>   mControllerInfo.mMappingType = dom::GamepadMappingType::_empty;
>   mControllerInfo.mNumButtons = gNumOpenVRButtonMask;
>-  mControllerInfo.mNumAxes = gNumOpenVRAxisMask;
>+  mControllerInfo.mNumAxes = gNumOpenVRAxis;
> }
> 
> VRControllerOpenVR::~VRControllerOpenVR()
> {
>   MOZ_COUNT_DTOR_INHERITED(VRControllerOpenVR, VRControllerHost);
> }
> 
> void
>@@ -554,32 +568,46 @@ VRControllerManagerOpenVR::Destroy()
>   mOpenVRInstalled = false;
> }
> 
> void
> VRControllerManagerOpenVR::HandleInput()
> {
>   RefPtr<impl::VRControllerOpenVR> controller;
>   vr::VRControllerState_t state;
>+  uint32_t axis = 0;
> 
>   MOZ_ASSERT(mVRSystem);
> 
>   // Process OpenVR controller state
>   for (uint32_t i = 0; i < mOpenVRController.Length(); ++i) {
>     controller = mOpenVRController[i];
> 
>     MOZ_ASSERT(mVRSystem->GetTrackedDeviceClass(controller->GetTrackedIndex())
>                == vr::TrackedDeviceClass_Controller);
> 
>     if (mVRSystem->GetControllerState(controller->GetTrackedIndex(), &state)) {
>       if (state.ulButtonPressed) {
>         HandleButtonPress(controller->GetIndex(), state.ulButtonPressed);
>       }
> 
>-      // Handle Axis support in Bug 1299930
>+      axis = static_cast<uint32_t>(VRControllerAxisType::TrackpadXAxis);
>+      VRControllerAxisInfo axisInfo(gOpenVRAxes[axis]);
>+      HandleAxisMove(controller->GetIndex(), axis,
>+                     state.rAxis[axisInfo.mIdx].x, axisInfo.mInvert);
>+
>+      axis = static_cast<uint32_t>(VRControllerAxisType::TrackpadYAxis);
>+      axisInfo = gOpenVRAxes[axis];
>+      HandleAxisMove(controller->GetIndex(), axis,
>+                     state.rAxis[axisInfo.mIdx].y, axisInfo.mInvert);
>+
>+      axis = static_cast<uint32_t>(VRControllerAxisType::Trigger);
>+      axisInfo = gOpenVRAxes[axis];
>+      HandleAxisMove(controller->GetIndex(), axis,
>+                     state.rAxis[axisInfo.mIdx].x, axisInfo.mInvert);
>     }
>   }
> }
> 
> void
> VRControllerManagerOpenVR::HandleButtonPress(uint32_t aControllerIdx,
>                                              uint64_t aButtonPressed)
> {
>@@ -587,16 +615,26 @@ VRControllerManagerOpenVR::HandleButtonPress(uint32_t aControllerIdx,
> 
>   for (uint32_t i = 0; i < gNumOpenVRButtonMask; ++i) {
>     buttonMask = gOpenVRButtonMask[i];
>     NewButtonEvent(aControllerIdx, i, aButtonPressed & buttonMask);
>   }
> }
> 
> void
>+VRControllerManagerOpenVR::HandleAxisMove(uint32_t aControllerIdx, uint32_t aAxis,
>+                                          float aValue, int32_t aInvert)
>+{
>+  if (aValue != 0.0f) {
>+    printf_stderr("controller %d axis %d value %f \n", aControllerIdx, aAxis, aValue * aInvert);
>+    NewAxisMove(aControllerIdx, aAxis, aValue * aInvert);
>+  }
>+}
>+
>+void
> VRControllerManagerOpenVR::GetControllers(nsTArray<RefPtr<VRControllerHost>>& aControllerResult)
> {
>   if (!mOpenVRInstalled) {
>     return;
>   }
> 
>   aControllerResult.Clear();
>   for (uint32_t i = 0; i < mOpenVRController.Length(); ++i) {
>@@ -626,12 +664,12 @@ VRControllerManagerOpenVR::ScanForDevices()
> 
>     RefPtr<VRControllerOpenVR> openVRController = new VRControllerOpenVR();
>     openVRController->SetIndex(mControllerCount);
>     openVRController->SetTrackedIndex(trackedDevice);
>     mOpenVRController.AppendElement(openVRController);
> 
>     // Not already present, add it.
>     AddGamepad("OpenVR Gamepad", GamepadMappingType::_empty,
>-               gNumOpenVRButtonMask, gNumOpenVRAxisMask);
>+               gNumOpenVRButtonMask, gNumOpenVRAxis);
>     ++mControllerCount;
>   }
> }
>\ No newline at end of file
>diff --git a/gfx/vr/gfxVROpenVR.h b/gfx/vr/gfxVROpenVR.h
>index 636b7e7..3cbf4a3 100644
>--- a/gfx/vr/gfxVROpenVR.h
>+++ b/gfx/vr/gfxVROpenVR.h
>@@ -117,16 +117,18 @@ public:
>   virtual void ScanForDevices() override;
> 
> private:
>   VRControllerManagerOpenVR();
>   ~VRControllerManagerOpenVR();
> 
>   virtual void HandleButtonPress(uint32_t aControllerIdx,
>                                  uint64_t aButtonPressed) override;
>+  virtual void HandleAxisMove(uint32_t aControllerIdx, uint32_t aAxis,
>+                              float aValue, int32_t aInvert) override;
> 
>   bool mOpenVRInstalled;
>   nsTArray<RefPtr<impl::VRControllerOpenVR>> mOpenVRController;
>   vr::IVRSystem *mVRSystem;
> };
> 
> } // namespace gfx
> } // namespace mozilla
>-- 
>2.8.1.windows.1
>

Comment 6

[:kip (Kearwood Gilbert)]

I think it would be okay to implement the same as the Chromium WebVR builds, exposing as regular axis and button until https://github.com/w3c/gamepad/issues/34 is concluded.

Comment 7

[Daosheng Mu[:daoshengmu]]

Attachment: Bug 1299930 - Support OpenVR controller trackpads via Gamepad API

In this update, I remove the invert function for VRControllerAxis because I think axis move in [top, bottom] to be [1, -1] is more make sense.

Comment 8

[Daosheng Mu[:daoshengmu]]

Attachment: Bug 1299930 - Support OpenVR controller trackpads via Gamepad API;

Review commit: https://reviewboard.mozilla.org/r/87706/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/87706/

Comment 9

[:kip (Kearwood Gilbert)]

Comment on attachment 8803428 [details]
Bug 1299930 - Support OpenVR controller trackpads via Gamepad API;

https://reviewboard.mozilla.org/r/87706/#review86700

LGTM

Comment 11

[Daosheng Mu[:daoshengmu]]

Comment on attachment 8803428 [details]
Bug 1299930 - Support OpenVR controller trackpads via Gamepad API;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/87706/diff/1-2/

Comment 12

[Daosheng Mu[:daoshengmu]]

Rebase and try looks good, https://treeherder.mozilla.org/#/jobs?repo=try&revision=0a35397627fa

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6a52e0212b16
Support OpenVR controller trackpads via Gamepad API; r=kip

Comment 14

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/6a52e0212b16