680289 - (gamepad-rumble) Rumble Effect support for Gamepad API

Status: ASSIGNED

(Core :: DOM: Device Interfaces, enhancement, P4)

Description

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Once we have basic joystick support, we should add support for vibrator control.

I propose that we do this by adding the two properties, .hasVibrator and .vibratorStrength, to the joystick object introduced in bug 604039.

By adding the properties on the Joystick object, it ensures that pages don't get access to them unless the user has interacted with the joystick on the page. This should remove concerns about fingerprinting using the .hasVibrator property.

The main risk that I can see is that if the user accidentally uses the joystick while viewing a website, the website can annoy the user by turning on the vibrator. However I think this is an acceptable risk, and the user can fix it by either leaving the page or reloading it.

As for .vibratorStrength. I don't know if joysticks usually support a sliding scale, or a set number of distinct levels? Would simply making the strength be a number between 0 and 1 work? And if the joystick only supports distinct levels, this would turn on the lowest level that is at least as strong as the selected value.

Or should .hasVibrator return a number indicating the number of levels available?

Comment 1

[(not currently active) Ted Mielczarek]

It's also commonly referred to as "rumble" support. Note that some controllers have more than one motor, and you can control them independently. The XBox 360 controller has two motors, for example:
http://msdn.microsoft.com/en-us/library/microsoft.directx_sdk.reference.xinput_vibration%28v=vs.85%29.aspx

One is low-frequency, and one is high-frequency. (Unfortunately those APIs are only available via XInput, and my Win32 backend uses DirectInput currently.)

Comment 2

[John Drinkwater (:beta)]

I commented on Bug 679966 about rumble support for the device, what if I don’t have a joystick but do have rumble, where does the vibration interface get parented? See Xperia Play and PSVita for the direction mobiles will go.

Comment 3

[Bobby Richter [:secretrobotron]]

Attachment: Patch for applying gamepad vibration to gamepad API patchset

I have put a preliminary patch together to get this working in Linux. I built it last night, and I'm trying it again on another Linux machine as I write this.

It builds on bug #604039, including its patchset, so make sure you're basing off of that if you build this.

I have a reliable git branch here: https://github.com/secretrobotron/mozilla-central/tree/gamepad-api-vibration

Note: There is a bug which will probably screw up >1 controllers plugged in, but I know where it is, and it's gonna get axed today.

Comment 4

[Scott Graham]

Cool! I only skimmed the patch, but this is the interface on the gamepad, right?

    void setVibration(in unsigned long duration, in unsigned long strongMagnitude, in unsigned long weakMagnitude);

where it oscillates between the two magnitudes? That seems a bit over-specialized to the linux ff interface and rumble to me. Could it be simplified to just:

void vibrate(in unsigned long motor, in float normalizedMagnitude);

? (or setVibration, don't care) That's the interface I was thinking of hacking in myself. I guess gamepad objects would also need a numMotors added.

I'd also drop getVibration(). Seems like extra stuff that might be hard to query some places, unless it's really important for something.

http://www.w3.org/TR/2011/WD-vibration-20111117/ is, uh, there too. I don't know how important/useful it really is to try to make phone vibration and gamepad vibration the same.

Comment 5

[Scott Graham]

To expand/clarify, I was thinking that doing the equivalent of the current patch would then look like:

    pad.vibrate(0, strong_mag);
    pad.vibrate(1, weak_mag);

    // after duration ...

    var off = 0.0;
    pad.vibrate(0, off);
    pad.vibrate(1, off);

Comment 6

[Bobby Richter [:secretrobotron]]

> where it oscillates between the two magnitudes?
Not really. There are two motors in most XInput-ish gamepads now. FF on linux supports it by default with FF_RUMBLE. All the other kinds of FF actually seem more complex.

> That seems a bit over-specialized to the linux ff interface and rumble to me.

I wasn't really sure how to generalize this properly until I did a bit of searching. I still don't really have a good idea other than to just steal XInput's since it's used a lot now, and there's really not a lot wrong with it: http://msdn.microsoft.com/en-us/library/windows/desktop/ee417001(v=vs.85).aspx#setting_vibration

On my blog, I wrote about why I did what if you're interested: http://robothaus.org/blog/blog/2011/12/09/gamepad-vibration-zero-to-hero/

Very happy to have some feedback though. Nobody seems to know how to tackle this in a general way yet.

> http://www.w3.org/TR/2011/WD-vibration-20111117/ is, uh, there too. I don't know how important/useful it really is to try to make phone vibration and gamepad vibration the same.

I was actually considering this too. There's a WebVibrator bug (679966) that is attempting to deal with it already. There's no reason why we can't push this closer to that.

> I'd also drop getVibration(). Seems like extra stuff that might be hard to query some places, unless it's really important for something.

You're probably right, especially considering the push, on-demand nature of vibration. Keeps the DOM impl simpler too. I can pretty much scrap nsDOMGamepadEffect. I hesitate a bit though, because it might actually be useful in the future if we intend to support more effects, and not just simple rumbling. Might be overkill for this though.

Comment 7

[Bobby Richter [:secretrobotron]]

I should also expand on the "two motors" thing: in no way am I attached to always supporting two motors.

void vibrate( unsigned long duration, unsigned long strength );

actually looks perfect.

I was intending to finish the implementation for optional args here though, so...

void vibrate( unsigned long duration, unsigned long strength, unsigned long otherStrength = 0 );

Then, if there is another motor, we can use that other arg, but it's not required.

Comment 8

[(not currently active) Ted Mielczarek]

You'll note that the Vibration API spec supports two methods:
 void vibrate (optional unsigned long time)
Which is just "vibrate for |time| milliseconds at whatever frequency".

 void vibrate (optional unsigned long[] pattern)
Which is "alternate vibrating and being still for N milliseconds for N in pattern".

It would probably be nice to be able to generalize this to >1 vibration motor, since we know most gamepads that support vibration have 2. If we can get them to work it into that spec, even better.

Comment 9

[Bobby Richter [:secretrobotron]]

I don't mind that, only because it doesn't really try to expose any other features of the force-feedback. i.e. it only involves time, and there is a minimal amount of code to write to support the `pattern` functionality over and above supporting gamepad FF in general.

I'd like to keep our API close to something like XInput's, which is very simple. So, I definitely don't mind this:

void vibrate(optional unsigned long[] pattern, optional unsigned long weakMagnitude, optional unsigned long strongMagnitude);

Comment 10

[(not currently active) Ted Mielczarek]

Maybe that should be:
void vibrate(optional unsigned long[] pattern, optional unsigned long[] magnitudes);
?

That way you could support an arbitrary number of motors.

Also the first method could become:
void vibrate (optional unsigned long time, optional unsigned long magnitude)

To vibrate everything at a specified magnitude.

Comment 11

[Scott Graham]

(In reply to Ted Mielczarek [:ted, :luser] from comment #10)
> Maybe that should be:
> void vibrate(optional unsigned long[] pattern, optional unsigned long[]
> magnitudes);
> ?

sgtm, except I don't really like the 65536-based magnitudes. I'd prefer float (or double?) normalized magnitudes that are scaled based on the device&API when applied. Seem reasonable?

Comment 12

[(not currently active) Ted Mielczarek]

I am completely faking it here, just trying to achieve harmony with the existing WebVibration stuff. :)

Comment 13

[Justin Lebar (not reading bugmail)]

Can you vibrate the N motors independently?

If so, it seems like the gamepad should expose an array of objects on which you can call vibrate.

Comment 14

[Bobby Richter [:secretrobotron]]

> If so, it seems like the gamepad should expose an array of objects on which
> you can call vibrate.

That's not a bad idea. It does deviate quite a bit from WebVibrator though, and adds a bunch of new code to the prototype.

Simplicity might be key.

Comment 15

[Justin Lebar (not reading bugmail)]

Well, WebVibrator as spec'ed by w3c [1] is just an interface:

interface Vibration {
    void vibrate (optional unsigned long time) raises (NotSupportedError);
    void vibrate (optional unsigned long[] pattern) raises (NotSupportedError);
};

you can stick that interface on whatever object you'd like.

But I guess you're right -- having multiple vibrators isn't a problem exclusively related to gamepads, so maybe we should solve it in the vibrator API itself.

How does the page know what values it can pass for magnitude[]?

[1] http://dev.w3.org/2009/dap/vibration/

Comment 16

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Bobby: You still looking at this? If not I might be interested in taking it.

Comment 17

[Bobby Richter [:secretrobotron]]

By all means! Unfortunately, my bandwidth for this ran out and my code is based on Ted's year-old joystick patches, so it might not be a useful starting point. Happy to help if I still can.

Comment 18

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

So, in true "taking over a bug" style... 

This is not how force feedback usually works. You don't just "call vibrate". I realize we'd like to keep our API "simple", but if we'd like to future proof this, it's not a call I feel like we get to make against the rest of gaming in general.

As Bobby outlined in his blog post, FF is usually a set of effects that are built into a struct, then uploaded to the controller, in order to reduce bandwidth and make common effects reusable. So you can upload bullet firing effects once during weapons changes, for instance, while still operating on the usual USB low-speed connection most HID devices enjoy. That's why there's things like RAM/ROM size block configuration in the PID spec, which is what most APIs are built off of, even if very few people actually use the specific PID spec anymore (see files ending with "ff" before the extension in the drivers/hid directory of the linux kernel to witness the vendor-specific sadness full on).

Not only that, effects are considered to be force mixed across motors via the controller firmware's translation of the effects, because force feedback is about simulating feeling, not just vibrating something, and that's what the spec was designed for (which is why you can't set PID per motor, like what WebVibrate does). Between crazy polymer based tactile surface solutions and what not, we're moving towards motors not being the only actuator for joysticks, and we expect controller firmware to still handle effect translation to whatever new actuators we add to controllers. So just vibrating isn't really a solution.

Assume we want to stay with the way everyone else does it, it's expected you have a struct that basically mirrors the effects laid out in the PID spec (which mirrors DirectInput, because MS and immersion wrote it), which are then sent in a platform specific way to the joystick (handled by the kernel on linux, vendor specific drivers or pid.dll on windows, magic fairy dust on OS X). The only API I know that currently handles full cross platform force feedback is libsdl 2.0. For those curious, the code is at

http://hg.libsdl.org/SDL/file/3242aa26100f/src/haptic

libsdl 2.0 isn't in full release yet, but now it's zlib licensed instead of lgpl now. So, we could feasibly make a struct similar to their unified effects system, and possibly just use the SDL 2 layer? We could also do our own implementation, but we'd basically be reimplementing what they've done.

Comment 19

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Also: This API should have nothing to do with WebVibrate in terms of similarity. They're used for far different things. 

That said, we could certainly implement WebVibrate backends for desktop under an actual FF API, just by having it send effects.

Comment 20

[(not currently active) Ted Mielczarek]

We're not actually aiming to implement "force feedback support". Most modern gamepads simply have one or two motors whose speed you can control. This is fairly similar to WebVibrate.

See XInput, for example:
http://msdn.microsoft.com/en-us/library/windows/desktop/ee417001%28v=vs.85%29.aspx#setting_vibration_effects

I think something like this is what we'd spec, if anything.

Comment 21

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Well damn. I saw XInput (missed the first msdn comment) and a linux implementation and was thinking "Why on earth are we going after X11/xorg style input drivers? Can those even hit FF?". Yay for overloaded terms.

Seeing the XInput API from Microsoft makes this whole thing make way more sense. Also wasn't aware that MS wasn't even officially supporting DI structures for the 360 controller anymore.

I will return to quietly rallying for more expressive haptics in games while also shutting up, returning the bug to rumble, and getting that implemented. :)

Comment 22

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

So, as bug 852944 gets closer to landing, I've been considering taking this bug up again. It'd really be nice to have something in the API for controller rumble. 

I'm still on board for something XInput-ish versus all out force feedback (my racing wheel will have to wait for the web yet another day :c ), but it feels like this should live in the Gamepad API, not the Vibration API. The Vibration API seems solely focused at mobile device vibration, and lacks the ability to select multiple vibration devices, set vibration magnitudes, etc.

Ted, in terms of spec work, should this be added to the gamepad API, or do we want to start an another spec and add it as a partial interface implementation?

Comment 23

[(not currently active) Ted Mielczarek]

I'm sorta on the fence about that. I don't want to have to innovate here if we can reuse, but at the same time the WebVibration spec doesn't seem to cover the use cases of even the XInput-style API where you have >1 vibration motor and they can be set to different magnitudes.

Do the vibration motors in phones that the Vibration API is targeted to cover support changing the magnitude? If not then there's probably no point in trying to unify the APIs. If so maybe it makes sense to try to unify them and turn it into a partial interface and then we could do something like
interface Gamepad {
  Vibration[] rumbleMotors;
}

Comment 24

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

The WebVibration API looks really similar to the Android Vibration API: 

http://developer.android.com/reference/android/os/Vibrator.html

Basically, the only functionality is "Turn the motor all the way on for a specified period of time, then all the way off for a specified period of time". There's no variability to magnitude of the signal.

With the very, very wide range of vibration actuators that can show up in phones (super cheap phones sometimes combine their speaker and their vibrator in the same unit, for instance), not to mention power concerns related to variable pulse width, it doesn't seem like there'd be much interest in adding variable vibration to the WebVibration API. 

This isn't to say you /can't/ do variable vibration on phones. For instance, Immersion has a Android library specifically for force feedback effects on phones (http://immersion.com/products/touchsense-tactile-feedback/3000-series/index.html). But as far as I'm aware, it's a JNI library, so they're doing things under the covers in C/C++, so I'm not sure that really "counts" for this discussion.

I'm still for adding force feedback specifically for Gamepad. We can discuss it with the Vibration spec people, but I have a feeling they're going to feel similarly.

Comment 25

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

I think you'll definitely need a new API for gamepad rumble. If for no other reason than that you need to specify which of the gamepads should rumble.

I also don't think that we should make a gamepad rumble if navigator.vibrate() is called on a desktop which has a gamepad connected. I.e. I think navigator.vibrate() should map to vibrating the device, and the device only. (If the device doesn't have vibration capabilities, we should probably hide the naviagator.vibrate() function, which is what we do now).

What we can reuse though is the syntax. So right now navigator.vibrate() takes an integer, or an array of integers. The same could be done for gamepad.vibrate() (or whatever the function will live).

But yeah, we'd also need to invent syntax for specifying strength. That's definitely a bummer that that can't be reused from the existing API.

Comment 26

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

I'm not sure how much of that syntax we can reuse, actually. navigator.vibrate() takes integers because it's only dealing with milliseconds, and the vibrator is either just on or off during those times. We'll also need to specify magnitudes, which may be best scaled between 0.0 and 1.0, and mentioned earlier in this thread? So we could do something like pairs of (magnitude, time) or something.

Comment 27

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Yeah, signaling one or more pairs of <time, strength> seems like the way to go. Maybe something like:

.vibrate([{ time: 200, strength: 0.5 }, { time: 200, strength: 0.3 }, { time: 200, strength: 0.1 }])

or

.vibrate([[200, 0.5], [200, 0.3], [200, 0.1]])


I don't have an opinion really.

Comment 28

[(not currently active) Ted Mielczarek]

(In reply to Jonas Sicking (:sicking) from comment #25)
> I think you'll definitely need a new API for gamepad rumble. If for no other
> reason than that you need to specify which of the gamepads should rumble.

My intent was never to make navigator.vibrate() work for Gamepads, but to see if we could make .vibrate() into a partial interface on Navigator and then reuse it for Gamepad.vibrate(). It sounds like that's probably not useful, so I'd say invent whatever is useful here and once we have a prototype in hand we can see if there's any way to harmonize them.

Comment 29

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Looks like there's development happening around this in WebVR that will most likely influence this and I'm quite busy on other things, so I'll let someone else take care of it.

Comment 30

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

There's discussion about this happening at https://github.com/mozilla/standards-positions/issues/49, and work happening in VR that may put this closer to happening on the regular gamepad side of things.

Comment 32

[Randell Jesup [:jesup] (needinfo me)]

Attachment: Bug 680289 - Add Gamepad vibrationActuator / playEffect / reset API with stubbed backends r=#webidl!,#dom-core!,cmartin!

Implements the W3C Gamepad haptics API surface end-to-end:

• WebIDL: GamepadHapticEffectType, GamepadHapticsResult, GamepadEffectParameters,
  playEffect()/reset() on GamepadHapticActuator, vibrationActuator on Gamepad
• IPC: PlayHaptic/ResetHaptic/ReplyGamepadHapticsResult messages
• GamepadPlatformService: deferred promise-resolution token registry
• All platform backends stubbed (resolve complete immediately)

Comment 33

[Randell Jesup [:jesup] (needinfo me)]

Attachment: Bug 680289 - Implement Windows XInput gamepad haptics backend for playEffect/reset r=jstutte!

Drive the XInput rumble motors from the deferred-result haptics path added
in the previous commit. Load XInputSetState, report one haptic actuator for
XInput pads, and run dual-rumble effects on the monitor thread with a timer
for startDelay/duration. Handle preemption, reset, and device removal by
resolving the outstanding promise (preempted) and zeroing the motors. Route
the legacy pulse() path through the same backend.

Comment 34

[Randell Jesup [:jesup] (needinfo me)]

Attachment: Bug 680289 - Add mochitest for Gamepad vibrationActuator / playEffect / reset r=cmartin!

Exercises the new dual-rumble API end-to-end through the mock GamepadService:
vibrationActuator is non-null when the pad reports a haptic actuator,
playEffect('dual-rumble') and reset() resolve 'complete' via the no-device
backend path, and an out-of-range magnitude rejects with a TypeError.

Comment 35

[Randell Jesup [:jesup] (needinfo me)]

Attachment: Bug 680289 - Implement Windows XInput gamepad haptics backend for playEffect/reset r=cmartin!

Comment 36

[Randell Jesup [:jesup] (needinfo me)]

Attachment: WIP: Bug 680289 - Rename GetVibrationActuator() to VibrationActuator()

Binding codegen expects the getter for a non-nullable interface-typed
attribute to be named after the attribute directly, without the 'Get'
prefix (that prefix is reserved for nullable attributes, where returning
nullptr represents the null case). vibrationActuator lost its '?' earlier
in this stack but the C++ getter kept the old name, so GamepadBinding.cpp's
generated code failed to find a matching method.