1136675 - Support for hardware cursors on WebGL contexts

Status: NEW

(Core :: General, defect)

Description

[Mike de Boer [:mikedeboer]]

Note up front: hardware cursors might be interesting to have in other areas of the web platform, but since the feature is mostly interesting in computationally expensive applications - read: games - scoping it down to WebGL seemed like a good idea to me.

Hardware cursors, aka 'native cursors', are mouse cursors that are rendered directly by the OS, thus bypassing the software render stack and hardware abstraction layers that exist in Gecko.

It allows for more, if not _full_, control over the looks and behavior of the cursor while it's hovering the canvas element. Because the mouse pointer rendering is done by the OS, it will keep being responsive to mouse movements while the WebGL/ game processing might halt the browser.

Reference implementations can be found in Flash and Unity:

[Flash] http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/ui/Mouse.html#registerCursor%28%29
[Flash] http://www.adobe.com/devnet/flashplayer/articles/native-mouse-cursors.html
[Unity] http://blogs.unity3d.com/2012/10/22/cursor-api/

There's a MS Technet article covering this subject of the cursor handling on the OS level: https://technet.microsoft.com/en-us/magazine/2009.02.windowsconfidential.aspx?pr=blog

This was also discussed on the dev-platform mailing list, starting here: https://lists.mozilla.org/pipermail/dev-platform/2015-January/008277.html

Since hardware cursors work differently on the various platforms we support, this will most likely morph into a meta-bug to be broken down in platform-specific implementation detail bugs.

It's worth noting that this feature would help the Shumway project in reaching feature-parity with the Flash Player 10 runtime.

Comment 1

[Till Schneidereit [:till]]

(In reply to Mike de Boer [:mikedeboer] from comment #0)
> It's worth noting that this feature would help the Shumway project in
> reaching feature-parity with the Flash Player 10 runtime.

That'd only be true if the scope is broadened to include 2D canvas elements. Is there any technical reason not to do that?

Comment 2

[Mike de Boer [:mikedeboer]]

(In reply to Till Schneidereit [:till] from comment #1)
> That'd only be true if the scope is broadened to include 2D canvas elements.
> Is there any technical reason not to do that?

I'm entirely going on assumptions here, but my gut tells me it's not an issue to expand the scope to 2D canvas too. I'll leave the floor to our experts!

Comment 3

[Thibaut Despoulain]

Thank you for bringing the discussion of hardware cursors here, we'll be following this bug closely!
If you ever need a real-life example of such usage or potentially a testbed game, feel free to send me a mail and we'll give you access to Project Atlas' closed alpha.

Comment 4

[Kelsey Gilbert [:jgilbert]]

This is a cool idea, but I believe it's a general web API request. I don't think it's dependent on WebGL, though WebGL (and to a lesser extent, canvas2d) apps are the ones most affected by it.

Nick, any idea where this should go?

Comment 5

[(not reading bugmail) Nick Desaulniers [:\n]]

Correct me if I'm wrong, but can't this already be done at the application level after a successful pointer lock?  You still get scroll events, you would just have to draw a cursor and bound it by the viewport, as opposed to rotating a virtual camera.

Comment 6

[Thibaut Despoulain]

@Nick > What you're suggesting is called a software cursor, and introduces a lot of latency because the rendering goes through the same path as the app's. What is suggested is to clip the pointer to an area while keeping the OS-rendered cursor which uses it's own fastpath.

Comment 7

[(not reading bugmail) Nick Desaulniers [:\n]]

I'm trying to fully understand the use case.

Are you trying to prevent the pointer from leaving a bounding box when a canvas element is not full screen?

Are you trying to change the cursor when hovered over a canvas element that is not full screen?

If you want to reduce latency, why not have rendering logic in a Web Worker, and debounce mouseover events?  Every modern browser that supports WebGL supports Web Workers.

What are the measured latencies of software cursors?

Is this use case solely to support not bogging down the software cursor when frame rate drops?

Do your mouse over callbacks even show up in a snapshot from a sampling profiler?  It is possible to have very fast mouseover events by simply storing x and y coords, then doing actual calculations in your requestAnimationFrame loop.

Thank you for taking the time to answer these questions; it will help flesh out the use case for a possible spec.

Comment 8

[Thibaut Despoulain]

(In reply to Nick Desaulniers [:\n] from comment #7)
> I'm trying to fully understand the use case.
> 
> Are you trying to prevent the pointer from leaving a bounding box when a
> canvas element is not full screen?

Our use case is to prevent the cursor from leaving a bounding box wether in fullscreen or not. (to introduce a "push-edge to scroll the map" behavior similar to Starcraft or League of Legends)
It's also useful to prevent the cursor from exiting the screen when multiple monitors are connected and the app goes fullscreen.

> 
> Are you trying to change the cursor when hovered over a canvas element that
> is not full screen?

If I get this question correctly, the answer is no.

> 
> If you want to reduce latency, why not have rendering logic in a Web Worker,
> and debounce mouseover events?  Every modern browser that supports WebGL
> supports Web Workers.

It is not currently possible to render WebGL content from a worker. And even if that was doable, we'd still be far from the latency of an OS-drawn cursor. 

> 
> What are the measured latencies of software cursors?

We're talking about at least 2 frames of latency (32ms+), plus, if your app drops frames, your cursor will also drop frames, which means even more cursor latency/stutters.
Hardware cursors have sub-frame latency (<16ms).
It might not seem like much for a regular app, but for games, it's like day and night. (https://www.google.com/search?q=software+versus+hardware+cursor+latency)

> 
> Is this use case solely to support not bogging down the software cursor when
> frame rate drops?

Like I said above, that is one part of it. The rest is just general latency reduction.

> 
> Do your mouse over callbacks even show up in a snapshot from a sampling
> profiler?  It is possible to have very fast mouseover events by simply
> storing x and y coords, then doing actual calculations in your
> requestAnimationFrame loop.

We're building a WebGL game, and are not concerned by that. All our input logic is way below 16ms. 

> 
> Thank you for taking the time to answer these questions; it will help flesh
> out the use case for a possible spec.

No problem, thanks for looking into it!

Comment 9

[Mike de Boer [:mikedeboer]]

Nick, is there a possibility to meet up in Whistler to talk about this? There is a perspective where this could be useful for Firefox Hello as well, not just WebGL games.

Thanks!

Comment 10

[(not reading bugmail) Nick Desaulniers [:\n]]

Sure, but I wouldn't be involved with implementing any of this.  Jeff Gilbert or Dan Glastonbury are probably who you want to talk to.

Comment 11

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

We already do use hardware cursors to implement the CSS cursor property.  So if Shumway isn't already using the CSS cursor property (and passing in a url(), perhaps a data/object URL for the image), it should be.

What seems to be wanted here is an extension to requestPointerLock that a) doesn't hide the cursor; and b) provides absolute coordinates [much like screenX/screenY], instead of movementX/movementY.

So perhaps an options dict...

   element.requestPointerLock({ hideCursor: false, absoluteCoordinates: true }) ?

Where the defaults are "true" and "false", respectively?

Comment 12

[Anne (:annevk)]

We should not use true as default argument value. From IDL: "It is strongly suggested not to use default value of true for boolean-typed arguments, as this can be confusing for authors who might otherwise expect the default conversion of undefined to be used (i.e., false)."

Comment 13

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Ah, good point.  Though it looks like movementX/movementY are available on all mousemove events, not just those delivered via pointer lock.. in which case we don't need absoluteCoordinates or anything.  Just a { showCursor: false } or true options dict parameter.

I think this bug then just turns into "add showCursor as parameter to requestPointerLock" ?