1487509 - Pointer events broken for pen devices

Status: NEW

(Core :: DOM: Events, defect, P2)

Description

[David Jameson]

User Agent: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36

Steps to reproduce:

1. Plug in a Wacom or Huion tablet.
2. Go to a web page that uses Pointer Events, e.g. http://labs.crosspop.in/Croquispop/croquispop.html
3. Draw with the tablet.



Actual results:

No pressure sensitivity. Pressure always comes out as 0.0 or 0.5.


Expected results:

Should register correct pressure.

Comment 1

[David Jameson]

This bug was caused by the bugfix for issue 1345398. Only works when dom.w3c_pointer_events.dispatch_by_pointer_messages is set to true. This should probably be set to true by default now.

Comment 2

[David Jameson]

Firefox version: 61.0.2 (32 bit) on Windows 10.

Comment 3

[laszlo.bialis]

As I don't have access to a graphics tablet, I am assigning a component to this issue in order to involve the development team and get an opinion on this.

Comment 5

[Swadical]

I've been doing some debugging and digging around in the firefox codebase to try and find the cause of this bug.

The clues I've had thus far are: 
1) pen events are somehow becoming touch events on pen contact with digitizer surface
2) enabling `dom.w3c_pointer_events.dispatch_by_pointer_messages` resolves this bug

Looking into the history of why `dispatch_by_pointer_messages` exists, it is probably not going to enabled by default due to potential browser freezes (though I am not sure if this issue has since resolved).

Therefore, assuming that we can't use `dispatch_by_pointer_messages`, I looked into `widget/windows/WinUtils.cpp`, `widget/windows/WinPointerEvents.cpp` and `widget/windows/nsWindow.cpp`. Disclaimer: I am very unfamiliar with the structure of the firefox codebase and how firefox works in general, so I have made many assumptions during my wild bug hunt and may be have interpreted the code incorrectly.

My first hunch was that pointer events from the pen weren't being cached properly in `WinPointerEvents::ConvertAndCachePointerInfo`. However, after further investigation, this function seemed to be working fine, and pen interactions AND pressure, etc. were correctly being saved into the WinPointerEvents class instance.

Then, my second hunch was that retrieving cached data from the class was bugged, so I looked into `WinPointerEvents::GetCachedPointerInfo`. Here's where things get interesting: this function's implementation is fine. It correctly retrieves cached data when asked to do so. When my pen hovers above my digitiser, it is being called, as expected. However, when my pen touches my digitiser, it was apparently not being called. So, I decided to look higher up the stack.

`GetCachedPointerInfo` is called from multiple points inside `nsWindow::ProcessMessage`. However,the result of `GetCachedPointerInfo` is eventually passed into `nsWindow::DispatchMouseEvent`, so I decided to investigate that function by breakpointing it and logging its output. When the pen is hovering, `aInputSource` is correctly 2 (MouseEvent_Binding::MOZ_SOURCE_PEN), but when the pen is in contact with the digitizer, `aInputSource` becomes 5 (MouseEvent_Binding::MOZ_SOURCE_TOUCH) with the first `eMouseDown` event and all subsequent `eMouseMove` events. This is consistent with one of my observations earlier in this report.

Now, I know what you might be thinking: "so what does this have to do with pen pressure not being present in events?". Well, interestingly, `WinPointerEvents::GetCachedPointerInfo` returns a nullptr and short circuits out of the function if it is called by an event whose source is NOT MOZ_SOURCE_PEN. Aha! I monkey-patch the event source check out of GetCachedPointerInfo and check if pen events are fixed aaand..... No dice.

Upon further investigation, and with the knowledge that our pen events are definitely somehow becoming touch events, I notice that `nsWindow::DispatchMouseEvent` has the potential to short circuit and stop execution when dealing with touch events if `TouchEventShouldStartDrag` returns false. And in reality, this does occur.

So, just as a summary, pen events are somehow becoming touch events and that's messing with the logic inside `nsWindow::DispatchMouseEvent`. The next thing I would suspect is `WinUtils::GetMouseInputSource`, but after some cursory searching on the internet, the implementation of this function seems correct.

So, unfortunately, I think we're dealing with an OS bug here. Windows is turning tablet pen inputs into touch inputs somehow. But, surprise, it's not an OS but, it's apparently a feature. Another cursory search about "windows treating pen as touch" reveals many posts from other disgruntled users finding out that the applications they used suddenly treat pen input as mouse input for scrolling across pages (sound familiar? This behaviour also occurs when the pen touches the digitiser surface).

How does Chromium do it then? They don't rely on differentiating between touch events and pen events when dealing with the windows API and obtaining data ( https://github.com/chromium/chromium/blob/329c5eb9c650996548d3442cd7755ba559481ba8/ui/events/win/events_win_utils.cc#L307 ).

So, what now? Perhaps we should reconsider the assumption that touch input events cannot be generated by pen/digitiser input? Perhaps we should just not cache pointer data and set `dispatch_by_pointer_messages` to true?

Comment 6

[Swadical]

I tried another monkeypatch. I made the assumption that if `aPointerInfo` is not null, the event we're dealing with in `nsWindow::DispatchMouseEvent` is probably from a pen: 

widget/windows/nsWindow.cpp:4388 ```

  if (!mWidgetListener) {
    return result;
  }

+  uint32_t pointerId = MOUSE_POINTERID();

+  if (aPointerInfo) {
+    aInputSource = MouseEvent_Binding::MOZ_SOURCE_PEN;
+    pointerId = aPointerInfo->pointerId;
+  }

...


-  uint32_t pointerId = aPointerInfo ? aPointerInfo->pointerId :
-                                      MOUSE_POINTERID();

```

and I changed the APZC logic to take this into account

widget/windows/nsWindow.cpp:4411 ```
-    if (WinUtils::GetIsMouseFromTouch(aEventMessage)) {
+    if (WinUtils::GetIsMouseFromTouch(aEventMessage) && !aInputSource) {
```

I also tweaked the logic in `WinPointerEvents::GetCachedPointerInfo` to take into account that touch events can be generated by pen events.

widget/windows/WinPointerEvents.cpp:161 ```
  if (!sLibraryHandle || !sPointerEventEnabled ||
-      MOUSE_INPUT_SOURCE() != dom::MouseEvent_Binding::MOZ_SOURCE_PEN ||
+      MOUSE_INPUT_SOURCE() == dom::MouseEvent_Binding::MOZ_SOURCE_MOUSE||
      ShouldFirePointerEventByWinPointerMessages()) {
    return nullptr;
  }


```

When testing these changes, `pointermove` events for the pen are correctly being handled as pen events and not touch events, which resolves one of the observed bugs. However, `event.pressure` is still always 0. Interestingly, in a `mousemove` event handler, `event.mozPressure` is correctly set. I'm not sure what is going on here.

Comment 7

[Maxime]

Firefox 70.0 doesn't have the dom.w3c_pointer_events.dispatch_by_pointer_messages pref anymore it seems (was moved to StaticPrefs, here I think https://hg.mozilla.org/mozilla-central/rev/5f00f7fcd77854dab538bb0606b573b3369957d8)
I'm not sure what static prefs are but is there any other trick now to get pressure working on windows? It worked fine before with dom.w3c_pointer_events.dispatch_by_pointer_messages set to true.

Comment 8

[Maxime]

Please disregard the previous comment. It works fine after I've manually reset the preference in about:config.

Sorry for the bother.

Comment 9

[grigoriy.lyukshin]

@Maxime and @Swadical: have you had any success in fixing it? PowerPoint online is also looking to have this bug fixed to improve user experience in FireFox.

Comment 10

[grigoriy.lyukshin]

This is blocking Office online.
Could you please increase priority of this bug?

Comment 11

[Hsin-Yi Tsai (she/her) [:hsinyi]]

(In reply to grigoriy.lyukshin from comment #10)

This is blocking Office online.
Could you please increase priority of this bug?

Hi thanks for letting us know.
There has been some ongoing effort on the stylus/pen meta bug.

Comment 12

[michelfeinstein]

I think my case is related to this.

I have an old Wacom Bamboo tablet. I tried to sign a document with Adobe Sign and then I discovered that when I click with the pen, and hold it, the context menu will pop-up when I release the pen.

I couldn't sign the document at all. The webpage accepts clicks, so I could place a dot in the signing field. But holding the pen to actually draw my signature was impossible, there was no drawing whatsoever, but as soon as I lifted the pen from the tablet the context menu would appear.

And yes, I have disabled the Windows option to "right click on press-and-hold". I even forced it in the regedit.

Comment 13

[Nahor]

I have an x-pen tablet and its also broken. Couple of scenarios I've noticed:

• open two firefox windows that overlap with multiple tabs. Using Windows taskbar and the pen, switch to the second window then try to change tab => tab changes on the first window instead of the second/active window. Using alt-tab to switch window does work properly
• have a plugin with a menu attached to its toolbar icon (e.g. no-script, u-block, ...), use the pen to open the menu, select a menu entry => nothing happen in the plugin and if the menu entry covers a link in the webpage, the link is clicked instead.

Comment 14

[Prashant]

(In reply to Maxime from comment #8)

Please disregard the previous comment. It works fine after I've manually reset the preference in about:config.

Sorry for the bother.
Firefox 70.0 doesn't have the dom.w3c_pointer_events.dispatch_by_pointer_messages pref anymore
Hi Maxim,
I don't see the preference dom.w3c_pointer_events.dispatch_by_pointer_messages in about:config in Firefox 91, by manually reset the preference, do you mean that you created a new preference and set it to TRUE?