1501744 - Handle pens with PointerEvents on the Gtk backend

Status: RESOLVED FIXED

(Core :: Widget: Gtk, enhancement, P2)

Description

[tpxp]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0

Steps to reproduce:

1. Get a pen/tablet recognized by Linux. I'm using a Bamboo Ink on a Dell XPS 13 2-in-1 (9365), but any Wacom tablet should work.
2. Test it in the gnome control center to make sure everything is recognized (including pressure and eraser)
3. Try using PointerEvents to develop an amazing app using a stylus - demo https://patrickhlauke.github.io/touch/tracker/tracker-force-pressure.html


Actual results:

Stylus input is recognized as a mouse


Expected results:

Stylus input should be recognized as a pen as per the PointerEvents specification : https://www.w3.org/TR/pointerevents/#pointerevent-interface

Comment 1

[tpxp]

Attachment: patchv0.diff

This is a bug I'm willing to work on to improve Firefox! Here is a first patch setting the input source to its correct values. I still have to handle button changes, and maybe make sure the button value remains -1 while moving, nevertheless I would appreciate some feedback !

Comment 2

[tpxp]

Attachment: patch_v0.diff

The previous patch I submitted was incorrect, my bad I messed up files. Here is a working one. Still, it looks like PointerEvents are not fired correctly when the event's device type is pen, I will investigate this.

Firefox is running under Xwayland on my system (actually that's the default behaviour) and it looks like, by default, Xwayland passes standard pen interaction (as a pen, without any behaviour modifiers like eraser) as a mouse. I will also investigate this (maybe I should use the Xevent data as well ?).

Comment 3

[tpxp]

Attachment: patch_v1.diff

After some investigations, I found out that pen data was passed as a touchscreen interaction. However, some more axes were (incorrectly) defined. This is probably a bug with GTK, but before a fix lands in most distributions, I believe a heuristic is needed. I added on heuristic that sets the input device as a pen if :
- We're using x11
- The event's source is a touchscreen
- X tilt axis and Y tilt axis are defined (please note that the values are incorrect though, so we don't set them).

The pointerevent that come from pen interactions are now correctly recognised by Firefox on my computer (running GNOME in Wayland on an up-to-date Archlinux) and the pointerevent data is correct ! Now, I'm going to focus on the pen's buttons.

Comment 4

[Stefan [:StefanG_QA]]

I have tested this report on Surface Pro with Windows 10 x64 using the latest Nightly and Fx release build. I did not manage to reproduce it. All events are properly registered. I will install a Linux distribution on the Surface and will test it. I`m assigning component in order to get more attention to the report. Please change if this is not the right component.

Comment 5

[tpxp]

Hi Stefan, and thank you for your response!

Indeed, the event is correctly reported on Windows since the Windows widget backend has the code to set the input type to pen when WIndows reports it (you can look for "MOZ_SOURCE_PEN" in the source code). This is platform-dependent code, since calling the host OS API is required to know more about the pointer. The GTK backend does not yet have the code to set the internal pointer event's device type when GTK reports a pen (it's interpreted as a mouse). This bug aims at adding this code so that pens reported by GTK are interpreted by Firefox as pens.

I admit the title of the bug might be misleading, maybe something like "Map Pen events from the GTK backend to their internal equivalents" would be more appropriate, since PointerEvents are only a consequence of it.

Comment 6

[Firefox Bug Husbandry Bot]

https://bugzilla.mozilla.org/show_bug.cgi?id=1552613

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The bug has a release status flag that shows some version of Firefox is affected, thus it will be considered confirmed.

Comment 8

[Martin Stránský [:stransky] (ni? me)]

Hello tpxp,

sotty for the late reply. Would be happy to review and take the patch. Do you mind to submit it via Phabricator and ask me for review?
https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html

Thanks.

Comment 9

[Martin Stránský [:stransky] (ni? me)]

Attachment: mzbz-1501744-v2.patch

Rebased patch_v1.diff patch to latest trunk. Feel free to use it as a template for the phabricator one.
Thanks.

Comment 10

[Martin Stránský [:stransky] (ni? me)]

Hello tpxp, I did that behalf of you.

Comment 11

[Martin Stránský [:stransky] (ni? me)]

Attachment: Bug 1501744 [Linux] Fill PointerEvents on the Gtk backend r?emilio

Patch author is tpxp <[e-mail]>

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/292f128d3cc6
[Linux] Fill PointerEvents on the Gtk backend r=emilio

Comment 14

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/292f128d3cc6