Closed Bug 1581126 Opened 2 years ago Closed 10 months ago

Hook up pinch gestures on Linux touchpads for APZ zooming

Categories

(Core :: Panning and Zooming, enhancement, P3)

71 Branch
enhancement

Tracking

()

RESOLVED FIXED
86 Branch
Tracking Status
firefox86 --- fixed

People

(Reporter: sawyerbergeron, Assigned: aemad)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(3 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

  1. Open a website on Firefox on Linux (either X11 or Wayland)
  2. Perform a pinch gesture on the touchpad

Actual results:

no-op

Expected results:

Currently: reflow zoom
Later: smooth APZ zoom as tracked by https://bugzilla.mozilla.org/show_bug.cgi?id=1461360 and described by https://bugzilla.mozilla.org/show_bug.cgi?id=789906

This is a sibling report to https://bugzilla.mozilla.org/show_bug.cgi?id=1568676, as I noticed there was no similar report for Linux. Expected result is similar/identical.

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: Untriaged → Panning and Zooming
Product: Firefox → Core

Thanks for filing.

Depends on: desktop-zoom-xp
Priority: -- → P3
Duplicate of this bug: 1607007

Fairly core functionality now for touchpads, which apparently no browser on linux has as of yet.

(In reply to Sury from comment #5)

Fairly core functionality now for touchpads, which apparently no browser on linux has as of yet.

There might be an opportunity to get ahead of Chrome here, and be the first browser to support this on Linux.

It was mentioned in bug 1607007 that Evince supports this. A good first step here would be investigate how Evince does this, i.e. what APIs it uses to listen for / detect pinch gestures.

(In reply to Botond Ballo [:botond] from comment #6)

(In reply to Sury from comment #5)

Fairly core functionality now for touchpads, which apparently no browser on linux has as of yet.

There might be an opportunity to get ahead of Chrome here, and be the first browser to support this on Linux.

It was mentioned in bug 1607007 that Evince supports this. A good first step here would be investigate how Evince does this, i.e. what APIs it uses to listen for / detect pinch gestures.

Yes, it really would be. Native Gnome apps have support, including Gnome-Web (ie. epiphany). However epiphany is far too choppy to be useful for web browsing.

There might be a clue here:

https://blogs.gnome.org/alexm/2019/09/13/gnome-and-gestures-part-1-webkitgtk/

The OP commented:

A large part of the reason is that all major browsers (Firefox, Chromium/Chrome) still run on X11, and there’s no way to have touchpad gestures (other than scroll-as-swipe) on X11.

For example, Chrome does support gestures on touchscreen, but not on touchpad precisely because of that.

I'm happy to try picking this up now that I'm a bit more familiar with APZ/APZC, a little less sure how to tie into the new granular zoom facility but I'm guessing some prior work should be visible in HandleGestureEvent for touchscreen pinch handling?

AFAIK it should be possible to get working on X, I could swear Xinput2 has support for smooth touchpad pinch gestures.

Evince does also support it on my end so that will also be helpful.

(In reply to Sawyer Bergeron from comment #8)

I'm happy to try picking this up now that I'm a bit more familiar with APZ/APZC,

Cool, thanks Sawyer!

a little less sure how to tie into the new granular zoom facility but I'm guessing some prior work should be visible in HandleGestureEvent for touchscreen pinch handling?

Here is the corresponding code that makes this work on macOS. Basically, it listens for OS events of the appropriate type, converts them into our PinchGestureInput events, and dispatches the converted events.

So, if we can figure out what to listen for in our GTK widget backend and convert that into PinchGestureInput events, that should basically just work. For an existing example of listening to a native GTK event, converting it into a Firefox event, and dispatching the event, see OnScrollEvent().

Looks like Evince uses GtkGestureZoom?

One issue I'm noticing is that although wayland itself supports pinch as a segmented [start][main gesture][gesture end] as detailed in https://github.com/wayland-project/wayland-protocols/blob/master/unstable/pointer-gestures/pointer-gestures-unstable-v1.xml, GTK itself only provides for deltas emitted during the middle of the pinch, and has no explicit way of saying a gesture will begin or the touch has ended. Not sure if that's going to end up being a problem later (especially if some sort of "zoom fling" or smooth animation out is planned)

(In reply to Sawyer Bergeron from comment #11)

Not sure if that's going to end up being a problem later (especially if some sort of "zoom fling" or smooth animation out is planned)

We don't currently do any sort of zoom fling or smooth animation on touchscreens (nor macOS touchpads), we just scale synchronously with the finger movement.

Actually correction to my earlier statement, it does broadcast the current phase of the gesture in the event: https://developer.gnome.org/gdk3/stable/gdk3-Event-Structures.html#GdkEventTouchpadPinch

I was taking a look at GtkGestureZoom and GdkEventTouchpadPinch and they require GDK 3.14 and 3.18 respectively. Using GtkGestureZoom even with a GTK_CHECK_VERSION(3, 14, 0) guard results in a compilation error:

warning: 'gtk_gesture_zoom_new' is deprecated: Not available before 3.14 [-Wdeprecated-declarations]

This is because GDK_MAX_ALLOWED_VERSION is set to 3.4 currently. Unfortunately I've never contributed to Firefox before so I don't know if there is a specific procedure to up this to 3.14.

(In reply to Benjamin Cheng from comment #14)

Because Firefox' primary distribution method is a binary which has to run with the oldest GTK supported but wants to use the newest GTK features available, you cannot use compile-time feature detection. The only use of GTK_CHECK_VERSION in the code is to define macros and types that were introduced in later GTK versions.

You need to use dlsym to discover the functions at runtime. For an example, check out sGdkWindowMoveToRect in widget/gtk/nsWindow.cpp.

Depends on: 1640186
Summary: Hook up pinch gestures on Linux touchpads in preparation for APZ zooming → Hook up pinch gestures on Linux touchpads for APZ zooming

(In reply to Sawyer Bergeron from comment #8)

AFAIK it should be possible to get working on X, I could swear Xinput2 has support for smooth touchpad pinch gestures.

Evince does also support it on my end so that will also be helpful.

Hey Sawyer, just wanted to follow up on this: have you actually experienced touchpad pinch-zooming working under X11 in any application (such as Evince)?

I tried just now and it works for me in Evince under Wayland, but not under X11.

I just retested this under gnome on x11. Touchpad pinch zooming does not appear to work, but touchscreen zooming did which was probably part of my confusion earlier.

Testing gnome web and evince on x11 touchpad pinch zoom does not work, but works for both on wayland.

Thanks. I have meanwhile looked at the GTK source code as well, and confirmed that it only supports touchpad pinch gestures in its Wayland and Quartz (macOS) backends, not in its X11 backend.

So, at least in the short term, this will be fixed for Wayland only.

Could you please give us an update after you have released a fix for Wayland if it's not too troublesome for you? Currently, using Firefox on X11 with the Multi-Touch Zoom add-on, holding control + scrolling the webpage already does a great job at zooming the webpage, and it would be great to integrate with pinch-to-zoom! Also, after some web searching, it does seem like X11 has bad support for multi-touch events. Really looking forward to this feature! Thanks in advance!

Yep, we'll post updates to this bug as the Wayland implementation is being developed.

It has come to my attention that support for multi-touch gestures in X11 and GTK's X11 backend is being worked on: https://bill.harding.blog/2020/10/06/q3-linux-touchpad-like-macbook-update-multitouch-gesture-test-packages-are-ready/

Firefox will use GTK APIs, so while initially pinch gestures will only work with Wayland, once the work referenced above lands in X11 and GTK and users upgrade to versions of X11 and GTK that include this support, pinch gestures should start working in X11 as well.

I've done some initial work on this and am able to get scale events successfully from gtk, I still need to work on hooking things up with APZ

I'm getting somewhat lost trying to work through how to get events into APZ now, the directmanip prior work is helping somewhat but I think I may need a few pointers. Should I push what I have to phabricator or discuss here?

You can upload what you have (you don't need to use phabricator yet if it's an early patch), and then ask questions about it here. There is also the #apz channel on https://chat.mozilla.org/

I think nsBaseWidget::DispatchInputEvent should be used for getting the events into APZ.

Just as a FYI for people following along, this bug is the focus of an Outreachy project that Mozilla is sponsoring. So the intern that gets selected will be working on completing this bug. Although implementation advice is welcome, for now we'd like to leave enough scope on this bug for it to remain a worthwhile intern project. Thanks!

(In reply to Botond Ballo [:botond] from comment #6)

(In reply to Sury from comment #5)

Fairly core functionality now for touchpads, which apparently no browser on linux has as of yet.

There might be an opportunity to get ahead of Chrome here, and be the first browser to support this on Linux.

Just wanted to follow up on this and mention that I have confirmed that:

  • Newer versions of Chrome (87 and later) can run in native Wayland mode (using the command line flags --enable-features=UseOzonePlatform --ozone-platform=wayland).
  • However, even in native Wayland mode, touchpad pinch-zooming does not work in Chrome. (I've looked through their experimental feature flags as well and didn't find one that would enable this feature.)

So, it does seem likely that Firefox will be the first major browser to support this feature on Linux.

Assignee: nobody → eng.alaaemad.ae

I built Firefox in debug mode with the patch applied. I get a crash on startup, with the attached stack trace.

Blocks: 1685395
Blocks: 1640186
No longer depends on: 1640186
Pushed by bballo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/09d418b19d52
Hook up pinch gestures on Linux touchpads for APZ zooming r=botond,stransky
Status: UNCONFIRMED → RESOLVED
Closed: 10 months ago
Resolution: --- → FIXED
Target Milestone: --- → 86 Branch

This doesn't work on wayland, any additional flags that we need to enable?

Yes, turn on apz.gtk.touchpad_pinch.enabled.

(In reply to Dan Elkouby from comment #34)

Yes, turn on apz.gtk.touchpad_pinch.enabled.

Thank you!

(In reply to Dan Elkouby from comment #34)

Yes, turn on apz.gtk.touchpad_pinch.enabled.

I am on latest Nightly and have this enabled. Still doesn't work.

(In reply to ugzuzg from comment #36)

I am on latest Nightly and have this enabled. Still doesn't work.

On my laptop a vertical pinch can't initiate the gesture, only horizontal pinching works. I think it's a gtk bug, or something lower level.

(In reply to Dan Elkouby from comment #37)

(In reply to ugzuzg from comment #36)

I am on latest Nightly and have this enabled. Still doesn't work.

On my laptop a vertical pinch can't initiate the gesture, only horizontal pinching works. I think it's a gtk bug, or something lower level.

On my laptop it works at any angle. I'm on GNOME over wayland. So I believe it's not a gtk bug or it's already fixed in the recent gtk version.

On my laptop after enabling apz.gtk.touchpad_pinch.enabled works beautifully. With any gesture direction.

I'm running up to date Arch, and can repro in multiple gtk programs (eog, and the demos), so this isn't a Firefox bug at least. It's just hard to know because there isn't really anything else that supports gestures. I'll investigate this and report it wherever it makes sense.

(In reply to Dan Elkouby from comment #37)

On my laptop a vertical pinch can't initiate the gesture, only horizontal pinching works. I think it's a gtk bug, or something lower level.

This is libinput-determined behavior. It's considered not-a-bug at this time since many users use a gesture with the bottom finger held still to do 2 finger scrolling. Gesture switching (see https://gitlab.freedesktop.org/libinput/libinput/-/issues/550) may improve this but it's still impossible to universally disambiguate what a user wants to do with the given gesture.

Thanks for shining some light on the problem. Unfortunately that makes pinch gestures nearly unusable for me. I'll see what I can do.

I am also running an up to date Arch, in evince and epiphany pinch to zoom works perfectly. What's the best way for me to help debug this?

(In reply to ugzuzg from comment #43)

I am also running an up to date Arch, in evince and epiphany pinch to zoom works perfectly. What's the best way for me to help debug this?

If you go to about:support and scroll down to the heading labeled "window protocol" what do you see there?

If you go to about:support and scroll down to the heading labeled "window protocol" what do you see there?
It was xwayland, thanks for the help!
Switched to wayland, now it works.

(In reply to Sawyer Bergeron from comment #41)

This is libinput-determined behavior. It's considered not-a-bug at this time since many users use a gesture with the bottom finger held still to do 2 finger scrolling. Gesture switching (see https://gitlab.freedesktop.org/libinput/libinput/-/issues/550) may improve this but it's still impossible to universally disambiguate what a user wants to do with the given gesture.

Seems like this may actually be more of a problem with my touchpad:

$ libinput measure touchpad-pressure
Using SYNA2B2C:01 06CB:7F27 Touchpad: /dev/input/event6

This device does not have the capabilities for pressure-based touch detection.
Details: Device does not have ABS_PRESSURE or ABS_MT_PRESSURE

$ libinput measure touch-size 
Using SYNA2B2C:01 06CB:7F27 Touchpad: /dev/input/event6

This device does not have the capabilities for size-based touch detection.
Details: Device does not have ABS_MT_TOUCH_MAJOR

I can only assume that it's doing its own thumb detection instead (or libinput gets the hint in another way). Further testing has shown that it has nothing to do with the orientation of the gesture, but using my index finger and my thumb break it. Any other combination of fingers works fine.

Blocks: 1688717

(In reply to Dan Elkouby from comment #34)

Yes, turn on apz.gtk.touchpad_pinch.enabled.

Right, yes, I forgot to announce that in this bug we landed the feature behind a pref.

Bug 1688717 tracks enabling the pref by default.

For those following this bug: the touchpad zoom feature is now enabled by default and will be released as part of Firefox 88.

I would like to extend a thank you to Alaa Emad, our Outreachy intern who has worked on this feature!

Depends on: 1702855
You need to log in before you can comment on or make changes to this bug.