1282089 - Touch simulation doesn't support mobile gestures

Status: RESOLVED FIXED

(DevTools :: Responsive Design Mode, enhancement, P3)

Description

[Benoit Chabod [:bigben]]

There are a lot of gestures commonly used on mobile devices to favor more direct forms of interactions. They've become so intuitive that you probably use them without thinking about it:
- Double tap to zoom
- Drag/flick to scroll
- Press and hold to display options (a bit like right click)

Of course, some of them are hard to reproduce with a mouse. I don't really have a scenario with expected/actual results, but I believe we should support this in our RDM touch simulation mode.

Chrome does, and it would really enhance the "mobile feeling" of the tool.

Comment 1

[Benoit Chabod [:bigben]]

I've looked up the current code for touch simulation, and almost everything is done in the simulator-content script [1]. This JS file is loaded in the viewport by the RDM code using the message manager [2].
The script basically captures every mouse event, and translates them into touch events when the simulator is enabled.

We could detect specific events like double click = double tap = force the viewport to zoom, but maybe this is already done somewhere in the codebase, for mobile devices, and we could just trick the platform into thinking this viewport is really on a mobile device.

Perhaps we should use other functions like InjectTouchEvent to force that behaviour? [3]
I'm not sure what the right approach here.

[1]: https://dxr.mozilla.org/mozilla-central/source/devtools/shared/touch/simulator-content.js
[2]: https://dxr.mozilla.org/mozilla-central/source/devtools/shared/touch/simulator.js#28

Comment 2

[Benoit Chabod [:bigben]]

Oops, forgot the link related to my previous comment.
[3]: https://dxr.mozilla.org/mozilla-central/source/dom/ipc/TabParent.cpp#2826

Comment 3

[Benoit Chabod [:bigben]]

Attachment: Part 1: Fix platform InjectTouchEvent function

Comment 4

[Benoit Chabod [:bigben]]

Attachment: Part 2: Allow per-document control of the meta-viewport preference

Comment 5

[Benoit Chabod [:bigben]]

Attachment: Part 3: Adapt RDM to fully support touch simulation

Comment 6

[Benoit Chabod [:bigben]]

Okay, here's my latest progress on bringing real touch simulation in RDM.
As mentioned in previous comments, the goal here is to use the platform just like on a mobile device, instead of trying to imitate the mobile behaviour with a frame script.

There are 3 parts in this patch:

- Part 1: Vivien Nicolas fixed injectTouchEvent for me, it's an old API used for B2G and provides an easy way to test "real" touch events. However, it's deprecated and we should use SendNativeTouchPoint as soon as it's available on all platforms (see bug 1288187, I'm discussing this with kats). For obscure thread reasons, injectTouchEvent tends to crash a lot, so I had to add a 500ms delay between the mouse events and the resulting touch actions (hopefully this will be removed when we use SendNativeTouchPoint).

- Part 2 adds an attribute in the docshell to manage the meta-viewport preference tab by tab, just like dom.w3c_touch_events.enabled in Bug 970346. This was needed because setting the preference for the whole browser breaks most of the UI.

- Part 3 mostly mods the touch simulator in the RDM to use this new method.
There is an invisible div above the iframe that I called the "touch proxy", and it captures every mouse event to call injectTouchEvent when needed. I set/remove the meta-viewport docshell attribute, and a few other preferences. A lot of useless code in the simulator script is also removed in this part.

This is still WIP, but it already does some cool things: you can double tap to zoom, drag to scroll, long press to open contextual menu, etc. The touch proxy thing also resolves Bug 1282084 (no more hover events), and the meta-viewport preference fixes a lot of websites that weren't rendered properly in RDM before.

We can also add fluffing easily with this approach (accept clicks even when there aren't exactly on the target), using ui.touch.radius preferences, and this would justify Bug 1271728.

All suggestions/advices are welcome!

Comment 7

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Since Benoit's internship has ended, I am going to assume he may not be working on this anymore.  If that's incorrect, feel free to pick it up again!

Comment 11

[Sebastian Zartner [:sebo]]

According to comment 6 it seems Benoit was pretty close to finish the patch.
Based on the duplicates, people on CC and questions in forums and other pages, I assume this is a feature many people are missing. Would someone of the team be willing to finish this up?

Sebastian

Comment 12

[lolipopplus]

giving that more tablets devices are being used, this feature is getting more desirable.

Comment 13

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Please avoid using needinfo (and Bugzilla generally) just to state an opinion on a feature's importance.  This is one feature of thousands that the DevTools team prioritizes.  We regularly revisit our feature backlog to decide what to work on next.

We do still want feedback though, and Discourse[1] is a good place for that, along with Twitter, IRC, Slack, etc.

[1]: https://discourse.mozilla.org/c/devtools

Comment 14

[Micah [:mtigley] (she/her)]

Mobile touch simulation will be an important feature for RDM this year. So I'll be picking this up to begin investigation on what needs to be done to get Benoit's work landed.

Comment 15

[Brad Werth [:bradwerth]]

Comment on attachment 8773304 [details] [diff] [review]
Part 1: Fix platform InjectTouchEvent function

This is in the tree in nsDOMWindowUtils::SendTouchEventCommon. First introduced in Bug 603008, pre-dating this patch. That seems to indicate that the existing pathway is adequate.

Comment 16

[Brad Werth [:bradwerth]]

Comment on attachment 8773305 [details] [diff] [review]
Part 2: Allow per-document control of the meta-viewport preference

Meta viewport is enabled if and only if touch simulation is enabled, since Bug 1290420 landed. This patch is no longer needed to force meta viewport, since any touch gestures will, by definition, have touch simulation enabled.

Comment 17

[Micah [:mtigley] (she/her)]

In order to resolve this, this bug will depend on already existing smaller bugs and we can solve it from those. For example, Part 3 introduces a possible fix to hover states, which we can fix from Bug 1282084.

Comment 18

[Micah [:mtigley] (she/her)]

Currently, the touch-simulator is using out of date methods to create and dispatch touch events. Now that SendNativeTouchPoint is available on all platforms, we can now use the platform API to simulate scrolling/swiping as they would appear on a mobile device. To see where these touch points are being created in the RDM viewport, I've been using https://patrickhlauke.github.io/touch/tracker/multi-touch-tracker-pointer-hud.html.

Using SendNativeTouchPoint, I've been observing a noticeable x and y offset of the created touch point from where the mouse has been clicked. So I'll be continuing more investigation on why this is.

Comment 19

[Guillaume]

If simulated touch events had event.pointerType === 'touch' instead of "mouse", this would allow developing and testing touch scroll/gestures libraries using Firefox. Obviously this will be the case when this problem is resolved... or is it worth opening a dedicated issue?

Comment 20

[Micah [:mtigley] (she/her)]

Now that support for mobile scrolling gestures in RDM has landed, RDM now has basic support for mobile gestures. The following gestures are supported (as of Firefox Nightly 79):

• Double tap to zoom (Bug 1621781)
• Swipe/scroll (Bug 1621781)
• Long press to open the contextual menu
• Pinch to zoom (on trackpads) thanks to work in the Bug 1620055 meta.

I believe we can close this issue now. If there are additional issues relating to simulating mobile touch gestures in RDM, we can continue to file them separately against the Bug 1401304 (rdm-touch) meta.