777972 - [responsive mode] translate click events to touch events

Status: RESOLVED FIXED

(DevTools :: Responsive Design Mode, defect, P2)

Comment 1

[Cameron Paul]

I'm interested in tackling this one. I've got a working prototype that is generating touchstart events on mousedown, but I have some questions before putting together a complete patch.

Do you just want touchstart and touchend events to fire along with mousedown and mouseup, or should it cancel the mouse events?

Do you want to simulate touchmove events?

If touchmove is handled, do you want to simulate page scrolling as well?

Do you want click events to happen as normal, or should I try to simulate the 300ms delay a device would have and cancel if the user drags?

Should this just be always on in responsive mode, or would something like a checkbox next to the rotate button be good?

Are you interested in simulating gesture/multitouch events for people with multitouch touchpads? (I have no idea how to implement that one)

Sorry for so many questions, just want to make sure all the bases are covered.

Comment 2

[Girish Sharma [:Optimizer]]

(In reply to cpaul37 from comment #1)
Some input below :

> I'm interested in tackling this one. I've got a working prototype that is
> generating touchstart events on mousedown, but I have some questions before

This is awesome!
 
> Do you just want touchstart and touchend events to fire along with mousedown
> and mouseup, or should it cancel the mouse events?

We can have an option somewhere (either on the toolbar of responsive mode or options panel) to choose one of them (or both). The options could be like
[ ] Emulate touch events
[ ] Don't emit mouse events while emulating touch events

> Do you want to simulate touchmove events?

If possible, yes.
 
> If touchmove is handled, do you want to simulate page scrolling as well?

Maybe yes, maybe not. Not sure how difficult it is to get into the first patch. If this is not difficult, then yes, otherwise, followups .
 
> Do you want click events to happen as normal, or should I try to simulate
> the 300ms delay a device would have and cancel if the user drags?

What is the behavior on mobile devices ? If it is the latter, then do that.

> Should this just be always on in responsive mode, or would something like a
> checkbox next to the rotate button be good?

Like I said above, some sort of checkbox should be there.

> Are you interested in simulating gesture/multitouch events for people with
> multitouch touchpads? (I have no idea how to implement that one)

We can take this up in follow ups.

Comment 3

[Cameron Paul]

Attachment: Simulated touch events in responsive mode

Here is a partial patch implementing some really basic touch event emulation. So far you can enable touch events which will fire alongside mousedown, mousemove if you're dragging, and mouseup. You can also suppress mouse events preventing mousedown, mousemove, and mouseup while emulating touch.

On an actual device, click events are simulated after 300ms if the phone hasn't detected some other type of action happening such as a scroll. I haven't implemented this delay/cancel functionality since it requires having scrolling/gesture detection to be useful. Click events do still fire after any mouseup/touchend events, so the device's simulated click behavior is partially covered.

I didn't implement scrolling on drag because getting it to behave like actual devices seems complicated, and if it doesn't behave like a device then it's not very useful. It also interacts with the 300ms delay described above, as well as multitouch/gesture events, so I think those could all be implemented together as part of a follow up.

I also haven't finished populating the touch events with data in this patch. This is a trivial but tedious task so I skipped it in favor of getting what I have out for review.

The one major limitation I have right now is I can't generate an actual TouchEvent. Currently I create a UIEvent and set the type to touchstart/touchmove/touchend. Functionally this is exactly the same as an actual TouchEvent, it just shows up as a UIEvent if you log it to the console.

Right now if you enable touch event emulation and don't suppress mouse events the behavior appears to be the same as the touch event emulation in Chrome, with the exception of the event data not being filled and the type being UIEvent.

If I'm on the right track here I can probably have a complete patch ready sometime tomorrow.

Comment 4

[Paul Rouget [:paul]]

Wow, happy to see someone looking at this!

@cpaul, some context: we don't want to perfectly emulate the touch experience. For example, we don't need multitouch recording, or kinetic scrolling. But we want basic events to be well supported.

We have been doing that in the past for desktop versions of Fennec also for the Gaia simulator.

If you can re-use existing code, that would be much better.

See here: https://github.com/mozilla-b2g/gaia/blob/master/tools/extensions/desktop-helper/content/touch-events.js

Ask me for reviews.

Thanks!

Comment 5

[Paul Rouget [:paul]]

Comment on attachment 769466 [details] [diff] [review]
Simulated touch events in responsive mode

Cancelling the review.

See if you can use Gaia code (you will have to copy this file in devtools/shared). Also - we will need some mochitests.

Comment 6

[Paul Rouget [:paul]]

@cpaul, should I assign this bug to you?

Comment 7

[Cameron Paul]

(In reply to Paul Rouget [:paul] from comment #6)
> @cpaul, should I assign this bug to you?

Yes, please do.

Comment 8

[Paul Rouget [:paul]]

@cpaul, let me know if you need any help here.

Comment 9

[Paul Rouget [:paul]]

@cpaul, still working on this?

Comment 10

[Cameron Paul]

I'm still working on it, I've just been really busy the last few weeks. If anyone else wants to take this that's cool, otherwise I'll have it done soon.

Comment 11

[Paul Rouget [:paul]]

Attachment: patch1.diff

Comment 12

[Paul Rouget [:paul]]

Comment on attachment 797251 [details] [diff] [review]
patch1.diff

Boris, in the developer tools, we want to simulate touch events. We basically translate mouse events to touch events. So we need to use `document.createEvent("touchevent")`, which is not possible if the pref `w3c_touch_events` is not set to `1`. We programmatically  switch the pref in the developer tools when the "simulate touch events" feature is turned on, but because this pref is cached in nsDOMTouchEvent.cpp, it doesn't work. We could use Preferences::RegisterCallback and re-request the pref value in nsDOMTouchEvent.cpp, but I thought it might be better to just not cache the value.

Let me know what you think.

Comment 13

[Paul Rouget [:paul]]

Attachment: patch2.diff

Comment 14

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 797251 [details] [diff] [review]
patch1.diff

Olli, does this seem reasonable to you?  Note that this function is called during HTMLElement prototype setup and a few other places... but I think this should still be ok.

Comment 15

[Olli Pettay [:smaug][[e-mail]]]

I don't know how slow or fast mozilla::widget::IsTouchDeviceSupportPresent() is, so its return value
should be cached. And nsContentUtils::InitializeTouchEventTable() should be called just once...
except that nsContentUtils::InitializeTouchEventTable() itself makes sure it is initialized just once.

So, cache the return value of mozilla::widget::IsTouchDeviceSupportPresent()

Comment 16

[Paul Rouget [:paul]]

Attachment: Patch v2 - nsDOMTouchEvent.cpp

Comment 17

[Paul Rouget [:paul]]

Attachment: patch v2 - responsive mode

Mike, I had to change a couple of things:
- reload if needed
- new way to handle the pref

Comment 18

[Paul Rouget [:paul]]

Attachment: Patch v1 - add notification box

Mike, because some events might have been added before the button is clicked, the user might need to reload the page to make the listeners work. I think we should show a notification.

Can you also please check the locales are in good english?

Comment 19

[Mike Ratcliffe [:miker] [:mratcliffe] [:mikeratcliffe]]

Comment on attachment 797920 [details] [diff] [review]
Patch v1 - add notification box

Review of attachment 797920 [details] [diff] [review]:
-----------------------------------------------------------------

Simple enough

Comment 20

[Paul Rouget [:paul]]

Try is green: https://tbpl.mozilla.org/?tree=Fx-Team&rev=61a110ae7914

https://hg.mozilla.org/integration/fx-team/rev/61a110ae7914
https://hg.mozilla.org/integration/fx-team/rev/8b82a374ece5
https://hg.mozilla.org/integration/fx-team/rev/acd1d8427718

Comment 21

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/acd1d8427718
https://hg.mozilla.org/mozilla-central/rev/8b82a374ece5
https://hg.mozilla.org/mozilla-central/rev/61a110ae7914