750271 - Add touch support to Marionette

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Malini Das [:mdas] - Away, not checking bugmail]

Add the selenium atoms for touch support as soon as possible, as many of the gaia tests will rely on gestures.

Comment 1

[Jonathan Griffin (:jgriffin)]

wlach has found a way to send touch events in android, by writing them to /dev/input. See sample program here:

http://pastebin.mozilla.org/1649403

This simulates a swipe-down event.

This might be useful in some cases if higher-level events don't always work reliably.

Comment 2

[Malini Das [:mdas] - Away, not checking bugmail]

We can also send touch events over the adb bridge. They look like:

adb shell sendevent /dev/input/event0 0003 0 251

so we can send it from the client side.

Comment 3

[Malini Das [:mdas] - Away, not checking bugmail]

I should add I already have code to convert gestures done on an android device into these shell commands, so we can easily capture and reuse data client-side.

Comment 4

[William Lachance (:wlach)]

(In reply to Malini Das [:mdas] from comment #3)
> I should add I already have code to convert gestures done on an android
> device into these shell commands, so we can easily capture and reuse data
> client-side.

I have found this doesn't actually work very well in practice due to the latency involved in starting the sendevent utility. A drag gesture, for example, involves many touches over a brief (< 1 sec period). Even though sendevent doesn't take particularly long to start (a few msecs on a galaxy nexus), it's long enough that you can't really simulate the original event.

In any case, I've been slowing working on developing my original example code into a more general replacement for the Monkey tool which I've been using for Eideticker. So far I have it executing basic scripts of events, but anything up-to-and-including a custom TCP/IP server protocol is possible. I'd love help and/or extra hands! :)

https://github.com/wlach/orangatun

Comment 5

[Malini Das [:mdas] - Away, not checking bugmail]

We'll be implementing this using DOM level touch events.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So what's the difference to bug 809245?

Comment 7

[Malini Das [:mdas] - Away, not checking bugmail]

bug 809245 essentially sent over a gesture library that marionette would use for the touch events. It just does an import_script(...), and adds helper methods on top of it. 

This bug will actually implement the touch commands on the Marionette side so we won't rely on the outside gesture library. Some untested functions like pinch will be tested and made to work if they don't already.

Comment 8

[Malini Das [:mdas] - Away, not checking bugmail]

Also, we shouldn't neglect that we shouldn't be able to act on hidden elements. We should always check that an element is displayed before sending events to it.

Comment 9

[Malini Das [:mdas] - Away, not checking bugmail]

This was meant to be a tracking bug, but now that we have action chains and the like, we can close this.