1025167 - Pinch action does not work in Browser app

Status: RESOLVED INCOMPLETE

(Remote Protocol :: Marionette, defect)

Description

[No-Jun Park [:njpark]]

Following code zooms in/out images in Gallery app, but does not work in a Browser page:

    def test_UI_Actions(self):
        """https://moztrap.mozilla.org/manage/case/2462/"""
        gallery = Gallery(self.marionette)
        gallery.launch()
        gallery.wait_for_files_to_load(self.image_count)


         # Tap first image to open full screen view.
        image = gallery.tap_first_gallery_item()

        action = Actions(self.marionette)
        screen = self.marionette.find_element(*self._current_image_locator)

        time.sleep(5)

        pinch = MultiActions(self.marionette)
        current_image_mid_x = screen.size['width'] / 2
        current_image_mid_y = screen.size['height'] / 2

        index_finger = Actions(self.marionette)
        thumb = Actions(self.marionette)
        index_finger.press(screen,current_image_mid_x+10,current_image_mid_y-10).wait(0.15).move_by_offset(current_image_mid_x, -current_image_mid_y).release()
        thumb.press(screen,current_image_mid_x-10,current_image_mid_y+10).wait(0.15).move_by_offset(-current_image_mid_x, current_image_mid_y).wait().release()
        pinch.add(thumb).add(index_finger).perform()

Also tried a flick-like method, and while it does scroll the browser page, it still does not pinch zoom when combined together in a MultiActions object:

from gaiatest import GaiaTestCase
from marionette.marionette import Actions
from marionette.marionette import MultiActions
import sys,time
from marionette.by import By
from gaiatest.apps.browser.app import Browser

class TestUIActions(GaiaTestCase):

    _browser_frame_locator = (By.CSS_SELECTOR, 'iframe.browser-tab')
    _main_screen_locator = (By.ID, 'main-screen')

    def setUp(self):
        GaiaTestCase.setUp(self)

        self.data_layer.connect_to_wifi()

    def test_UI_Actions(self):
        browser = Browser(self.marionette)
        browser.launch()

        browser.go_to_url('http://mozilla.org/firefoxos')
        time.sleep(10)
        screen = self.marionette.find_element(*browser._browser_frame_locator)

        pinch = MultiActions(self.marionette)
        current_image_mid_x = screen.size['width'] / 2
        current_image_mid_y = screen.size['height'] / 2

        index_finger = Actions(self.marionette)
        thumb = Actions(self.marionette)
        #pdb.set_trace()
        index_finger.press(screen,current_image_mid_x-50,current_image_mid_y-50)
        x = 0
        while x < 100:
            index_finger.move_by_offset(-5, -5)
            index_finger.wait(0.001)
            x += 1
        index_finger.release()
      
        thumb.press(screen,current_image_mid_x+50,current_image_mid_y+50)
        y = 0
        while y < 100:
            thumb.move_by_offset(5, 5)
            thumb.wait(0.001)
            y += 1
        thumb.wait()
        thumb.release()
      
        pinch.add(index_finger).add(thumb).perform()

Comment 1

[No-Jun Park [:njpark]]

It looks to me Browser's response to UI action is either slow, or requires something different from other apps.

Badly worded comment above: for the second code above, if I perform the actions individually, it does scroll up and down, but when combined as above, there was no UI action shown.

Comment 2

[No-Jun Park [:njpark]]

mdas actually didn't get around fixing this issue, wondering whether there is any chance this can be fixed?  if so we can automate a lot of use case scenarios.  ni?ing :automatedtester for comments.

Comment 3

[David Burns :automatedtester]

Unfortunately the A-Team doesn't have bandwidth for this for some time. I think you would be better asking the FXOS Automation Team if they can do it. I am happy to do reviews.

Comment 4

[James Lal [:lightsofapollo]]

This is unlikely to happen this quarter.... Are there any known workarounds?

Comment 5

[No-Jun Park [:njpark]]

CC'ing FXOS automation team

Comment 6

[Martijn Wargers (dead)]

Attachment: pinchzoom.diff

I tried to get the current pinch action to work, which is currently used here:
http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/graphics/orientation_zoom_base.py#59

But I couldn't get it working. I don't think what's in the tree currently, is working either.

Comment 7

[Martijn Wargers (dead)]

Attachment: test_gallery_zoom.py

I tried something like this based on comment 0, but that doesn't work either.
What I noticed with using print(datetime.datetime.now().time()) is that MultiActions takes only 0.015s, where it should take at least 1.15s, because of th wait() calls in the Actions().

Comment 8

[Martijn Wargers (dead)]

The tests that are supposed to test this behavior in Marionette unit tests are disabled because of bug 1060060.

Comment 9

[Martijn Wargers (dead)]

Attachment: test_gallery_zoom.py

This is something that works.

Comment 10

[Martijn Wargers (dead)]

Attachment: test_browser_zoom.py

I tried the same in a browser zoom test, but there, it doesn't work for some reason.

Comment 11

[Martijn Wargers (dead)]

Attachment: test_browser_zoom.py

Using sendNativeTouchPoint like this works. This also works for the gallery app.

Thanks to :kats on irc who explained this to me.
In gaia browser, pinch zooming is handled by apz, in that case you need to use sendNativeTouchPoint. In the gallery app, pinch zooming is handled by the gallery app itself, so that's why sendTouchEvent works there as well:
https://groups.google.com/forum/#!msg/mozilla.dev.fxos/1Kh8RrK7QWI/lX74BeHJAQAJ
https://hg.mozilla.org/mozilla-central/raw-file/tip/gfx/doc/B2GInputFlow.svg

Comment 12

[Martijn Wargers (dead)]

Attachment: test_browser_zoom.py

This was my last attempt that points to a page that shows the various touch events. This is not zooming in the page on pinch zoom.
I noticed that when clientY is negative for the 2nd touch, then the page gets indeed zoomed in suddenly. This is what happened with the previous test that I attached.

Comment 13

[David Burns :automatedtester]

closing as b2g related and new actions work has been done