1197910 - Create Imagecompare RTL test for Settings:Device

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[No-Jun Park [:njpark]]

Create Imagecompare script that covers the Device section of Settings app

Comment 1

[GitHub Autolander Bot]

Attachment: [gaia] npark-mozilla:1197910 > mozilla-b2g:master

Comment 2

[Martijn Wargers (dead)]

Comment on attachment 8652997 [details] [review]
[gaia] npark-mozilla:1197910 > mozilla-b2g:master

I'm still looking at the pull request, but I see one pattern that needs to be fixed in many places.
You do something like this:
        Wait(self.marionette).until(expected.element_displayed(
+            self.marionette.find_element(*self._notice_text_locator)))

Where you can actually do this:
        Wait(self.marionette).until(expected.element_displayed(*self._notice_text_locator))

You need to fix all of those instances.

Comment 3

[No-Jun Park [:njpark]]

Comment on attachment 8652997 [details] [review]
[gaia] npark-mozilla:1197910 > mozilla-b2g:master

Is there a way to look into the content of Iframe?  I can't seem to find basic elements within.  I also put a detailed question in the PR.

Comment 4

[Martijn Wargers (dead)]

Comment on attachment 8652997 [details] [review]
[gaia] npark-mozilla:1197910 > mozilla-b2g:master

Looks good to me, apart for some nits.
These are the mismatches, I got:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches5/
One related to battery charging, one related to Downloads (the "No Downloads" text comes a bit later, you might want to adjust your screenshot taking time for that) and one related to crash reporting (which was my fault, because one crash interfered during the test).

Comment 5

[Martijn Wargers (dead)]

Oh, btw, I tried to run the test on the Aries device and that didn't work.
It started failing in tap_legal_info(), when I added scrollintoview before the element tapping there, it worked (apparently, the resolution difference is interfering with your scrolling).
After that, I got another failure in return_to_prev_menu with .tap(25, 25). I gave up on that. I guess that coordinate causes it to not work in Aries.

Comment 6

[No-Jun Park [:njpark]]

Aha, so this doesn't work on Aries?  I guess I'll have to open a new bug to make the RTL tests to work with Aries device.  Thanks for letting me know.

I saw the mismatches too, I'll get them on new builds because it lists the gecko and gaia build version as well, I'm documenting the discrepancies so the testers would be aware of them.

Comment 7

[No-Jun Park [:njpark]]

Comment on attachment 8652997 [details] [review]
[gaia] npark-mozilla:1197910 > mozilla-b2g:master

resubmitting for review after using the iframe check.

Comment 8

[Martijn Wargers (dead)]

Comment on attachment 8652997 [details] [review]
[gaia] npark-mozilla:1197910 > mozilla-b2g:master

Looks great, weird that I didn't get this switch to iframe thing working when I tried it, but it's certainly working now. This time I also didn't get any imagecompare failures.

(In reply to No-Jun Park [:njpark] from comment #6)
> Aha, so this doesn't work on Aries?  I guess I'll have to open a new bug to
> make the RTL tests to work with Aries device.  Thanks for letting me know.

Yes, what I saw was that apparently, the scrolling you do is not scrolling far enough on the Aries device, which results at some point that the element can't be tapped upon, because it's out of view.
Another failure was  in return_to_prev_menu with .tap(25, 25), which looks also like a coordinate issue.

Comment 9

[No-Jun Park [:njpark]]

merged:
https://github.com/mozilla-b2g/gaia/commit/ab26e8748a63cbc6c7c7fa26574001ef15acca2d