1189378 - Create Imagecompare RTL test for Settings:Personalization

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[No-Jun Park [:njpark]]

Create Imagecompare script that covers the Personalization section of Settings app

Comment 1

[GitHub Autolander Bot]

Attachment: [gaia] npark-mozilla:RTLWork > mozilla-b2g:master

Comment 2

[No-Jun Park [:njpark]]

Comment on attachment 8641922 [details] [review]
[gaia] npark-mozilla:RTLWork > mozilla-b2g:master

Hi Martijn, i know it looks like a lot of code, but basically what i did was
- Break up the script into several pieces
- Complete the screenshot taking action under Personalization field
- Add some helper methods in settings helper class
Thanks!

Comment 3

[Martijn Wargers (dead)]

Comment on attachment 8641922 [details] [review]
[gaia] npark-mozilla:RTLWork > mozilla-b2g:master

You need to add a manifest.ini file, I think, right?

For the rest I have added some comments, but they are more like questions for explanation.

Comment 4

[No-Jun Park [:njpark]]

Comment on attachment 8641922 [details] [review]
[gaia] npark-mozilla:RTLWork > mozilla-b2g:master

made some corrections

Comment 5

[Martijn Wargers (dead)]

Comment on attachment 8641922 [details] [review]
[gaia] npark-mozilla:RTLWork > mozilla-b2g:master

I'm getting errors while running these tests in test_settings_personalization_RTL.py, test_settings_device_RTL.py and test_settings_NC_RTL.py
I posted the errors in the pull request comments. Could you take a look at them?

In case you think they are not serious, just ask re-review then.

Comment 6

[No-Jun Park [:njpark]]

Comment on attachment 8641922 [details] [review]
[gaia] npark-mozilla:RTLWork > mozilla-b2g:master

So I fixed the error on one script, and the other one is caused by the memory pressure. (https://bugzilla.mozilla.org/show_bug.cgi?id=1181344) which can be fixed by using 512 or 1024MB config.

Also added other scripts in the manifest file, but I didn't remove the origin reference since it would make the script fail in arabic mode.  Perhaps I should wait until your fix is merged?

Comment 7

[No-Jun Park [:njpark]]

Reverted the name fix, awaiting for Bug 1188926

Comment 8

[Martijn Wargers (dead)]

Ok, I'm now able to run test_settings_NC_RTL.py, although I get these mismatches: http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches3/
One due to the wifi names all different, the other one due to the password field showing slightly differently, apparently.

Comment 9

[Martijn Wargers (dead)]

With 512MB on the Flame device, I'm able to run test_settings_personalization_RTL.py, however I do get a lot of mismatches again:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches3/
Some of them are related to time, but the other one is about an Addon "First Test" that is enabled in one and not the other? Perhaps you should look into that?

Comment 10

[Martijn Wargers (dead)]

Btw, your errors are like this:
TEST-UNEXPECTED-ERROR | test_settings_personalization_RTL.py TestSettingsRTL.test_settings_app | AssertionError: WARNING: 209 pixels mismatched between screenshots/test_settings_app_date_and_time_flame_en-US_30+2015-08-07-10-55-42.png and reference_images/test_settings_app_date_and_time_flame_en-US_30.png

But I think those image files are now inside the folder screenshots/mismatches/ . Could you fix that in the reporting?

Comment 11

[No-Jun Park [:njpark]]

oh, that's intentional, I want to separate the failed images (and its source) so the reviewer can identify them faster.  The review now only needs to visible screenshots/mismatches/ folder.

I also need to do a writeup about RTL test process, which will document the steps on how to use the tool.  (which will be a separate bug)

Comment 12

[No-Jun Park [:njpark]]

(In reply to Martijn Wargers [:mwargers] (QA) from comment #8)
> Ok, I'm now able to run test_settings_NC_RTL.py, although I get these
> mismatches:
> http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches3/
> One due to the wifi names all different, the other one due to the password
> field showing slightly differently, apparently.

Yes, this is expected, and the reviewer will need to be aware of this.

Comment 13

[No-Jun Park [:njpark]]

(In reply to Martijn Wargers [:mwargers] (QA) from comment #10)
 
> But I think those image files are now inside the folder
> screenshots/mismatches/ . Could you fix that in the reporting?

ah, I see what you mean, I think that's a good idea.

Comment 14

[No-Jun Park [:njpark]]

just added a fix in PR to indicate the failed diff's whereabouts.

Comment 15

[No-Jun Park [:njpark]]

Merged: 

https://github.com/mozilla-b2g/gaia/commit/8f8f673bf2ef8d523161e0f039a4c79bdcf4c027