1200321 - Create Imagecompare RTL test for Settings:Network

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[No-Jun Park [:njpark]]

Create Imagecompare script that covers the Network and Connectivity section of Settings app

Comment 1

[GitHub Autolander Bot]

Attachment: [gaia] npark-mozilla:1200321 > mozilla-b2g:master

Comment 2

[No-Jun Park [:njpark]]

Comment on attachment 8656705 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

sorry for the big PR, but I have broken down to two commits:
- The older commit is strictly about Network & Connectivity section, and the improvements to the settings.return_to_prev_menu method.

- 2nd commit is mainly updating the use of return_to_prev_menu method.

Comment 3

[Martijn Wargers (dead)]

Comment on attachment 8656705 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

I get this error every time:
TEST-START | test_settings_NC_RTL.py TestSettingsRTLNC.test_settings_app
TEST-UNEXPECTED-ERROR | test_settings_NC_RTL.py TestSettingsRTLNC.test_settings_app | TimeoutException: TimeoutException: Timed out after 10.1 seconds


Traceback (most recent call last):
  File "/Users/mwargers/.virtualenvs/test3/lib/python2.7/site-packages/marionette_client-0.17-py2.7.egg/marionette/marionette_test.py", line 296, in run
    testMethod()
  File "/Users/mwargers/B2G/gaia_clean/tests/python/gaia-ui-tests/gaiatest/tests/graphics/RTL/test_settings_NC_RTL.py", line 104, in test_settings_app
    cs_page.wait_until_cb_info_received()
  File "/Users/mwargers/B2G/gaia_clean/tests/python/gaia-ui-tests/gaiatest/apps/settings/regions/call_settings.py", line 193, in wait_until_cb_info_received
    Wait(self.marionette).until(lambda m: status.get_attribute('data-l10n-id') == 'disabled')
  File "/Users/mwargers/.virtualenvs/test3/lib/python2.7/site-packages/marionette_driver-0.13-py2.7.egg/marionette_driver/wait.py", line 143, in until
    cause=last_exc)
TEST-INFO took 211202ms

The output of Marionette page gives me the indication that the frame that is being worked upon is the system frame for some reason.

I didn't get this problem when I ran with --store-reference-image , the first time, but the second time I ran it, I also got this error there.

Comment 4

[Martijn Wargers (dead)]

I just tried now with 512MB as ram on the Flame, but I still get this failure.

Comment 5

[Martijn Wargers (dead)]

Btw, I did look for the #baoc-desc element in webIDE when this failure occurs, and when I look for it, I can find out, this is the output of the surrounding element:
<ul id="list-callBarring-outgoing">
        <li id="li-cb-baoc">
          <label class="pack-switch checkbox-label">
            <input type="checkbox">
            <span data-l10n-id="callBarring-baoc">All</span>
            <small data-l10n-id="disabled" id="baoc-desc">Disabled</small>
          </label>
        </li>
        <li id="li-cb-boic">
          <label class="pack-switch checkbox-label">
            <input type="checkbox">
            <span data-l10n-id="callBarring-boic">International</span>
            <small data-l10n-id="disabled" id="boic-desc">Disabled</small>
          </label>
        </li>
        <li id="li-cb-boicExhc">
          <label class="pack-switch checkbox-label">
            <input type="checkbox">
            <span data-l10n-id="callBarring-boicExhc">International except Home Country</span>
            <small data-l10n-id="disabled" id="boicExhc-desc">Disabled</small>
          </label>
        </li>
      </ul>


However, after a while (half a minute or so?), this #baoc-desc element can disappear and instead the surrounding element is showing this:
<ul id="list-callBarring-outgoing">
        baoc-desc
        <li id="li-cb-boic">
          <label class="pack-switch checkbox-label">
            <input type="checkbox">
            <span data-l10n-id="callBarring-boic">International</span>
            <small data-l10n-id="disabled" id="boic-desc">Disabled</small>
          </label>
        </li>
        <li id="li-cb-boicExhc">
          <label class="pack-switch checkbox-label">
            <input type="checkbox">
            <span data-l10n-id="callBarring-boicExhc">International except Home Country</span>
            <small data-l10n-id="disabled" id="boicExhc-desc">Disabled</small>
          </label>
        </li>
      </ul>


This call barring menu was implemented in bug 910552.
Fernando, could you perhaps help here? Apparently, sometimes the outgoing calls All option can be not there. Is that desired?

Comment 6

[Martijn Wargers (dead)]

Ah, I see, when this bug occurs, the page is still in the "Requesting network info.." phase. Ok, I think something in the script needs to increase a timeout here. I just added a comment to that effect in the pull request page.

Comment 7

[Martijn Wargers (dead)]

I added the failures when I did a run here:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches6/

One type of failure:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches6/test_settings_app_alwaysFwd_flame_en-US_21+2015-09-09-11-59-15_diff.png
That is due to the caret blinking.
We should just add ui.caretBlinkTime=0 to: DEFAULT_PREFS in gaia_test.py, to prevent the caret from blinking (it's still showing then, only not blinking): http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/gaia_test.py#49


This failure, I don't know why:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches6/test_settings_app_bluetooth-renameDevice_flame_en-US_51+2015-09-09-12-01-41_diff.png

Here, there is a failure due to a name difference:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches6/test_settings_app_bluetooth-enabled_flame_en-US_50+2015-09-09-12-01-36_diff.png

Here, there is a failure due to the vkb still sliding in, while the screenshot was taken:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches6/test_settings_app_busyFwd_flame_en-US_22+2015-09-09-11-59-19_diff.png
Also here, I guess:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches6/test_settings_app_callBarring-passcode_flame_en-US_27+2015-09-09-11-59-52_diff.png
And also here, I guess:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches6/test_settings_app_sim_manager-pin_page_flame_en-US_10+2015-09-09-11-58-24_diff.png
And here:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches6/test_settings_app_unansweredFwd_flame_en-US_23+2015-09-09-11-59-24_diff.png

A failure in a differenc in a checkbox that is checked:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches6/test_settings_app_message-sim_1_flame_en-US_32+2015-09-09-12-00-17_diff.png
Perhaps you need a bigger timeout here before the checkbox is checked or something?

And here failures due to the different wifi names appearing:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches6/test_settings_app_wifi-enabled_flame_en-US_0+2015-09-09-11-57-39_diff.png

Comment 8

[Martijn Wargers (dead)]

Some of these lines you added seem very long. I did pep9 on call_settings.py and I see lines that are longer than 100. I don't think we are very strict on this, but I thing we want to keep the lines under 100 characters wide, in general, if possible.

Comment 9

[Martijn Wargers (dead)]

Comment on attachment 8656705 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

I would like to test this again, after you've addressed my commments/questions/remarks.

Comment 10

[No-Jun Park [:njpark]]

Comment on attachment 8656705 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

made fixes, and bluetooth.py is using PageRegion again with a quick workaround, same for return_to_prev_menu in settings.py

Comment 11

[No-Jun Park [:njpark]]

addressed the keyboard and caret related issues as well

Comment 12

[Martijn Wargers (dead)]

Comment on attachment 8656705 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

Looks good. I ran test_settings_NC_RTL.py again, I got 2 failures:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches7/test_settings_app_internet_sharing-hotspot_flame_en-US_54+2015-09-10-17-14-28_diff.png
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches7/test_settings_app_wifi-enabled_flame_en-US_0+2015-09-10-17-10-04_diff.png

The wifi page is know to be unstable and I guess the password can change on the Hotspot page.

Comment 13

[GitHub Autolander Bot]

Attachment: [gaia] npark-mozilla:1200321 > mozilla-b2g:master

Comment 14

[No-Jun Park [:njpark]]

Comment on attachment 8660127 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

I merged to the tip of master, and the next commit shows the changes I had to made to accommodate the code changes

Comment 15

[Martijn Wargers (dead)]

Comment on attachment 8660127 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

The additional fixes look fine to me and I assume you tested this.
If you want me to do additional testing, let me know.

Comment 16

[No-Jun Park [:njpark]]

Yup, I tested this repeatedly after the fix.  Just to make sure, I'll merge to the tip again, and if it looks good, I'll merge it.  Thanks!

Comment 17

[No-Jun Park [:njpark]]

Comment on attachment 8660127 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

Hi martijn, apparently RTL switches the location of the enable USB toggle switch, so I had to accomodate for it.  I made some fixes, (the 2nd and 3rd commit in the PR.) could you take a quick look and let me know?  thanks!

Comment 18

[Martijn Wargers (dead)]

Comment on attachment 8660127 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

I added some comments in the pull request. Let me know what you think. Perhaps we should discuss together on this.

Comment 19

[No-Jun Park [:njpark]]

Comment on attachment 8660127 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

final version

Comment 20

[Martijn Wargers (dead)]

Comment on attachment 8660127 [details] [review]
[gaia] npark-mozilla:1200321 > mozilla-b2g:master

Looks good, you need to fix the commenting out stuff in test_settings_personalization_RTL.py as we talked about on irc.
There are a couple of nits that you could fix also.

I ran test_settings_NC_RTL.py again and I only git this one meaningful mismatch:
http://people.mozilla.org/~mwargers/imgcomparescreenshots/mismatches8/test_settings_app_message-sim_1_flame_en-US_32+2015-09-16-16-32-35_diff.png

No idea why that happens, it looks like some rounding difference to me.

Comment 21

[No-Jun Park [:njpark]]

Merged:
https://github.com/mozilla-b2g/gaia/commit/e4490110b43c7636d0e34760ff2bf8eeef5ee9d2