1071647 - [v2.1] Fix intermittent failure in test_cost_control_reset_wifi.py

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Viorela Ioia [:viorela]]

Test test_cost_control_reset_wifi.py is failing intermittently on v2.1: http://jenkins1.qa.scl3.mozilla.com/job/flame-319.mozilla-aurora.ui.functional.non-smoke/59/HTML_Report/
The test is failing because Homescreen is returned when calling self.apps.displayed_app.name, after the Browser app is launched, instead of Browser app. We need to add a better wait after launching Browser app. 

I didn't see this failing on master so far. I was able to replicate the failure locally, by running the automated test, but not manually.
Repro rate: 2 out of 10 times. 

Traceback (most recent call last):
  File "/home/viorelaioia/.virtualenvs/gaia_tests/local/lib/python2.7/site-packages/marionette_client-0.8.3-py2.7.egg/marionette/marionette_test.py", line 171, in run
    testMethod()
  File "/home/viorelaioia/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/cost_control/test_cost_control_reset_wifi.py", line 36, in test_cost_control_reset_wifi
    search.launch()
  File "/home/viorelaioia/gaia/tests/python/gaia-ui-tests/gaiatest/apps/search/app.py", line 19, in launch
    self.wait_for_condition(lambda m: self.apps.displayed_app.name == self.name)
  File "/home/viorelaioia/gaia/tests/python/gaia-ui-tests/gaiatest/apps/base.py", line 56, in wait_for_condition
    Wait(self.marionette, timeout).until(method, message=message)
  File "/home/viorelaioia/.virtualenvs/gaia_tests/local/lib/python2.7/site-packages/marionette_client-0.8.3-py2.7.egg/marionette/wait.py", line 143, in until
    cause=last_exc)

Device firmware (date) 	22 Sep 2014 16:16:10
Device firmware (incremental) 	eng.cltbld.20140922.191600
Device firmware (release) 	4.3
Device identifier 	flame
Gaia date 	22 Sep 2014 15:04:45
Gaia revision 	3742913e11f6
Gecko build 	20140922160204
Gecko revision 	b24d4d412645
Gecko version 	34.0a2

Comment 1

[Robert Chira [:RobertC]]

Attachment: browse.png

The test still fails intermittently on v2.1. When the test fails the screenshot shows the browser app not fully loaded (screenshot attached).
In the logcat we can see that the displayed app is the vertical homescreen, but I didn't notice any errors.

Build info v2.1 flame:
Gaia-Rev        94dcc25f2e34a4900ea58310c26be52bcb089161
Gecko-Rev       https://hg.mozilla.org/releases/mozilla-aurora/rev/baaa0c3ab8fd
Build-ID        20141002000202
Version         34.0a2
Device-Name     flame
FW-Release      4.4.2
FW-Incremental  eng.cltbld.20141002.031332
FW-Date         Thu Oct  2 03:13:43 EDT 2014
Bootloader      L1TC10011800

Comment 2

[Robert Chira [:RobertC]]

Attachment: logcat.txt

Comment 3

[Viorela Ioia [:viorela]]

I was able to get a failure of test_cost_control_reset_wifi.py on v2.2, because of the issue described here.
I started an adhoc run of this test on v2.2 and I got 1 failure out of 30 runs: http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc/187/HTML_Report/. 
The repro rate on v2.2 is very low, compared to repro rate on v2.1. 

Device firmware (date) 	07 Oct 2014 16:16:20
Device firmware (incremental) 	eng.cltbld.20141007.191609
Device firmware (release) 	4.4.2
Device identifier 	flame
Gaia date 	07 Oct 2014 06:32:28
Gaia revision 	9050edcda308
Gecko build 	20141007160202
Gecko revision 	0c8ae792f1c0
Gecko version 	35.0a1

We are still working on investigating this issue

Comment 4

[Zac C (:zac)]

Couldn't replicate this myself locally so I've thrown up a test PR in an adhoc job and will check the results on Monday.

Comment 5

[Zac C (:zac)]

Attachment: github pr

This removes a hack that viorela put in (and subsequently removed the need for with the atom change) and also uses local content which gets around the slow loading issue that RobertC identified.

Adhoc results here:
http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc/191/

Comment 6

[Viorela Ioia [:viorela]]

Comment on attachment 8503990 [details] [review]
github pr

r+, the adhoc looks good, one failure unrelated to the fix in this PR.

Comment 7

[Robert Chira [:RobertC]]

Comment on attachment 8503990 [details] [review]
github pr

Merged on master:
https://github.com/mozilla-b2g/gaia/commit/511a5a4b7f33749d1ac323549ffffc8a29561389

Comment 8

[Robert Chira [:RobertC]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25088

Uplift to 2.1
Adhoc: http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc/196/console

Comment 9

[Viorela Ioia [:viorela]]

Comment on attachment 8504047 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25088

r+, adhoc looks good.

Comment 10

[Viorela Ioia [:viorela]]

Merged on v2.1: https://github.com/mozilla-b2g/gaia/commit/d51956f84ebec0dd8998654f01f9f24fe8527f51