1170090 - Intermittent failure in test_cost_control_data_alert_mobile.py at settings.reset_mobile_usage()

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Martijn Wargers (dead)]

I saw this failure:
http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.mozilla-central.tinderbox.ui.functional.smoke.bitbar/318/testReport/%28root%29/test_cost_control_data_alert_mobile_py%20TestCostControlDataAlertMobile_test_cost_control_data_alert_mobile/test_cost_control_data_alert_mobile_py_TestCostControlDataAlertMobile_test_cost_control_data_alert_mobile/
Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.tinderbox.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_client-0.13-py2.7.egg/marionette/marionette_test.py", line 296, in run
    testMethod()
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.tinderbox.ui.functional.smoke.bitbar/workspace/tests/python/gaia-ui-tests/gaiatest/tests/functional/cost_control/test_cost_control_data_alert_mobile.py", line 38, in test_cost_control_data_alert_mobile
    settings.reset_mobile_usage()
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.tinderbox.ui.functional.smoke.bitbar/workspace/tests/python/gaia-ui-tests/gaiatest/apps/cost_control/regions/settings.py", line 94, in reset_mobile_usage
    Wait(self.marionette).until(expected.element_not_displayed(reset_dialog))
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.tinderbox.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_driver-0.7-py2.7.egg/marionette_driver/wait.py", line 122, in until
    rv = condition(self.marionette)
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.tinderbox.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_driver-0.7-py2.7.egg/marionette_driver/expected.py", line 226, in __call__
    return not super(element_not_displayed, self).__call__(marionette)
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.tinderbox.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_driver-0.7-py2.7.egg/marionette_driver/expected.py", line 192, in __call__
    return self.el.is_displayed()
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.tinderbox.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_driver-0.7-py2.7.egg/marionette_driver/marionette.py", line 122, in is_displayed
    return self.marionette._send_message('isElementDisplayed', 'value', id=self.id)
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.tinderbox.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_driver-0.7-py2.7.egg/marionette_driver/decorators.py", line 36, in _
    return func(*args, **kwargs)
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.tinderbox.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_driver-0.7-py2.7.egg/marionette_driver/marionette.py", line 692, in _send_message
    raise errors.TimeoutException("Connection timed out")

TimeoutException: TimeoutException: Connection timed out

Comment 1

[Martijn Wargers (dead)]

Attachment: error2.txt

I did run test_cost_control_reset_wifi.py with a repeat of 100 times, I attached the log for that here.
http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/functional/cost_control/test_cost_control_reset_wifi.py#49
It often fails, apparently 33 times (although the counting of the log has a discrepancy of 10 somewhere).

What I noticed is that when this failure in settings.reset_wifi_usage() happens, in the next repeat, it's failing in MarionetteTestCase.setUp(self). Perhaps Gaia UI test automatically tries to rerun the test and is failing at that or something?

Comment 2

[Martijn Wargers (dead)]

This is similar to the issue in bug 1164078, btw.

Comment 3

[GitHub Autolander Bot]

Attachment: [gaia] mwargers:1170090 > mozilla-b2g:master

Comment 4

[Martijn Wargers (dead)]

Comment on attachment 8614017 [details] [review]
[gaia] mwargers:1170090 > mozilla-b2g:master

I've tested this locally with test_cost_control_reset_wifi.py repeating 50 times and it would pass every time.
Btw, I could make a switch_to_usage_setting() method for switching to the usage app settings frame and combine that within __init__. Let me know if you want that.

Comment 5

[No-Jun Park [:njpark]]

Comment on attachment 8614017 [details] [review]
[gaia] mwargers:1170090 > mozilla-b2g:master

this looks okay to me, as for combining to change frames, I'm guessing that might not be needed since we usually expect to manually change the frames, and that step is not hidden underneath extra layer.

Comment 6

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8614017 [details] [review]
[gaia] mwargers:1170090 > mozilla-b2g:master

To me, the less duplication, the better. It'll also shorten tap_confirm_reset().

Comment 7

[Martijn Wargers (dead)]

Comment on attachment 8614017 [details] [review]
[gaia] mwargers:1170090 > mozilla-b2g:master

Ok, updated the pull request with your comments. 
It turns out the whole "Wait(self.marionette).until(lambda m: self.apps.displayed_app.src == self._src)" line wasn't necessary, because the displayed app is the Usage app all the time, anyway.
Removing the line doesn't cause any problems.

Comment 8

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8614017 [details] [review]
[gaia] mwargers:1170090 > mozilla-b2g:master

LGTM!

Comment 9

[Martijn Wargers (dead)]

Merged: https://github.com/mozilla-b2g/gaia/commit/63bc2510f960cf1574cea186addd88a0780cce80