1177553 - test_cost_control_data_alert_mobile.py: " TimeoutException: Timed out after 20.1 seconds "

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Oliver Nelson [:oliverthor]]

Attachment: jun25_adhoc.html

Description:
Test is failing with high repro rate. Unsure what the fault is, as this does not reproduce manually. Test fails early on when attempting to reset the initial recorded, after performing in the FTU:

File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/cost_control/test_cost_control_data_alert_mobile.py", line 38, in test_cost_control_data_alert_mobile
    settings.reset_mobile_usage()


Repro Steps:
1) Update phone to 20150624080416
2) Enable Data, disable Wi-Fi
3) Open the Cost Control app; enable data limit of 0.1 MB
4) Reset current observed data before performing 'test'

Actual:
Timeout occurs when attempting to reset data

Expected: 
Data is reset and test continues

Traceback (most recent call last):
  File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/.env/local/lib/python2.7/site-packages/marionette_client-0.13-py2.7.egg/marionette/marionette_test.py", line 296, in run
    testMethod()
  File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/cost_control/test_cost_control_data_alert_mobile.py", line 38, in test_cost_control_data_alert_mobile
    settings.reset_mobile_usage()
  File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/apps/cost_control/regions/settings.py", line 105, in reset_mobile_usage
    self.tap_confirm_reset()
  File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/apps/cost_control/regions/settings.py", line 83, in tap_confirm_reset
    self.switch_to_settings_iframe()
  File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/apps/cost_control/regions/settings.py", line 42, in switch_to_settings_iframe
    *self._settings_title_locator))))
  File "/home/flash/Desktop/oliverthor/git/B2G/gaia/tests/python/gaia-ui-tests/.env/local/lib/python2.7/site-packages/marionette_driver-0.7-py2.7.egg/marionette_driver/wait.py", line 143, in until
    cause=last_exc)
TimeoutException: TimeoutException: Timed out after 20.1 seconds


Environmental Variables:
Device firmware (base)	L1TC100118D0
Device firmware (date)	24 Jun 2015 12:38:44
Device firmware (incremental)	eng.cltbld.20150624.153831
Device firmware (release)	4.4.2
Device identifier	flame
Gaia date	23 Jun 2015 18:48:14
Gaia revision	eb0d4aefa62b
Gecko build	20150624080416
Gecko revision	4cdc1a95a672
Gecko version	41.0a1

Reproducible manually: NO

Repro frequency: 5/6

Comment 1

[Martijn Wargers (dead)]

I'll take a look at this on Monday (data plans in a foreign country is probably very expensive).

Comment 2

[Martijn Wargers (dead)]

Hrm, while running this test, I get this failure:
TEST-UNEXPECTED-ERROR | test_cost_control_data_alert_mobile.py TestCostControlDataAlertMobile.test_cost_control_data_alert_mobile | TimeoutException: TimeoutException: Timed out after 40.0 seconds with message: Data usage bar did not breach limit


Traceback (most recent call last):
  File "/Users/mwargers/.virtualenvs/gaia-py-latest/lib/python2.7/site-packages/marionette_client-0.16-py2.7.egg/marionette/marionette_test.py", line 296, in run
    testMethod()
  File "/Users/mwargers/B2G/gaia_clean/tests/python/gaia-ui-tests/gaiatest/tests/functional/cost_control/test_cost_control_data_alert_mobile.py", line 66, in test_cost_control_data_alert_mobile
    message='Data usage bar did not breach limit')
  File "/Users/mwargers/.virtualenvs/gaia-py-latest/lib/python2.7/site-packages/marionette_driver-0.9-py2.7.egg/marionette_driver/wait.py", line 143, in until
    cause=last_exc)
TEST-INFO took 224103ms

SUMMARY
-------
passed: 0
failed: 1
todo: 0

When I look at that code, I'm surprised it didn't fail on this before, actually.

Comment 3

[Martijn Wargers (dead)]

Ok, when this failure is happening, there is an alert in the way that says you've reached the limit of mobile usage, just after the 0.1MB value is put in.
Switching  settings.toggle_data_alert_switch(True) and settings.reset_mobile_usage() makes it work.
I guess there is something in the system that is already eating quite some band width, before setting the limit.

Comment 4

[Martijn Wargers (dead)]

(In reply to Martijn Wargers [:mwargers] (QA) from comment #2)
> When I look at that code, I'm surprised it didn't fail on this before,
> actually.

Never mind, I misunderstood the test a bit. But this test can be improved a bit, anyway.

Comment 5

[Martijn Wargers (dead)]

Attachment: https://github.com/mozilla-b2g/gaia/pull/30781

This fixes the failure, but I wonder about this dialog I'm seeing while running this test shortly after putting in the 01.MB value.
I'll attach a screenshot of that dialog shortly.

Comment 6

[Martijn Wargers (dead)]

Attachment: 100kb_reached.png

I'm seeing this dialog shortly after the 0.1MB is filled in. This is before the browser is opened even, so I don't know where this mobile data usage is coming from.

Comment 7

[Martijn Wargers (dead)]

Ok, while running this test, I'm seeing mobile data usage already at 24.5MB or something before resetting mobile data. I wonder where that's coming from. I also wonder why this isn't deleted by a Gaiatest rerun.

In any case, the pull request fixes the failure.

I guess I should at least disable mobile data usage at tearDown.

Comment 8

[Martijn Wargers (dead)]

Ok, I guess that somehow the Marionette communication is being added to the mobile data usage. Because I keep seeing numbers (MBs of data being used) that don't make sense.

This looks like a regression and we should try to find out what caused it.

I still think the pull request would be good to have, although it would stop catching issues like this. But I think this should better be covered in a regression unit test.

Comment 9

[Martijn Wargers (dead)]

Seems to work in:
Build ID               20150622010206
Gaia Revision          d8bac9577a537b5e6bafc3f7ed088af5ea60c99e
Gaia Date              2015-06-19 13:21:50
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/a38c23ccca0a
Gecko Version          41.0a1

Fails in:
Build ID               20150622160206
Gaia Revision          311c4e59936a407e64509f54fecb440d8a78e3c8
Gaia Date              2015-06-20 20:21:42
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/be81b8d6fae9
Gecko Version          41.0a1

Comment 10

[Martijn Wargers (dead)]

https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2015-06-20+10%3A21%3A42&enddate=2015-06-22+23%3A21%3A42

I bet this is a regression from bug 1168658. Especially see bug 1168658, comment 18:
"
Fabrice, now that Bug 1157081 is fixed, it would be a good idea to flip the Safe Browsing prefs off by default (and test that TP still works) because Safe Browsing is way more data-hungry than TP.
"

I especially noticed that the longer you wait, the higher the mobile data usage, btw.

Comment 11

[Martijn Wargers (dead)]

Attachment: all.diff

Johan, No-Jun. This is more a request for feedback. This would disable safebrowsing globally.
As an alternative, I could disable these prefs in/as part of connect_to_cell_data().
Or I could only do it for the test that's failing in this bug.

Comment 12

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8628348 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/30781

Looks great! One small question left at the end, but I wouldn't block on it.

Comment 13

[Martijn Wargers (dead)]

I've now added disable_cell_data() in tearDown to all the tests that turn mobile data on in their tests.

Comment 14

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8628502 [details] [diff] [review]
all.diff

Review of attachment 8628502 [details] [diff] [review]:
-----------------------------------------------------------------

Based on the discussion in bug 1168658, I think that's the best thing we can do.

Comment 15

[Martijn Wargers (dead)]

Started Jenkins adhoc run with the last pull request (which also has the gaia_test.py changes in): http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc/848/

Comment 16

[Martijn Wargers (dead)]

It would be good to know what Dave Hunt thinks about this, disabling safe browsing globally (although it might be temporarily, until bug 1179770 gets fixed).

Comment 17

[No-Jun Park [:njpark]]

Comment on attachment 8628502 [details] [diff] [review]
all.diff

 I think for testing purpose, this shouldn't be too much of a problem?  (Unless some of our test cases rely on safe browsing and tracking protection, but I doubt it)

Comment 18

[No-Jun Park [:njpark]]

Comment on attachment 8628348 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/30781

LGTM too.

Comment 19

[Martijn Wargers (dead)]

(In reply to Martijn Wargers [:mwargers] (QA) from comment #15)
> Started Jenkins adhoc run with the last pull request (which also has the
> gaia_test.py changes in):
> http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc/848/

Ok, most tests pass and the failures are unrelated to this pull request.
test_cost_control_data_alert_mobile.py is actually failing on Jenkins, but I think that is happening because bug 1177800 is checked in and Jenkins bitbar is using already a build with that checked in (and I didn't).
I'll file a new bug about that and fix the locators.

Merged: https://github.com/mozilla-b2g/gaia/commit/d22b4cd9d7a90fcd772bc9793fb0104eff019a41

Comment 20

[Martijn Wargers (dead)]

(In reply to Martijn Wargers [:mwargers] (QA) from comment #19)
> I'll file a new bug about that and fix the locators.

I filed bug 1180062 for it.

Comment 21

[Martijn Wargers (dead)]

I guess with bug 1180503 fixed, we could remove the safe browsing prefs (see all.diff diff).
I filed bug 1181225 for it.