1229322 - test_cost_control_data_alert_mobile.py intermittently fails: The data limit notification appears before we wait on it

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Johan Lorenzo [:jlorenzo]]

Bug 1225549 initially reported the problem on aries. In the latest automation report[1], test_cost_control_data_alert_mobile.py didn't pass. This part of automation still runs on Flame.

This is not a recent regression. I ran the test locally on today's aries[2] => it passed (like in bug 1225549 comment 23). I did the same on today's Flame[3] => it failed, no notification ever occurred. I apologize, it seems the verification I did in bug 1225549 wasn't complete; I should have run the test against Flame too. 

[1] http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-512.mozilla-central.nightly.ui.functional.smoke.bitbar/548/HTML_Report/
[2] Build ID               20151201023059
Gaia Revision          538c96ab7f17adbbd15b82f750e7247c5685e831
Gaia Date              2015-11-30 21:06:04
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/1f2684eae8bfb10b6cd86e631bf67956bbb2d451
Gecko Version          45.0a1
Device Name            aries
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.worker.20151201.015003
Firmware Date          Tue Dec  1 01:50:11 UTC 2015
Bootloader             s1

[3] Build ID               20151130150225
Gaia Revision          702773bee0b70e479ccebe5e061f571e977bc376
Gaia Date              2015-11-30 02:51:04
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/a18630f9ab42ddfde03ba8c7757a42069c48c7ed
Gecko Version          45.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20151130.184457
Firmware Date          Mon Nov 30 18:45:12 EST 2015
Bootloader             L1TC000118D0

Comment 1

[Johan Lorenzo [:jlorenzo]]

Tim, do you know if the threshold is different between aries and flame?

Comment 2

[Johan Lorenzo [:jlorenzo]]

Smoketest blocker.

Comment 3

[Isabel Rios[:isabel_rios]]

Fyi, Could not reproduce with Flame 2.6 either manually or running the automated test.
I also used two different sims, Simyo (214-19) and Tuenti (214-07).

Build ID               20151130150225
Gaia Revision          702773bee0b70e479ccebe5e061f571e977bc376
Gaia Date              2015-11-30 02:51:04
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/a18630f9ab42ddfde03ba8c7757a42069c48c7ed
Gecko Version          45.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20151130.184457
Firmware Date          Mon Nov 30 18:45:12 EST 2015
Bootloader             L1TC000118D0

Comment 4

[Tim Huang[:timhuang]]

The threshold is defined by the webidl, and aries and flame share the same webidl for the NetworkStat. Thus, I do not think they are different in terms of the threshold. 

I had tried on my flame, that the NetworkStat alarm works fine for me. But I did't use the automated test, I only tried it manually.

Comment 5

[Johan Lorenzo [:jlorenzo]]

Based on comment 3, comment 4 and the latest automation report[1], this might be only due on certain SIM cards. I'll give it another shot today. In the meantime, removing the smoketest blocker status.

[1] http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-512.mozilla-central.nightly.ui.functional.smoke.bitbar/550/HTML_Report/

Comment 6

[Johan Lorenzo [:jlorenzo]]

My apologies, I found out why the test was failing on my end, the notification appeared way before when the test waits on it.
On my setup, it happens when you type the URL on rocketbar. I checked the report showing the failure (in comment 0), you can see on the screenshot the blue line showing that you have an unread notification.

So there's a problem in the test. Moving and renaming the bug.

Comment 7

[BMO Automation]

Firefox OS is not being worked on