1164073 - test_system_notification_bar.py: "AssertionError: Expected one notification."

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Oliver Nelson [:oliverthor]]

Description:
Test is failing to on reading notifications from system tray, as an extra notification has appeared. The phone this test was performed on appears to have adopted a 'voicemail' notification, which is interfering with it's final check. This issue was not reproduced via local automation, as the tray was clear except for the propagation by the test.

* http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2/218/HTML_Report/
* http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc/830/HTML_Report/

Traceback (most recent call last):
File "/var/jenkins/2/workspace/flame-kk.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.12-py2.7.egg/marionette/marionette_test.py", line 296, in run
testMethod()
File "/var/jenkins/2/workspace/flame-kk.ui.adhoc/tests/python/gaia-ui-tests/gaiatest/tests/functional/system/test_system_notification_bar.py", line 34, in test_notification_bar
self.assertEqual(1, len(notifications), 'Expected one notification.')
AssertionError: Expected one notification.

Jenkins Repro frequency: 0/5
Local Repro frequency: 5/5 passing
Environmental Variables:
Device firmware (base) 	L1TC100118D0
Device firmware (date) 	12 May 2015 08:32:19
Device firmware (incremental) 	eng.cltbld.20150512.043209
Device firmware (release) 	4.4.2
Device identifier 	flame
Device memory 	219772 kB
Device serial 	e481d8da
Device uptime 	0 days 0 hours 2 minutes 10 seconds
Gaia date 	10 May 2015 13:57:12
Gaia revision 	6089234ace8b
Gecko build 	20150512010209
Gecko revision 	502e1a5e722f
Gecko version 	41.0a1


Reproducible manually: NO

Comment 1

[Oliver Nelson [:oliverthor]]

* http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.mozilla-central.nightly.ui.functional.smoke/228/HTML_Report/

test_IMAP_email_notification.py just failed today in the mozilla-central.nightly smoke report on 05/14 for the same error. It appears these phones are receiving a call during their tests which is propogating the voicemail and throwing off it's count. I attempted to reproduce this the other day locally after having called the device prior to local automation testing, and the voicemail removed itself from the tray during the scope of the tests. 
May just be bad luck in receiving a call during the test that's pushing the additional notification. Or if the tray doesn't reset between a string of tests, it may encounter this issue as well.

Traceback (most recent call last):
File "/var/jenkins/2/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.13-py2.7.egg/marionette/marionette_test.py", line 296, in run
testMethod()
File "/var/jenkins/2/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.smoke/tests/python/gaia-ui-tests/gaiatest/tests/functional/email/test_IMAP_email_notification.py", line 63, in test_IMAP_email_notification
self.assertEqual(1, len(notifications), 'Expected one notification.')
AssertionError: Expected one notification.

Comment 2

[GitHub Autolander Bot]

Attachment: [gaia] mwargers:1164073 > mozilla-b2g:master

Comment 3

[Martijn Wargers (dead)]

Comment on attachment 8608016 [details] [review]
[gaia] mwargers:1164073 > mozilla-b2g:master

I think that putting the device to flight mode and then enabling wifi would circumvent this.
If this is an agreed upon thing to do, we might also have to do this with:
http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/functional/lockscreen/test_lockscreen_wake_with_notification.py?force=1
http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/functional/lockscreen/test_lockscreen_notification.py?force=1
http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/functional/system/test_system_notification_bar.py?force=1

It also makes me wonder if Gaia UI test should have these setting as in:
http://mxr.mozilla.org/gaia/source/shared/test/integration/profile.js#16
Because I was seeing an apps update notification, which might have potentially interfered with this test.

Comment 4

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8608016 [details] [review]
[gaia] mwargers:1164073 > mozilla-b2g:master

The fix for this test looks coherent.

I agree, factorizing the state of the phone is a good idea. However, airplane mode by default will make fail the tests that require a carrier and cell data.
So maybe we could infer the state of the phone depending on the manifest. What do you think, Martijn?

Comment 5

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8608016 [details] [review]
[gaia] mwargers:1164073 > mozilla-b2g:master

Clearing the review, per the discussion we had on IRC with :davehunt. We can improve the test by waiting on the first EMail notification instead of the first notification

Comment 6

[Martijn Wargers (dead)]

Ok, this fix is not preferred by Dave. Basically, we should stick to the default setup as much as possible, because that mimicks real life testing the most.

What needs to be done:
- All the sims need to have their voicemail disabled by their carrier.
- These tests should be hardened, so they will test that the correct notification appears and so that they don't respond to other notifications.

Comment 7

[Martijn Wargers (dead)]

Comment on attachment 8608016 [details] [review]
[gaia] mwargers:1164073 > mozilla-b2g:master

Something like this would harden the tests. What do you think?

Comment 8

[Martijn Wargers (dead)]

(In reply to Martijn Wargers [:mwargers] (QA) from comment #6)
> What needs to be done:
> - All the sims need to have their voicemail disabled by their carrier.

I've filed bug 1167666 for this.

Comment 9

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8608016 [details] [review]
[gaia] mwargers:1164073 > mozilla-b2g:master

r+ modulo a refactor I think we could do to make the tests simpler.

Comment 10

[Martijn Wargers (dead)]

Comment on attachment 8608016 [details] [review]
[gaia] mwargers:1164073 > mozilla-b2g:master

Thanks for you suggestions, I've followed these and I think you need to take a relook at it.

Comment 11

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8608016 [details] [review]
[gaia] mwargers:1164073 > mozilla-b2g:master

You seem to have a perma-orange on test_lockscreen_notification.py. I currently don't what change affected this test.

Comment 12

[Martijn Wargers (dead)]

Comment on attachment 8608016 [details] [review]
[gaia] mwargers:1164073 > mozilla-b2g:master

Sorry, a stupid leftover, which I now reverted. I test locally now that the test is passing.

Comment 13

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8608016 [details] [review]
[gaia] mwargers:1164073 > mozilla-b2g:master

Looks good to me!

Comment 14

[No-Jun Park [:njpark]]

Comment on attachment 8608016 [details] [review]
[gaia] mwargers:1164073 > mozilla-b2g:master

makes sense, so we now should state what kind of notifications we are waiting for.

Comment 15

[Martijn Wargers (dead)]

(In reply to No-Jun Park [:njpark] from comment #14)
> Comment on attachment 8608016 [details] [review]
> [gaia] mwargers:1164073 > mozilla-b2g:master
> 
> makes sense, so we now should state what kind of notifications we are
> waiting for.

Yes, I also think it might be good to replace all .notifications usage with .get_notifications():
http://mxr.mozilla.org/gaia/search?string=.notifications&find=.py

https://github.com/mozilla-b2g/gaia/commit/74e4793d4a8618255c4a255fa13ff11a97f7cc72