1090281 - Investigate failure in test_setup_and_send_imap_email.py

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Viorela Ioia [:viorela]]

Test test_setup_and_send_imap_email.py is failing intermittently in Jenkins: http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.mozilla-central.ui.functional.smoke/89/HTML_Report/
I was not able to reproduce the failure locally, by running the automated test several times, or manually.
The test is failing only on master, it works as expected on v2.1. 

Traceback (most recent call last):
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.4-py2.7.egg/marionette/marionette_test.py", line 264, in run
testMethod()
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.smoke/tests/python/gaia-ui-tests/gaiatest/tests/functional/email/test_setup_and_send_imap_email.py", line 70, in test_setup_and_send_imap_email
self.assertEqual(_body, read_email.body.splitlines()[0])
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.smoke/tests/python/gaia-ui-tests/gaiatest/apps/email/regions/read_email.py", line 17, in body
return self.marionette.find_element(*self._body_locator).text
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.4-py2.7.egg/marionette/marionette.py", line 1279, in find_element
response = self._send_message('findElement', 'value', **kwargs)
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.4-py2.7.egg/marionette/decorators.py", line 35, in _
return func(*args, **kwargs)
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.4-py2.7.egg/marionette/marionette.py", line 638, in _send_message
self._handle_error(response)
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.4-py2.7.egg/marionette/marionette.py", line 672, in _handle_error
raise errors.NoSuchElementException(message=message, status=status, stacktrace=stacktrace)
NoSuchElementException: NoSuchElementException: Unable to locate element: .card.center .msg-body-content

Device firmware (date) 	28 Oct 2014 04:31:24
Device firmware (incremental) 	eng.cltbld.20141028.073110
Device firmware (release) 	4.4.2
Device identifier 	flame
Gaia date 	27 Oct 2014 10:43:23
Gaia revision 	6a7fb482a03c
Gecko build 	20141028040202
Gecko revision 	a255a234946e
Gecko version 	36.0a1

We need to still try to reproduce the issue.

Comment 1

[Viorela Ioia [:viorela]]

I've been able to reproduce this failure locally by running the automated test, on latest v2.2. 
It is intermittent, repro rate would be 1/10

The test is failing because the subject of the most recent received email is not tapped, so we won't reach the read email view, to get the subject & body of the read email. 
I'm working on finding a better wait before tapping on capture button. 
I will add a PR afterwards.

Comment 2

[Viorela Ioia [:viorela]]

Test test_setup_and_send_imap_email.py is also failing at some other step: 
http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.ui.functional.smoke/965/HTML_Report/

Traceback (most recent call last):
File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.4-py2.7.egg/marionette/marionette_test.py", line 264, in run
testMethod()
File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/tests/python/gaia-ui-tests/gaiatest/tests/functional/email/test_setup_and_send_imap_email.py", line 70, in test_setup_and_send_imap_email
self.assertEqual(_body, read_email.body.splitlines()[0])
AssertionError: 'b1415176107152861' != u'Test email message 1415134095.91'

I suppose this failure is caused by the fact that the wrong subject was tapped.
It is possible to receive a new email right before tapping the subject, as the IMAP account is used by several devices.
In that case, that new email's subject will be tapped, because it is the most recent in the list. 

I did some changes in the test, so that we tap the email that has the expected subject, and not the most recent email.
I don't think these changes will fix the issue described in comment 0, but they are an improvement.

Comment 3

[Viorela Ioia [:viorela]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25765

Comment 4

[Zac C (:zac)]

Comment on attachment 8517379 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25765

r+

Comment 5

[Viorela Ioia [:viorela]]

Merged in https://github.com/mozilla-b2g/gaia/commit/460c43d6db3382235314d4f5a6e8fdaab7610394

Comment 6

[Viorela Ioia [:viorela]]

Sorry, this fix was for the error described in comment 2. I'm not sure it also fixed the error from comment 0. I started an adhoc to check this: http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc/283/console

Comment 7

[Florin Strugariu [:Bebe]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/26145

Uplift for 2.1

Comment 8

[Viorela Ioia [:viorela]]

Comment on attachment 8522887 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/26145

Merged in v2.1: https://github.com/mozilla-b2g/gaia/commit/7f346739120703a16d9535daf6c7325bc4085336

Comment 9

[Viorela Ioia [:viorela]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25894

Fix for the error described in comment 0. See adhoc runs results in the PR.

Comment 10

[Robert Chira [:RobertC]]

Comment on attachment 8525909 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25894

comment in the PR

Comment 11

[Florin Strugariu [:Bebe]]

https://github.com/mozilla-b2g/gaia/commit/717808b02cd904d3c6ec694faab6d4652c7b0615