Closed Bug 999447 Opened 10 years ago Closed 10 years ago

Active sync inbox seems empty in the tests on Jenkins

Categories

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Bebe, Unassigned)

References

Details

Attachments

(5 files)

Description: 	test_setup_and_send_active_sync_email.TestSetupAndSendActiveSyncEmail

Test can't find any email in the active sync inbox

We are seeing this issue on multiple devices and different build's


Environment:
Gaia      f0463704888881b8ed1619e8d4b0d851b0e0311b
Gecko     https://hg.mozilla.org/mozilla-central/rev/1d0496e30feb
BuildID   20140422040203
Version   31.0a1
ro.build.version.incremental=324
ro.build.date=Thu Dec 19 14:04:55 CST 2013

Traceback (most recent call last):

Traceback (most recent call last):
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.7.6-py2.7.egg/marionette/marionette_test.py", line 163, in run
testMethod()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/tests/functional/email/test_setup_and_send_active_sync_email.py", line 56, in test_setup_and_send_active_sync_email
self.email.wait_for_email(_subject)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/apps/email/app.py", line 124, in wait_for_email
self.email_exists(self, subject))
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.7.6-py2.7.egg/marionette/wait.py", line 122, in until
rv = condition(self.marionette)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/apps/email/app.py", line 138, in __call__
self.app.mails[0].scroll_to_message()
IndexError: list index out of range


Reproducible manually: NO
Can someone from MV verify this tests in MV and also please check that the credentials are OK on the nodes.

I checked the test_vars in the credentials repo and all seems OK
retornam can you take a look over this issue?
Flags: needinfo?(mozbugs.retornam)
(In reply to Florin Strugariu [:Bebe] from comment #1)
> Can someone from MV verify this tests in MV and also please check that the
> credentials are OK on the nodes.
> 
> I checked the test_vars in the credentials repo and all seems OK

The credentials are fine on all the machines. Is the an email in the active sync mailbox you are using? Which branch should I test on? Can you provide more information here.
Flags: needinfo?(mozbugs.retornam)
We are waiting for email's to be in the Inbox.
As we don't delete the emails from the account there are for sure emails in the Inbox

We run these tests on master 

Last reproducible build was:
Gaia      f0463704888881b8ed1619e8d4b0d851b0e0311b
Gecko     https://hg.mozilla.org/mozilla-central/rev/1d0496e30feb
BuildID   20140422040203
Version   31.0a1
ro.build.version.incremental=324
ro.build.date=Thu Dec 19 14:04:55 CST 2013


Keep in mind I could not reproduce this locally
It looks like perhaps the http request is timing out and it's assuming there's no mail in the folder.

Retornam can you run the test locally at your desk and observe whether it's completing the steps correctly?

We just can't tell from one screenshot (running remotely) whether it is working properly or not.
Flags: needinfo?(mozbugs.retornam)
Attached image email.png
This issue is still reproduced intermittently on the latest v1.4 build (Hamachi) with automated tests.

Build info:
Gaia      b1242f33981024de59b8b4c26bacff8b876211b1
Gecko     https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/fe4080728c60
BuildID   20140506000202
Version   30.0
ro.build.version.incremental=eng.tclxa.20131223.163538
ro.build.date=Mon Dec 23 16:36:04 CST 2013
Johan, would you be able to a) find the device(s) from our QA lab this has been failing on and b) get the device and run the test, locally (wait for any jobs to finish their run, first, of course)

Thanks!
Flags: needinfo?(jlorenzo)
Created an adhoc job and run it on 3 different devices: http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Hamachi/job/b2g.hamachi.mozilla-central.tinderbox.ui.active-sync-adhoc/

The 3 devices failed 100% times. Reproduces locally, same thing.

After talking to asuth, it is related to 825538. ActiveSync servers maintain a limited set of known devices for synchronization purposes. Hitting refresh won't do anything until there's a couple of new messages in the account. So a workaround is to have a couple of new messages in the inbox and then tap refresh button on the device to reset the synchronization with ActiveSync. I tried it locally and I could retrieve some message and go further in the test case (without passing it for the same reason).

TL;DR from asuth: It's absolutely essential that the automated tests never run concurrently with the same credentials until 825538 is fixed.
Depends on: 825538
Flags: needinfo?(jlorenzo)
asuth, jlorenzo the email account has tons of emails in it... 
I saw the same result with the new emails and refreshing.


@asuth @stephend would it work if we would use a different email account for every phone
Flags: needinfo?(stephen.donner)
Flags: needinfo?(bugmail)
Jlorenzo, thanks for the v thorough research.

Let's just disable this test until bug 825538 is resolved, IMO maintaining sets of credentials for all devices will be painful.

We still have imap tests run on-device as a smoke check.
I agree not running the test may be the simplest thing.  Note that even once we address bug 825538 it's still possible that there will be problems for the tests if you have a large number of devices running concurrently.  ActiveSync servers have some unknown internal limitation on the amount of devices they will track for an account; they could still invalidate credentials out from under devices.

Also, note that most IMAP servers enforce some types of connection limit.


Having said that, if there's some way you can just run the test ~once a day from a single device running a trunk or most-recent-stable version, I think that would be useful.  It seems like there are 2 things currently under test here:

1) Has the ActiveSync server changed its implementation and broken the email back-end?

2) Have we regressed the e-mail back-end or front-end as it relates to ActiveSync.

The second case is already largely covered by back-end tests and the JS Marionette tests will shortly be running against our ActiveSync fake-server too.  We don't have anything happening for the first case right now, although there is a medium term plan.
Flags: needinfo?(bugmail)
Thanks asuth, thorough as usual!

We'll disable the test for now.
Flags: needinfo?(stephen.donner)
Flags: needinfo?(mozbugs.retornam)
Attachment #8419368 - Flags: review?(zcampbell)
Attachment #8419368 - Flags: review?(andrei.hutusoru)
I don't think we should be waiting for bug 825538 to get resolved here, as I don't think there's an ETA on resolution of that bug. What we should be doing instead is working on modifying the test infrastructure to ensure that only one email test can run. That's going to require some overhead work, so the temporary disabling of that test is okay, but we can't let this stay disabled, since this is a smoketest of the phone.
Comment on attachment 8419368 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/19064

r-, make it disabled =
Attachment #8419368 - Flags: review?(zcampbell) → review-
Attachment #8419368 - Flags: review- → review?(zcampbell)
Comment on attachment 8419368 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/19064

r+

test is disabled:
https://github.com/mozilla-b2g/gaia/commit/4f352142a54f7f7ae2c460aad9049eda4b0edb14
Attachment #8419368 - Flags: review?(zcampbell)
Attachment #8419368 - Flags: review?(andrei.hutusoru)
Attachment #8419368 - Flags: review+
Comment on attachment 8420076 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/19113

r+
Attachment #8420076 - Flags: review?(zcampbell) → review+
Whiteboard: [xfail]
Resovled by using different accounts for all devices:
https://bugzilla.mozilla.org/show_bug.cgi?id=1012616
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Whiteboard: [xfail]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: