1022693 - [UI Tests] Intermittent Travis failure: TEST-UNEXPECTED-FAIL | test_persona_app.py test_persona_app.TestPersonaStandard.test_persona_standard_sign_in | KeyError: 'email'

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect, P1)

Description

[Julien Wajsberg [:julienw]]

TEST-START test_persona_app.py

test_persona_standard_sign_in (test_persona_app.TestPersonaStandard) ... ERROR

======================================================================

ERROR: None

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/build/mozilla-b2g/gaia/travis_venv/local/lib/python2.7/site-packages/marionette_client-0.7.7-py2.7.egg/marionette/marionette_test.py", line 145, in run

self.setUp()

File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/persona/test_persona_app.py", line 26, in setUp

"verifier": TESTUSER_VERIFIER}

File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/mocks/persona_test_user.py", line 56, in create_user

return MockUser(email=decode['email'], password=decode['pass'], name=decode['email'].split('@')[0])

TEST-UNEXPECTED-FAIL | test_persona_app.py test_persona_app.TestPersonaStandard.test_persona_standard_sign_in | KeyError: 'email'

----------------------------------------------------------------------

Ran 1 test in 71.985s

FAILED (errors=1)

Comment 1

[Julien Wajsberg [:julienw]]

https://travis-ci.org/mozilla-b2g/gaia/jobs/27125442

Comment 2

[Kevin Grandon :kgrandon]

This test has recently gone perma-red. This is a different failure, but putting it here because I think we need to disable this:

TEST-START test_persona_app.py

test_persona_standard_sign_in (test_persona_app.TestPersonaStandard) ... ERROR

======================================================================

ERROR: None

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/build/mozilla-b2g/gaia/travis_venv/local/lib/python2.7/site-packages/marionette_client-0.7.7-py2.7.egg/marionette/marionette_test.py", line 163, in run

testMethod()

File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/persona/test_persona_app.py", line 38, in test_persona_standard_sign_in

persona.login(self.user.email, self.user.password)

File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/apps/persona/app.py", line 28, in login

self.tap_continue()

File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/apps/persona/app.py", line 62, in tap_continue

self.wait_for_element_not_displayed(*self._continue_button_locator)

File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/apps/base.py", line 48, in wait_for_element_not_displayed

lambda m: not m.find_element(by, locator).is_displayed())

File "/home/travis/build/mozilla-b2g/gaia/travis_venv/local/lib/python2.7/site-packages/marionette_client-0.7.7-py2.7.egg/marionette/wait.py", line 143, in until

cause=last_exc)

TimeoutException: TimeoutException: Timed out after 10.0 seconds

TEST-UNEXPECTED-FAIL | test_persona_app.py test_persona_app.TestPersonaStandard.test_persona_standard_sign_in |

----------------------------------------------------------------------

Ran 1 test in 28.899s

FAILED (errors=1)

Comment 3

[Kevin Grandon :kgrandon]

Zac - Could you have someone look at this? It's gone perma-red, and this test constantly fails. Is it because persona goes down? 

Disabled: https://github.com/mozilla-b2g/gaia/commit/0361adc64a17dc2e45d851a7edaf7fd3312667a0

Comment 4

[Al Tsai [:atsai]]

It seems persona will take whole life to connecting. I'm checking the root cause.

Comment 5

[Zac C (:zac)]

It looks like the connection to the provider was just very slow.

Comment 6

[Al Tsai [:atsai]]

Finding:

There are two issues I found till now.

a) I cannot get the login page. The page stopped with "Sign In" title and empty contain.
b) I cannot finish loading from "Checking with your email provider"

This doesn't happen on mobile testing.

Comment 7

[Zac C (:zac)]

I'm getting stuck on b) 100% on desktopb2g

Comment 8

[[:AndreiH]]

I will take a look over this after issue: https://github.com/mozilla/persona/issues/4147 is resolved

Comment 9

[[:AndreiH]]

I started a build on Travis with the required changes, running only this test, let's see how it performs:
https://travis-ci.org/mozilla-b2g/gaia/builds/27938716

Comment 10

[[:AndreiH]]

https://travis-ci.org/mozilla-b2g/gaia/builds/29429911

Comment 11

[[:AndreiH]]

From the looks of it, this is a Persona issue, not something from our tests.
Running just this test on travis passed:
https://travis-ci.org/mozilla-b2g/gaia/jobs/29429917

Comment 12

[Julien Wajsberg [:julienw]]

Could we simulate a persona server in our tests?
(I know it's likely not easy but this would make our tests more resilient)

Comment 13

[[:AndreiH]]

Attachment: PR https://github.com/mozilla-b2g/gaia/pull/21522

Comment 14

[Florin Strugariu [:Bebe]]

https://github.com/mozilla-b2g/gaia/commit/9800aceb6e294502d4ae94c31b19d45e6f22f9b3

Comment 15

[Zac C (:zac)]

(In reply to Julien Wajsberg [:julienw] from comment #12)
> Could we simulate a persona server in our tests?
> (I know it's likely not easy but this would make our tests more resilient)

I've looked into this and it's possible but we need the Persona team to take a bit more control here I think.

One thing we could do is try and get the credentials but then skip the test if we couldn't get the credentials.

Comment 16

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

For future reference, please notify test authors (or at least someone/anyone on the team responsible for the tested app) before you disable their tests.

We have seen a number of critical Persona bugs in the past week that should have been caught by this test.

While Persona is not a high priority right now, some of the bugs seem to be popping up on 2.0, where it's far too late to do anything about it.

Comment 17

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Aw man, after looking at the persona bug referenced in comment 8, I see that you did tell someone on the team - but Jed left Mozilla shortly after the bug was filed :-(

Sorry for the misunderstanding, looks like this thread got dropped when he left