925342 - travis gaia-ui-tests busted with "unable to locate element" errors

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Ben Kelly [:bkelly, not reviewing]]

This morning we started getting travis bustage on the gaia-ui-tests with errors like these:

TEST-UNEXPECTED-FAIL | test_everythingme_launch_packaged_app.py TestEverythingMeSearchPanel.test_launch_packaged_app_from_search_panel | NoSuchElementException: Unable to locate element: button.keyboard-key[data-keycode="67"]

TEST-UNEXPECTED-FAIL | test_persona_app.py TestPersonaStandard.test_persona_standard_sign_in | TimeoutException: Element li.login not present before timeout

Comment 1

[Ben Kelly [:bkelly, not reviewing]]

Bisecting suggests this is caused by:

  https://github.com/mozilla-b2g/gaia/commit/99096794

Note, its a bit intermittent, but reproduces about 50% or 60% of the time.

Comment 2

[Ben Kelly [:bkelly, not reviewing]]

Backed out:

  https://github.com/mozilla-b2g/gaia/commit/f5c2e987fcc859e371fedb44006ca94fc0ebcb28

Comment 3

[Evyatar 'Tron' Amitay (everything.me)]

First off thanks for backing it out Ben! And of course sorry for introducing it.

Can someone from automation help me understand this? It looks like according to the errors it can't find the letter 'C' on the keyboard, or it can't find the "li.login" element - neither of which are relevant (as far as I know) to e.me code. Also it seems to work well on an actual device (Unagi).

So I'm just trying to make sense of this issue.

Comment 4

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Evyatar 'Tron' Amitay (everything.me) from comment #3)
> Can someone from automation help me understand this? It looks like according
> to the errors it can't find the letter 'C' on the keyboard, or it can't find
> the "li.login" element - neither of which are relevant (as far as I know) to
> e.me code. Also it seems to work well on an actual device (Unagi).

First, the backout I made was mainly in regards to the keyboard error.  I didn't actually see the other error duplicating, so thats probably a rare intermittent.

I thought perhaps the keyboard issue might be related to the changes in keyboard visibility events for the refactor.  Perhaps its trying to type before the keyboard is actually displayed.

Comment 5

[Zac C (:zac)]

Attachment: eme_twitter_ss.png

Tron, if it helps we are seeing these tests fail on device but with different symptoms. (which is not uncommon because the device is less powerful than desktopb2g)
I've attached a screenshot that shows when typing some characters go missing. The search term is supposedto be "twitter". Visually watching the device during the test it's all 'a bit slow' as best as I can describe it.

Usually this is quite reliable since Ran made reliability fixes a couple of weeks ago. Ran can show you how to run these tests against a device locally.

Comment 6

[Ben Kelly [:bkelly, not reviewing]]

Zac, I think that might be a different problem.  In this case the test fails before ever typing a single character.  It can't find the first key to hit when it tries to start typing.

I would really recommend running against b2gdesktop in order to use the same environment travis is using.

(Having them work correctly on devices is important, too, of course!)

Comment 7

[Ben Kelly [:bkelly, not reviewing]]

To reproduce on b2gdesktop you can run the same steps travis uses:

  cd gaia
  npm install
  ./node_modules/.bin/travisaction gaia_ui_tests install
  ./node_modules/.bin/travisaction gaia_ui_tests script

That last step runs all the tests, which takes a long time.  So you can instead run just the single failing test with:

  gaiatest --app=b2gdesktop --binary=b2g/b2g-bin --profile=profile --restart tests/python/gaia-ui-tests/gaiatest/tests/manifest.ini /srv/gaia-master/tests/python/gaia-ui-tests/gaiatest/tests/functional/everythingme/test_everythingme_launch_packaged_app.py