1227186 - Certain tests fail in MV lab consistently while passing on bitbar and local setup

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[No-Jun Park [:njpark]]

STR:
fetch https://github.com/npark-mozilla/gaia/tree/scroll, and run gaiatest/tests/graphics/RTL/test_settings_personalization_RTL.py  (It does not have any imagecompare specific methods) on MV lab.

Expected:
Passes normally, as this script only opens settings and tap themes page
Actual:
Consistently fails on MV lab, while consistently passing locally and on bitbar setup.

07:05:03  0:04.84 TEST_START: MainThread test_settings_personalization_RTL.py TestSettingsRTLPersonalization.test_settings_app
07:06:14  1:15.29 TEST_END: MainThread ERROR, expected PASS
07:06:14 Traceback (most recent call last):
07:06:14   File "/var/jenkins/1/workspace/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-1.0.0-py2.7.egg/marionette/marionette_test.py", line 296, in run
07:06:14     testMethod()
07:06:14   File "/var/jenkins/1/workspace/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/tests/python/gaia-ui-tests/gaiatest/tests/graphics/RTL/test_settings_personalization_RTL.py", line 146, in test_settings_app
07:06:14     themes_page = settings.open_themes()
07:06:14   File "/var/jenkins/1/workspace/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/tests/python/gaia-ui-tests/gaiatest/apps/settings/app.py", line 229, in open_themes
07:06:14     return self._open_subpage(self._theme_menu_item_locator, 'themes', 'Themes')
07:06:14   File "/var/jenkins/1/workspace/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/tests/python/gaia-ui-tests/gaiatest/apps/settings/app.py", line 290, in _open_subpage
07:06:14     self._tap_menu_item(locator)
07:06:14   File "/var/jenkins/1/workspace/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/tests/python/gaia-ui-tests/gaiatest/apps/settings/app.py", line 329, in _tap_menu_item
07:06:14     menu_item.tap()
07:06:14   File "/var/jenkins/1/workspace/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/.env/local/lib/python2.7/site-packages/marionette_driver-1.0.0-py2.7.egg/marionette_driver/marionette.py", line 83, in tap
07:06:14     self.marionette._send_message("singleTap", body)
07:06:14   File "/var/jenkins/1/workspace/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/.env/local/lib/python2.7/site-packages/marionette_driver-1.0.0-py2.7.egg/marionette_driver/decorators.py", line 36, in _
07:06:14     return func(*args, **kwargs)
07:06:14   File "/var/jenkins/1/workspace/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/.env/local/lib/python2.7/site-packages/marionette_driver-1.0.0-py2.7.egg/marionette_driver/marionette.py", line 716, in _send_message
07:06:14     self._handle_error(resp)
07:06:14   File "/var/jenkins/1/workspace/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/.env/local/lib/python2.7/site-packages/marionette_driver-1.0.0-py2.7.egg/marionette_driver/marionette.py", line 770, in _handle_error
07:06:14     raise errors.lookup(error)(message, stacktrace=stacktrace)
07:06:14 ElementNotVisibleException: ElementNotVisibleException: Element is not currently visible and may not be manipulated

Comment 1

[No-Jun Park [:njpark]]

Jlorenzo, could it be that the job configs are set up incorrectly for me?  I copied an old MV job, and I have checked that it creates virtualenvs with the correct versions of the tools.  Do you see anything wrong with the job config here for example? http://jenkins1.qa.scl3.mozilla.com/view/Graphics/job/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/

The test is a very simple 4-line script as below- not sure what could make it different, unless that device is faulty.
        settings = Settings(self.marionette)
        settings.launch()
        themes_page = settings.open_themes()
        settings.return_to_prev_menu(settings.screen_element, themes_page.screen_element)

Comment 2

[No-Jun Park [:njpark]]

See also https://bugzilla.mozilla.org/show_bug.cgi?id=1226761 for another MV lab only issue as well.

Comment 3

[No-Jun Park [:njpark]]

Here is the bitbar adhoc that passes this test: 
http://jenkins1.qa.scl3.mozilla.com/view/Graphics/job/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc.bitbar/83/

Comment 4

[No-Jun Park [:njpark]]

Even with the latest base image, it still fails:
http://jenkins1.qa.scl3.mozilla.com/view/Graphics/job/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/63/console

Comment 5

[Johan Lorenzo [:jlorenzo]]

Discussed with No-Jun yesterday on IRC, but we didn't find any solution. The latest builds[1] show green. I'm not sure what changed over the last night. Did you manage to fix it, No-Jun?

[1] http://jenkins1.qa.scl3.mozilla.com/view/Graphics/job/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc/ from build 68 to 72

Comment 6

[No-Jun Park [:njpark]]

Sorry, turns out that the job was downloading flame-kk spark eng build, when it was supposed to download the nightly.  After getting the nightly eng build, this went away.