Closed Bug 838236 Opened 11 years ago Closed 8 years ago

Most recent b2g panda Gaia UI test failures

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: armenzg, Unassigned)

References

Details

https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g_&rev=ff4b5bb4f0df

07:42:34 INFO - TEST-UNEXPECTED-FAIL | test_edit_contact.py TestContacts.test_edit_contact | AssertionError: Unable to insert contact {'comment': 'Gaia automated test', 'city': 'London', 'tel': {'type': 'Mobile', 'value': '55554110454'}, 'name': 'gaia110454 test', 'zip': '90210', 'country': 'UK', 'familyName': 'test', 'street': '101 Testing street', 'givenName': 'gaia110454', 'email': 'gaia110454@restmail.net'}
07:46:15 INFO - TEST-UNEXPECTED-FAIL | test_cards_view_with_three_apps.py TestCardsViewThreeApps.test_cards_view | AssertionError: First opened app should not be visible in cards view
07:47:09 INFO - TEST-UNEXPECTED-FAIL | test_everythingme.py TestEverythingMe.test_launch_everything_me_app | TimeoutException: Element //div/b[text()='Facebook'] not visible before timeout
07:48:27 ERROR - Return code: 10
All tests are failing again:
https://tbpl.mozilla.org/php/getParsedLog.php?id=19496624&tree=Cedar&full=1

All of them fail with this:
> TypeError: object of type 'bool' has no len()

This already started happening with rail's merge:
https://hg.mozilla.org/projects/cedar/rev/41fa159f8bb4
(In reply to Armen Zambrano G. [:armenzg] from comment #1)
> All tests are failing again:
> https://tbpl.mozilla.org/php/getParsedLog.php?id=19496624&tree=Cedar&full=1
> 
> All of them fail with this:
> > TypeError: object of type 'bool' has no len()
> 
> This already started happening with rail's merge:
> https://hg.mozilla.org/projects/cedar/rev/41fa159f8bb4

See https://github.com/mozilla/gaia-ui-tests/pull/372
(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> (In reply to Armen Zambrano G. [:armenzg] from comment #1)
> > All tests are failing again:
> > https://tbpl.mozilla.org/php/getParsedLog.php?id=19496624&tree=Cedar&full=1
> > 
> > All of them fail with this:
> > > TypeError: object of type 'bool' has no len()
> > 
> > This already started happening with rail's merge:
> > https://hg.mozilla.org/projects/cedar/rev/41fa159f8bb4
> 
> See https://github.com/mozilla/gaia-ui-tests/pull/372

This is now fixed.  The current errors are:

14:41:35     INFO -  TEST-UNEXPECTED-FAIL | test_edit_contact.py TestContacts.test_edit_contact | AssertionError: Unable to insert contact {'comment': 'Gaia automated test', 'city': 'London', 'tel': {'type': 'Mobile', 'value': '55595380149'}, 'name': 'gaia380149 test', 'zip': '90210', 'country': 'UK', 'familyName': 'test', 'street': '101 Testing street', 'givenName': 'gaia380149', 'email': 'gaia380149@restmail.net'}
14:41:45     INFO -  TEST-UNEXPECTED-FAIL | test_clock_all_items_present_new_alarm.py TestClockTestAllItemsPresentNewAlarm.test_all_items_present_new_alarm | NoSuchElementException: Unable to locate element: picker-container
14:45:00     INFO -  TEST-UNEXPECTED-FAIL | test_cards_view_with_three_apps.py TestCardsViewThreeApps.test_cards_view | AssertionError: First opened app should not be visible in cards view
14:47:26     INFO -  TEST-UNEXPECTED-FAIL | test_everythingme.py TestEverythingMe.test_launch_everything_me_app | TimeoutException: Element //div/b[text()='Facebook'] not visible before timeout
14:48:42    ERROR - Return code: 10

https://tbpl.mozilla.org/php/getParsedLog.php?id=19508280&tree=Mozilla-Inbound&full=1
Is anyone going to be able to look at this during this week?

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> (In reply to Jonathan Griffin (:jgriffin) from comment #2)
> > (In reply to Armen Zambrano G. [:armenzg] from comment #1)
> > > All tests are failing again:
> > > https://tbpl.mozilla.org/php/getParsedLog.php?id=19496624&tree=Cedar&full=1
> > > 
> > > All of them fail with this:
> > > > TypeError: object of type 'bool' has no len()
> > > 
> > > This already started happening with rail's merge:
> > > https://hg.mozilla.org/projects/cedar/rev/41fa159f8bb4
> > 
> > See https://github.com/mozilla/gaia-ui-tests/pull/372
> 
> This is now fixed.  The current errors are:
> 
> 14:41:35     INFO -  TEST-UNEXPECTED-FAIL | test_edit_contact.py
> TestContacts.test_edit_contact | AssertionError: Unable to insert contact
> {'comment': 'Gaia automated test', 'city': 'London', 'tel': {'type':
> 'Mobile', 'value': '55595380149'}, 'name': 'gaia380149 test', 'zip':
> '90210', 'country': 'UK', 'familyName': 'test', 'street': '101 Testing
> street', 'givenName': 'gaia380149', 'email': 'gaia380149@restmail.net'}
> 14:41:45     INFO -  TEST-UNEXPECTED-FAIL |
> test_clock_all_items_present_new_alarm.py
> TestClockTestAllItemsPresentNewAlarm.test_all_items_present_new_alarm |
> NoSuchElementException: Unable to locate element: picker-container
> 14:45:00     INFO -  TEST-UNEXPECTED-FAIL |
> test_cards_view_with_three_apps.py TestCardsViewThreeApps.test_cards_view |
> AssertionError: First opened app should not be visible in cards view
> 14:47:26     INFO -  TEST-UNEXPECTED-FAIL | test_everythingme.py
> TestEverythingMe.test_launch_everything_me_app | TimeoutException: Element
> //div/b[text()='Facebook'] not visible before timeout
> 14:48:42    ERROR - Return code: 10
> 
> https://tbpl.mozilla.org/php/getParsedLog.php?id=19508280&tree=Mozilla-
> Inbound&full=1

This is still the same set of oranges as comment 3.
https://tbpl.mozilla.org/php/getParsedLog.php?id=19674036&tree=Mozilla-Inbound&full=1
I should have the Gaia-Master tree unhidden tomorrow.

Can we disable those 4 tests so we can unhide the Gaia UI tests on all trees?
We can't unhide them until we have a realistic sheriffing story, which we currently don't.  Let's see how helpful the gaia-master work is in being able to identify the cause of future regressions; if that seems to work well, we can consider what we have to do to unhide the tests.
I've filed bug 843906 for the problem in test_edit_contacts.py.
The symbols issue got fixed yesterday (after a reconfiguration of our masters) but we had to wait for some mozpool issues that were happening to clear up.

The most recent failures are these:
https://tbpl-dev.allizom.org/php/getParsedLog.php?id=20559885&tree=Gaia-Master&full=1

I also see some logcat errors:
04:29:09     INFO - 01-16 18:49:21.430 E/GeckoConsole( 4346): Content JS ERROR at chrome://marionette/content/marionette-listener.js:1482 in GaiaDataLayer.getAllContacts/req.onerror: error finding contacts Not Allowed
04:29:09     INFO - 01-16 18:49:21.571 E/GeckoConsole( 4346): Content JS ERROR at chrome://marionette/content/marionette-listener.js:1482 in GaiaDataLayer.getAllContacts/req.onerror: error finding contacts Not Allowed
04:29:09     INFO - 01-16 18:49:21.571 E/GeckoConsole( 4346): Content JS LOG at chrome://marionette/content/marionette-listener.js:1503 in GaiaDataLayer.removeAllContacts/<: no contacts to remove
04:29:09     INFO - 01-16 18:49:27.071 E/GeckoConsole( 4346): [JavaScript Error: "[Exception... "Component returned failure code: 0x80470002 (NS_BASE_STREAM_CLOSED) [nsIOutputStream.write]"  nsresult: "0x80470002 (NS_BASE_STREAM_CLOSED)"  location: "JS frame :: chrome://global/content/devtools/dbg-transport.js :: DT_onOutputStreamReady :: line 94"  data: no]" {file: "chrome://global/content/devtools/dbg-transport.js" line: 94}]
and few more.


Summary of test failures:
############################
04:21:11     INFO -  TEST-UNEXPECTED-FAIL | test_browser_lan.py TestBrowserLAN.test_browser_lan | TimeoutException: Element page-title not found before timeout
04:21:11     INFO -  TEST-UNEXPECTED-FAIL | test_browser_lan.py TestBrowserLAN.test_browser_lan | JavascriptException: TypeError: window.navigator.mozSettings is null
04:28:33     INFO -  TEST-UNEXPECTED-FAIL | test_settings_bluetooth.py TestBluetoothSettings.test_toggle_bluetooth_settings | InvalidResponseException: Could not successfully complete transport of message to Gecko, socket closed?
04:28:33     INFO -  TEST-UNEXPECTED-FAIL | test_everythingme.py TestEverythingMe.test_launch_everything_me_app | InvalidResponseException: Could not successfully complete transport of message to Gecko, socket closed?
04:28:33     INFO -  TEST-UNEXPECTED-FAIL | test_delete_app.py TestDeleteApp.test_delete_app | error: [Errno 32] Broken pipe
04:28:33     INFO -  TEST-UNEXPECTED-FAIL | test_keyboard.py TestKeyboard.test_keyboard_basic | error: [Errno 32] Broken pipe
04:28:33     INFO -  TEST-UNEXPECTED-FAIL | test_launch_app.py TestLaunchApp.test_launch_app | error: [Errno 32] Broken pipe


Test failures:
################

04:20:09     INFO -  TEST-START test_browser_lan.py
04:21:10     INFO -  test_browser_lan (test_browser_lan.TestBrowserLAN) ... ERROR
04:21:11     INFO -  ERROR
04:21:11     INFO -  ======================================================================
04:21:11     INFO -  ERROR: test_browser_lan (test_browser_lan.TestBrowserLAN)
04:21:11     INFO -  ----------------------------------------------------------------------
04:21:11     INFO -  Traceback (most recent call last):
04:21:11     INFO -    File "/builds/panda-0094/test/build/gaia-ui-tests/gaiatest/tests/browser/test_browser_lan.py", line 46, in test_browser_lan
04:21:11     INFO -      self.wait_for_element_present(*self._page_title_locator)
04:21:11     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 388, in wait_for_element_present
04:21:11     INFO -      'Element %s not found before timeout' % locator)
04:21:11     INFO -  TEST-UNEXPECTED-FAIL | test_browser_lan.py TestBrowserLAN.test_browser_lan | TimeoutException: Element page-title not found before timeout
04:21:11     INFO -  ======================================================================
04:21:11     INFO -  ERROR: test_browser_lan (test_browser_lan.TestBrowserLAN)
04:21:11     INFO -  ----------------------------------------------------------------------
04:21:11     INFO -  Traceback (most recent call last):
04:21:11     INFO -    File "/builds/panda-0094/test/build/gaia-ui-tests/gaiatest/tests/browser/test_browser_lan.py", line 53, in tearDown
04:21:11     INFO -      GaiaTestCase.tearDown(self)
04:21:11     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 481, in tearDown
04:21:11     INFO -      f.write(json.dumps(self.data_layer.all_settings))
04:21:11     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 155, in all_settings
04:21:11     INFO -      return self.get_setting('*')
04:21:11     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 151, in get_setting
04:21:11     INFO -      return self.marionette.execute_async_script('return GaiaDataLayer.getSetting("%s")' % name, special_powers=True)
04:21:11     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 539, in execute_async_script
04:21:11     INFO -      scriptTimeout=script_timeout)
04:21:11     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 284, in _send_message
04:21:11     INFO -      self._handle_error(response)
04:21:11     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 319, in _handle_error
04:21:11     INFO -      raise JavascriptException(message=message, status=status, stacktrace=stacktrace)
04:21:11     INFO -  TEST-UNEXPECTED-FAIL | test_browser_lan.py TestBrowserLAN.test_browser_lan | JavascriptException: TypeError: window.navigator.mozSettings is null
04:21:11     INFO -  ----------------------------------------------------------------------
04:21:11     INFO -  Ran 1 test in 62.143s
04:21:11     INFO -  FAILED (errors=2)

04:28:02     INFO -  TEST-START test_settings_bluetooth.py
04:28:02     INFO -  test_toggle_bluetooth_settings (test_settings_bluetooth.TestBluetoothSettings)
04:28:33     INFO -  Toggle Bluetooth via Settings - Networks & Connectivity ... ERROR
04:28:33     INFO -  ======================================================================
04:28:33     INFO -  ERROR: test_toggle_bluetooth_settings (test_settings_bluetooth.TestBluetoothSettings)
04:28:33     INFO -  Toggle Bluetooth via Settings - Networks & Connectivity
04:28:33     INFO -  ----------------------------------------------------------------------
04:28:33     INFO -  Traceback (most recent call last):
04:28:33     INFO -    File "/builds/panda-0094/test/build/gaia-ui-tests/gaiatest/tests/settings/test_settings_bluetooth.py", line 46, in tearDown
04:28:33     INFO -      self.data_layer.set_setting('bluetooth.enabled', False)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 160, in set_setting
04:28:33     INFO -      result = self.marionette.execute_async_script('return GaiaDataLayer.setSetting("%s", %s)' % (name, value), special_powers=True)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 539, in execute_async_script
04:28:33     INFO -      scriptTimeout=script_timeout)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 269, in _send_message
04:28:33     INFO -      response = self.client.send(message)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/client.py", line 95, in send
04:28:33     INFO -      response = self.receive()
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/client.py", line 58, in receive
04:28:33     INFO -      status=ErrorCodes.INVALID_RESPONSE)
04:28:33     INFO -  TEST-UNEXPECTED-FAIL | test_settings_bluetooth.py TestBluetoothSettings.test_toggle_bluetooth_settings | InvalidResponseException: Could not successfully complete transport of message to Gecko, socket closed?
04:28:33     INFO -  ----------------------------------------------------------------------
04:28:33     INFO -  Ran 1 test in 30.867s
04:28:33     INFO -  FAILED (errors=1)
04:28:33     INFO -  TEST-START test_everythingme.py
04:28:33     INFO -  test_launch_everything_me_app (test_everythingme.TestEverythingMe) ... ERROR
04:28:33     INFO -  ======================================================================
04:28:33     INFO -  ERROR: test_launch_everything_me_app (test_everythingme.TestEverythingMe)
04:28:33     INFO -  ----------------------------------------------------------------------
04:28:33     INFO -  Traceback (most recent call last):
04:28:33     INFO -    File "/builds/panda-0094/test/build/gaia-ui-tests/gaiatest/tests/everythingme/test_everythingme.py", line 23, in setUp
04:28:33     INFO -      GaiaTestCase.setUp(self)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 317, in setUp
04:28:33     INFO -      MarionetteTestCase.setUp(self)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py", line 131, in setUp
04:28:33     INFO -      self.marionette.test_name = self.test_name
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 378, in test_name
04:28:33     INFO -      if self._send_message('setTestName', 'ok', value=test_name):
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 269, in _send_message
04:28:33     INFO -      response = self.client.send(message)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/client.py", line 95, in send
04:28:33     INFO -      response = self.receive()
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/client.py", line 58, in receive
04:28:33     INFO -      status=ErrorCodes.INVALID_RESPONSE)
04:28:33     INFO -  TEST-UNEXPECTED-FAIL | test_everythingme.py TestEverythingMe.test_launch_everything_me_app | InvalidResponseException: Could not successfully complete transport of message to Gecko, socket closed?
04:28:33     INFO -  ----------------------------------------------------------------------
04:28:33     INFO -  Ran 1 test in 0.001s
04:28:33     INFO -  FAILED (errors=1)
04:28:33     INFO -  TEST-START test_delete_app.py
04:28:33     INFO -  test_delete_app (test_delete_app.TestDeleteApp) ... ERROR
04:28:33     INFO -  ======================================================================
04:28:33     INFO -  ERROR: test_delete_app (test_delete_app.TestDeleteApp)
04:28:33     INFO -  ----------------------------------------------------------------------
04:28:33     INFO -  Traceback (most recent call last):
04:28:33     INFO -    File "/builds/panda-0094/test/build/gaia-ui-tests/gaiatest/tests/test_delete_app.py", line 25, in setUp
04:28:33     INFO -      GaiaTestCase.setUp(self)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 317, in setUp
04:28:33     INFO -      MarionetteTestCase.setUp(self)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py", line 131, in setUp
04:28:33     INFO -      self.marionette.test_name = self.test_name
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 378, in test_name
04:28:33     INFO -      if self._send_message('setTestName', 'ok', value=test_name):
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 269, in _send_message
04:28:33     INFO -      response = self.client.send(message)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/client.py", line 93, in send
04:28:33     INFO -      self.sock.send(packet)
04:28:33     INFO -  TEST-UNEXPECTED-FAIL | test_delete_app.py TestDeleteApp.test_delete_app | error: [Errno 32] Broken pipe
04:28:33     INFO -  ----------------------------------------------------------------------
04:28:33     INFO -  Ran 1 test in 0.001s
04:28:33     INFO -  FAILED (errors=1)

04:28:33     INFO -  TEST-START test_keyboard.py
04:28:33     INFO -  test_keyboard_basic (test_keyboard.TestKeyboard) ... ERROR
04:28:33     INFO -  ======================================================================
04:28:33     INFO -  ERROR: test_keyboard_basic (test_keyboard.TestKeyboard)
04:28:33     INFO -  ----------------------------------------------------------------------
04:28:33     INFO -  Traceback (most recent call last):
04:28:33     INFO -    File "/builds/panda-0094/test/build/gaia-ui-tests/gaiatest/tests/test_keyboard.py", line 26, in setUp
04:28:33     INFO -      GaiaTestCase.setUp(self)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 317, in setUp
04:28:33     INFO -      MarionetteTestCase.setUp(self)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py", line 131, in setUp
04:28:33     INFO -      self.marionette.test_name = self.test_name
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 378, in test_name
04:28:33     INFO -      if self._send_message('setTestName', 'ok', value=test_name):
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 269, in _send_message
04:28:33     INFO -      response = self.client.send(message)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/client.py", line 93, in send
04:28:33     INFO -      self.sock.send(packet)
04:28:33     INFO -  TEST-UNEXPECTED-FAIL | test_keyboard.py TestKeyboard.test_keyboard_basic | error: [Errno 32] Broken pipe
04:28:33     INFO -  ----------------------------------------------------------------------
04:28:33     INFO -  Ran 1 test in 0.000s
04:28:33     INFO -  FAILED (errors=1)
04:28:33     INFO -  TEST-START test_launch_app.py
04:28:33     INFO -  test_launch_app (test_launch_app.TestLaunchApp) ... ERROR
04:28:33     INFO -  ======================================================================
04:28:33     INFO -  ERROR: test_launch_app (test_launch_app.TestLaunchApp)
04:28:33     INFO -  ----------------------------------------------------------------------
04:28:33     INFO -  Traceback (most recent call last):
04:28:33     INFO -    File "/builds/panda-0094/test/build/gaia-ui-tests/gaiatest/tests/test_launch_app.py", line 16, in setUp
04:28:33     INFO -      GaiaTestCase.setUp(self)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 317, in setUp
04:28:33     INFO -      MarionetteTestCase.setUp(self)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py", line 131, in setUp
04:28:33     INFO -      self.marionette.test_name = self.test_name
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 378, in test_name
04:28:33     INFO -      if self._send_message('setTestName', 'ok', value=test_name):
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 269, in _send_message
04:28:33     INFO -      response = self.client.send(message)
04:28:33     INFO -    File "/builds/panda-0094/test/build/venv/lib/python2.7/site-packages/marionette/client.py", line 93, in send
04:28:33     INFO -      self.sock.send(packet)
04:28:33     INFO -  TEST-UNEXPECTED-FAIL | test_launch_app.py TestLaunchApp.test_launch_app | error: [Errno 32] Broken pipe
04:28:33     INFO -  ----------------------------------------------------------------------
04:28:33     INFO -  Ran 1 test in 0.000s
04:28:33     INFO -  FAILED (errors=1)
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.