974471 - Expected failures in setUp/tearDown are counted as errors

Status: RESOLVED WONTFIX

(Remote Protocol :: Marionette, defect, P3)

Description

[Andreas Tolfsen ❲:ato❳]

From bsilverberg's comment in bug 941158:

I'm not sure that this has been resolved properly. Currently, if a test is marked as an expected failure and it errors during tearDown it gets reported as both an expected failure and an error. I believe we should not be getting the latter.

Here is a sample stack trace from a job [1] on our CI:

test_import_contacts_from_gmail (test_import_contacts_from_gmail.TestImportContactsFromGmail) ... expected failure
ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette_test.py", line 173, in run
    self.tearDown()
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 1119, in tearDown
    MarionetteTestCase.tearDown(self)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette_test.py", line 318, in tearDown
    self.marionette.set_context("content")
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette.py", line 847, in set_context
    return self._send_message('setContext', 'ok', value=context)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette.py", line 581, in _send_message
    raise MarionetteException(message="Please start a session")
MarionetteException: MarionetteException: Please start a session
TEST-UNEXPECTED-FAIL | test_import_contacts_from_gmail.py test_import_contacts_from_gmail.TestImportContactsFromGmail.test_import_contacts_from_gmail | 
----------------------------------------------------------------------
Ran 1 test in 418.978s

FAILED (errors=1, expected failures=1)


[1] http://qa-selenium.mv.mozilla.com:8080/job/b2g.hamachi.mozilla-central.ui.adhoc/199/

Comment 1

[David Burns :automatedtester]

[mass update] Setting Harness bugs to all P3

Comment 2

[Andreas Tolfsen ❲:ato❳]

Unlikely to ever be fixed.