941158 - expected failures not honored when failure occurs in setUp or tearDown

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Jonathan Griffin (:jgriffin)]

It appears that when a test is marked as an expected failure in the manifest, this status isn't honored when the test fails in either setUp or tearDown, causing the test to be marked as an unexpected failure.

A couple of cases in point from a gaia-ui-tests run:

06:53:47 TEST-START test_import_contacts_from_gmail.py
06:58:44 test_import_contacts_from_gmail (test_import_contacts_from_gmail.TestImportContactsFromGmail) ... expected failure
06:58:44 ERROR
06:58:44 
06:58:44 ======================================================================
06:58:44 ERROR: None
06:58:44 ----------------------------------------------------------------------
06:58:44 Traceback (most recent call last):
06:58:44   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette_test.py", line 173, in run
06:58:44     self.tearDown()
06:58:44   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 759, in tearDown
06:58:44     MarionetteTestCase.tearDown(self)
06:58:44   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette_test.py", line 318, in tearDown
06:58:44     self.marionette.set_context("content")
06:58:44   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 803, in set_context
06:58:44     return self._send_message('setContext', 'ok', value=context)
06:58:44   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 553, in _send_message
06:58:44     raise MarionetteException(message="Please start a session")
06:58:44 TEST-UNEXPECTED-FAIL | test_import_contacts_from_gmail.py test_import_contacts_from_gmail.TestImportContactsFromGmail.test_import_contacts_from_gmail | MarionetteException: Please start a session

08:23:41 TEST-START test_settings_airplane_mode.py
08:25:29 test_toggle_airplane_mode (test_settings_airplane_mode.TestAirplaneMode) ... ERROR
08:25:29 
08:25:29 ======================================================================
08:25:29 ERROR: None
08:25:29 ----------------------------------------------------------------------
08:25:29 Traceback (most recent call last):
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette_test.py", line 127, in run
08:25:29     self.setUp()
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/tests/functional/settings/test_settings_airplane_mode.py", line 15, in setUp
08:25:29     self.data_layer.connect_to_wifi()
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 298, in connect_to_wifi
08:25:29     self.enable_wifi()
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 287, in enable_wifi
08:25:29     result = self.marionette.execute_async_script("return GaiaDataLayer.enableWiFi()", special_powers=True)
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 1073, in execute_async_script
08:25:29     filename=os.path.basename(frame[0]))
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 577, in _send_message
08:25:29     self._handle_error(response)
08:25:29   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 626, in _handle_error
08:25:29     raise ScriptTimeoutException(message=message, status=status, stacktrace=stacktrace)
08:25:29 TEST-UNEXPECTED-FAIL | test_settings_airplane_mode.py test_settings_airplane_mode.TestAirplaneMode.test_toggle_airplane_mode | ScriptTimeoutException: timed out

Comment 1

[Zac C (:zac)]

mdas asked me to mark up important Marionette issues for Q1

Comment 2

[Andreas Tolfsen ❲:ato❳]

Attachment: 0001-Bug-941158-Honour-expected-failures-in-setUp-or-tear.patch

Attaching a patch that solves the problem, but solves it in a suboptimal way due to the existing design of the function.  I've filed bug 967566 about refactoring this to reduce code duplication and complexity.

try: https://tbpl.mozilla.org/?tree=Try&rev=af64e5347ad4

Comment 3

[Zac C (:zac)]

Comment on attachment 8370123 [details] [diff] [review]
0001-Bug-941158-Honour-expected-failures-in-setUp-or-tear.patch

Actually now I see it I don't think I'm the right one to r? this. I will push it back to jgriffin who originally authored the code.

Comment 4

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8370123 [details] [diff] [review]
0001-Bug-941158-Honour-expected-failures-in-setUp-or-tear.patch

Review of attachment 8370123 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks, lgtm!

Comment 5

[Jonathan Griffin (:jgriffin)]

And thanks for filing bug 967566.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a246bbc0c61c

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Backed out in http://hg.mozilla.org/integration/mozilla-inbound/rev/54c21f290a8d for breaking Marionette: https://tbpl.mozilla.org/php/getParsedLog.php?id=34165579&tree=Mozilla-Inbound

Comment 8

[Andreas Tolfsen ❲:ato❳]

Attachment: 0001-Bug-941158-Honour-expected-failures-in-setUp-or-tear.patch

Attaching revised patch which fixes problem.  Carrying on r+ from jgriffin.

Comment 9

[Andreas Tolfsen ❲:ato❳]

Attachment: 0001-Bug-941158-Honour-expected-failures-in-setUp-or-tear.patch

Fixed wrong reviewer in patch commit message.

To be on the safe side I've triggered another try run: https://tbpl.mozilla.org/?tree=Try&rev=8edcde22e53e

Comment 10

[Andreas Tolfsen ❲:ato❳]

Got a green Gu in latest try run.  Due to bug 967826 Gu on Linux has a ~90% failure rate at the moment, so I'm taking this as a sign that things are OK with the revised patch.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/799c9a7716ef

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/799c9a7716ef

Comment 13

[Bob Silverberg [:bsilverberg]]

I'm not sure that this has been resolved properly. Currently, if a test is marked as an expected failure and it errors during tearDown it gets reported as both an expected failure and an error. I believe we should not be getting the latter.

Here is a sample stack trace from a job [1] on our CI:

test_import_contacts_from_gmail (test_import_contacts_from_gmail.TestImportContactsFromGmail) ... expected failure
ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette_test.py", line 173, in run
    self.tearDown()
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 1119, in tearDown
    MarionetteTestCase.tearDown(self)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette_test.py", line 318, in tearDown
    self.marionette.set_context("content")
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette.py", line 847, in set_context
    return self._send_message('setContext', 'ok', value=context)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette.py", line 581, in _send_message
    raise MarionetteException(message="Please start a session")
MarionetteException: MarionetteException: Please start a session
TEST-UNEXPECTED-FAIL | test_import_contacts_from_gmail.py test_import_contacts_from_gmail.TestImportContactsFromGmail.test_import_contacts_from_gmail | 
----------------------------------------------------------------------
Ran 1 test in 418.978s

FAILED (errors=1, expected failures=1)


[1] http://qa-selenium.mv.mozilla.com:8080/job/b2g.hamachi.mozilla-central.ui.adhoc/199/

Comment 14

[Andreas Tolfsen ❲:ato❳]

Since this patch wasn't backed out, I filed bug 974471 about the problem bsilverberg describes.  Resolving this.