888506 - Unable to skip a test in setUp before calling MarionetteTestCase.setUp

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

In some tests we would like to skip when certain prerequisites are not met, such as certain test variables. It would be ideal to do this before even setting up a Marionette session or some other expensive setUp operations, however with the recent changes from bug 874599 and bug 886741 this is no longer possible.

If I have a setUp method such as:

def setUp(self):
    try:
        self.testvars['email']['ActiveSync']
    except KeyError:
        raise SkipTest('account details not present in test variables')
    GaiaTestCase.setUp(self)

I get the following traceback:

Traceback (most recent call last):
  File "/Users/dhunt/.virtualenvs/gaia-ui-tests/bin/gaiatest", line 8, in <module>
    load_entry_point('gaiatest==0.12', 'console_scripts', 'gaiatest')()
  File "/Users/dhunt/workspace/gaia-ui-tests/gaiatest/runtests.py", line 70, in main
    cli(runner_class=GaiaTestRunner, parser_class=GaiaTestOptions)
  File "/Users/dhunt/.virtualenvs/gaia-ui-tests/lib/python2.7/site-packages/marionette/runtests.py", line 766, in cli
    runner = startTestRunner(runner_class, options, tests)
  File "/Users/dhunt/.virtualenvs/gaia-ui-tests/lib/python2.7/site-packages/marionette/runtests.py", line 758, in startTestRunner
    runner.run_tests(tests)
  File "/Users/dhunt/.virtualenvs/gaia-ui-tests/lib/python2.7/site-packages/marionette/runtests.py", line 417, in run_tests
    self.run_test(test)
  File "/Users/dhunt/.virtualenvs/gaia-ui-tests/lib/python2.7/site-packages/marionette/runtests.py", line 520, in run_test
    results = runner.run(suite)
  File "/Users/dhunt/.virtualenvs/gaia-ui-tests/lib/python2.7/site-packages/marionette/runtests.py", line 153, in run
    test(result)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 395, in __call__
    return self.run(*args, **kwds)
  File "/Users/dhunt/.virtualenvs/gaia-ui-tests/lib/python2.7/site-packages/marionette/marionette_test.py", line 168, in run
    self.cleanTest()
  File "/Users/dhunt/.virtualenvs/gaia-ui-tests/lib/python2.7/site-packages/marionette/marionette_test.py", line 247, in cleanTest
    self._deleteSession()
  File "/Users/dhunt/.virtualenvs/gaia-ui-tests/lib/python2.7/site-packages/marionette/marionette_test.py", line 250, in _deleteSession
    self.duration = time.time() - self.start_time
AttributeError: 'TestSendActiveSyncEmail' object has no attribute 'start_time'

In this case duration should be 0 as the test is skipped. If I put the raise SkipTest after the setUp method I get no traceback.

Comment 1

[Jonathan Griffin (:jgriffin)]

We should check that the start_time attribute exists before attempting to use it.

Comment 2

[mathieu bultel:mbu]

Attachment: Prevent error if object as not attribute start_time

Prevent start_time attribute error and add unit tests.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 769639 [details] [diff] [review]
Prevent error if object as not attribute start_time

Review of attachment 769639 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/marionette/client/marionette/marionette_test.py
@@ +247,5 @@
>          self._deleteSession()
>  
>      def _deleteSession(self):
> +        if hasattr(self, 'start_time'):
> +            self.duration = time.time() - self.start_time

This would mean duration would not be set. Could we set duration in this case to zero? Otherwise we would need to catch this in the XML/HTML reports.

Comment 4

[mathieu bultel:mbu]

Duration is set into __init__ of CommonTestCase :

    def __init__(self, methodName):
        unittest.TestCase.__init__(self, methodName)
        self.loglines = None
        self.duration = 0

So, if setUp failed, duration is still set to zero.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Ah, thanks! :)

Comment 6

[Jonathan Griffin (:jgriffin)]

Comment on attachment 769639 [details] [diff] [review]
Prevent error if object as not attribute start_time

Review of attachment 769639 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks mbu.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/projects/birch/rev/df89dc4a968d

Please fix your hg settings so your username and email are properly formatted.
https://developer.mozilla.org/en-US/docs/Mercurial_FAQ#How_can_I_generate_a_patch_for_somebody_else_to_check-in_for_me.3F

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/df89dc4a968d

Comment 9

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/290e1a7a4fbc

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/290e1a7a4fbc