1021573 - Intermittent failure on Loop Marionette tests: self._addClassOrModuleLevelException(result, e, errorName)

Status: RESOLVED FIXED

(Hello (Loop) :: Client, defect, P1)

Description

[Mark Banner (:standard8)]

We're seeing this randomly across the three test files that we run. Only seen on Mac so far. I'm not quite sure why it is yet, as it seems to be not reaching our code (or maybe its the cleanup after our code).

https://tbpl.mozilla.org/php/getParsedLog.php?id=41140374&tree=Elm

00:42:17     INFO -  TEST-START test_desktop_all.py
00:42:17     INFO -  Traceback (most recent call last):
00:42:17     INFO -    File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runtests.py", line 33, in <module>
00:42:17     INFO -      cli()
00:42:17     INFO -    File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runtests.py", line 28, in cli
00:42:17     INFO -      runner = startTestRunner(runner_class, options, tests)
00:42:17     INFO -    File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runtests.py", line 19, in startTestRunner
00:42:17     INFO -      runner.run_tests(tests)
00:42:17     INFO -    File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 821, in run_tests
00:42:17     INFO -      self.run_test_sets()
00:42:17     INFO -    File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 1040, in run_test_sets
00:42:17     INFO -      self.run_test_set(in_process_tests)
00:42:17     INFO -    File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 1017, in run_test_set
00:42:17     INFO -      self.run_test(test['filepath'], test['expected'], test['oop'])
00:42:17     INFO -    File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 994, in run_test
00:42:17     INFO -      results = runner.run(suite)
00:42:17     INFO -    File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 292, in run
00:42:17     INFO -      test(result)
00:42:17     INFO -    File "/tools/python27/lib/python2.7/unittest/suite.py", line 70, in __call__
00:42:17     INFO -      return self.run(*args, **kwds)
00:42:17     INFO -    File "/tools/python27/lib/python2.7/unittest/suite.py", line 100, in run
00:42:17     INFO -      self._handleClassSetUp(test, result)
00:42:17     INFO -    File "/tools/python27/lib/python2.7/unittest/suite.py", line 153, in _handleClassSetUp
00:42:17    ERROR -      self._addClassOrModuleLevelException(result, e, errorName)
00:42:17    ERROR -    File "/tools/python27/lib/python2.7/unittest/suite.py", line 198, in _addClassOrModuleLevelException
00:42:17     INFO -      result.addError(error, sys.exc_info())
00:42:17     INFO -    File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 133, in addError
00:42:17     INFO -      self.add_test_result(test, output=self._exc_info_to_string(err, test), result_actual='ERROR')
00:42:17     INFO -    File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 121, in add_test_result
00:42:17     INFO -      time_start=test.start_time, result_expected=result_expected,
00:42:17     INFO -  AttributeError: '_ErrorHolder' object has no attribute 'start_time'
00:42:22    ERROR -  Exception AttributeError: "'NoneType' object has no attribute 'Popen'" in <bound method Marionette.__del__ of <marionette.Marionette object at 0x10159e7d0>> ignored
00:42:22    ERROR -  Exception AttributeError: "'NoneType' object has no attribute 'Popen'" in <bound method LocalRunner.__del__ of <mozrunner.local.LocalRunner object at 0x10159eb90>> ignored
00:42:22    ERROR - Return code: 1

Comment 1

[Mark Banner (:standard8)]

I think we'll need to fix this before merging to m-c given the frequency of it so far.

Comment 2

[Mark Banner (:standard8)]

Pushed a patch to try server with some additional debug for the class setup, my guess is that something is failing there:

https://tbpl.mozilla.org/?tree=Try&rev=583ff6218170

Comment 3

[Mark Banner (:standard8)]

The problem is that the address is already in use. I've tried setting socket options, but they didn't seem to work. I think the easiest thing might be just to use a random socket, re-trying if it fails.

Comment 4

[Maire Reavy [:mreavy]]

Mark -- I know you're already taking the lead on this, so I'm just officially assigning it to you.  Thanks.

Comment 5

[Mark Banner (:standard8)]

I've just found Python's got an unused port selector option, so I'm giving that a run on try:

https://tbpl.mozilla.org/?tree=Try&rev=e9f19aed5e9f

Comment 6

[Mark Banner (:standard8)]

I've incorporated the fix into bug 994483

Comment 7

[Mark Banner (:standard8)]

https://hg.mozilla.org/projects/elm/rev/2fb7df77825f