804549 - [B2G] Gaiatest runner test concurrency does not work

Status: RESOLVED WONTFIX

(Remote Protocol :: Marionette, defect)

Description

[Zac C (:zac)]

When attempting to queue two tests to run on the same device the first test will run successfully and the second will wait patiently.

However once the first test is completed and the setUp for the second test is started it falls down. A 500 response is sent from the device.

I've attached a stacktrace for the second test.



----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/zacdev/Mozilla/gaia-ui-tests/gaiatest/gaia_test.py", line 77, in setUp
    MarionetteTestCase.setUp(self)
  File "/home/zacdev/.virtualenvs/gaia-ui-tests/local/lib/python2.7/site-packages/marionette_client-0.5.1-py2.7.egg/marionette/marionette_test.py", line 116, in setUp
    CommonTestCase.setUp(self)
  File "/home/zacdev/.virtualenvs/gaia-ui-tests/local/lib/python2.7/site-packages/marionette_client-0.5.1-py2.7.egg/marionette/marionette_test.py", line 78, in setUp
    self.marionette.start_session()
  File "/home/zacdev/.virtualenvs/gaia-ui-tests/local/lib/python2.7/site-packages/marionette_client-0.5.1-py2.7.egg/marionette/marionette.py", line 251, in start_session
    self.session = self._send_message('newSession', 'value')
  File "/home/zacdev/.virtualenvs/gaia-ui-tests/local/lib/python2.7/site-packages/marionette_client-0.5.1-py2.7.egg/marionette/marionette.py", line 183, in _send_message
    self._handle_error(response)
  File "/home/zacdev/.virtualenvs/gaia-ui-tests/local/lib/python2.7/site-packages/marionette_client-0.5.1-py2.7.egg/marionette/marionette.py", line 241, in _handle_error
    raise MarionetteException(message=response, status=500)
TEST-UNEXPECTED-FAIL : MarionetteException: {u'ok': True, u'from': u'conn3.marionette1'}

Comment 1

[Zac C (:zac)]

This is different to https://bugzilla.mozilla.org/show_bug.cgi?id=800884 because the first test passed.

Comment 2

[Jonathan Griffin (:jgriffin)]

Zac, does this happen for any two tests?  Or does it require specific tests?  Does it matter if test_calculator.py is the first test run?

Comment 3

[Zac C (:zac)]

:jgriffin I just had another play around with this and it didn't seem to matter which order the tests were run in.

I tested with test_camera.py and test_calculator.py and just generally they would not queue politely. test_camera.py has two tests in it and if the first test was running the two queueing tests would then attempt to run at the same time, bang heads together and then sit in the corner and cry.

Sometimes even starting the second test would interrupt the existing test/session.

Comment 4

[Jonathan Griffin (:jgriffin)]

Just to confirm, you're queueing tests using this pattern, correct?

python runtests.py [args] test1.py test2.py test3.py

Comment 5

[Zac C (:zac)]

No I'm just running from two separate consoles.

Comment 6

[Jonathan Griffin (:jgriffin)]

That isn't supported, and it isn't at all a high priority to add support for.  The supported way to queue tests is just to specify multiple tests on a single invocation of the command-line.

You can also specify a dir (in which case it will run all the tests in that dir) or a manifest, see http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit-tests.ini for an example.

I'm going to mark this as WONTFIX, since I don't think this blocks anything.  Reopen if you disagree.

Comment 7

[Zac C (:zac)]

Well if we have an array of devices (or even one as shown here) in our CI then more than one source might try to target the devices.

There would need to be some load balance/distribution medium somewhere along the line.

Comment 8

[Jonathan Griffin (:jgriffin)]

So for now that situation will have to be controlled by smart use of the CI.  If you have multiple Jenkins jobs that might target the same device, you can use a lock for example to keep them from trampling each other.

Comment 9

[Zac C (:zac)]

OK we can manage that. I'll let the higher-ups decide if it's still a blocker.