758329 - test_emulator_order fails on the CI

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Jonathan Griffin (:jgriffin)]

test_emulator_order attempts to verify that a test will fail if it ends before any pending emulator callbacks are fired.

However, on the CI, the tests run slowly enough that this situation seems to be impossible.  The test does this:

        self.assertRaises(MarionetteException,
                          self.marionette.execute_async_script,
        """runEmulatorCmd("gsm status", function(result) {});
           marionetteScriptFinished(true);
        """);

The test fails on the CI with this:

TEST-START test_emulator.py

test_emulator_cmd (test_emulator.TestEmulatorChrome) ... ok

test_emulator_order (test_emulator.TestEmulatorChrome) ... FAIL

test_emulator_cmd (test_emulator.TestEmulatorContent) ... ok

test_emulator_order (test_emulator.TestEmulatorContent) ... FAIL
ERROR

======================================================================
ERROR: test_emulator_order (test_emulator.TestEmulatorContent)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/marionette_test.py", line 96, in tearDown
    self.marionette.delete_session()
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/marionette.py", line 205, in delete_session
    response = self._send_message('deleteSession', 'ok')
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/marionette.py", line 150, in _send_message
    self._handle_error(response)
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/marionette.py", line 189, in _handle_error
    raise MarionetteException(message=message, status=status, stacktrace=stacktrace)
MarionetteException: cb is not a function
stacktrace:
	emulatorCmdResult@chrome://marionette/content/marionette-listener.js:709
	


======================================================================
FAIL: test_emulator_order (test_emulator.TestEmulatorChrome)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/tests/unit/test_emulator.py", line 22, in test_emulator_order
    """);
AssertionError: MarionetteException not raised

======================================================================
FAIL: test_emulator_order (test_emulator.TestEmulatorContent)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/tests/unit/test_emulator.py", line 22, in test_emulator_order
    """);
AssertionError: MarionetteException not raised

----------------------------------------------------------------------
Ran 4 tests in 4.673s

It looks like the test is running slowly enough on the CI VM that the callback is in fact fired before the test ends.

I'm going to disable the test for now to try and get the CI green.  I'm not sure there's any way of really fixing this, other than to add some test parameter to runEmulatorCmd that would cause the Python bits to sleep for a bit before returning.

Comment 1

[Jonathan Griffin (:jgriffin)]

Disabled test: http://hg.mozilla.org/mozilla-central/rev/6ad95e65d74d

Comment 2

[Jonathan Griffin (:jgriffin)]

Attachment: Remove invalid test,

There's no way to guarantee that a callback won't return before marionetteScriptFinished is called, so let's delete this test.

Comment 3

[Jonathan Griffin (:jgriffin)]

I'm running this test on try, let's see what happens:  https://tbpl.mozilla.org/?tree=Try&rev=831b0ff3214e

Comment 4

[Jonathan Griffin (:jgriffin)]

Attachment: Enable test_emulator_order.py,

Turns out this test passes now on try, see https://tbpl.mozilla.org/?tree=Try&rev=831b0ff3214e

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/b2g-inbound/rev/87f584e4cbad

Comment 6

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/87f584e4cbad