test_emulator_order fails on the CI

RESOLVED FIXED in mozilla26

Status

Testing
Marionette
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: jgriffin, Assigned: jgriffin)

Tracking

unspecified
mozilla26
Points:
---
Bug Flags:
in-testsuite +

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Assignee)

Description

6 years ago
test_emulator_order attempts to verify that a test will fail if it ends before any pending emulator callbacks are fired.

However, on the CI, the tests run slowly enough that this situation seems to be impossible.  The test does this:

        self.assertRaises(MarionetteException,
                          self.marionette.execute_async_script,
        """runEmulatorCmd("gsm status", function(result) {});
           marionetteScriptFinished(true);
        """);

The test fails on the CI with this:

TEST-START test_emulator.py

test_emulator_cmd (test_emulator.TestEmulatorChrome) ... ok

test_emulator_order (test_emulator.TestEmulatorChrome) ... FAIL

test_emulator_cmd (test_emulator.TestEmulatorContent) ... ok

test_emulator_order (test_emulator.TestEmulatorContent) ... FAIL
ERROR

======================================================================
ERROR: test_emulator_order (test_emulator.TestEmulatorContent)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/marionette_test.py", line 96, in tearDown
    self.marionette.delete_session()
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/marionette.py", line 205, in delete_session
    response = self._send_message('deleteSession', 'ok')
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/marionette.py", line 150, in _send_message
    self._handle_error(response)
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/marionette.py", line 189, in _handle_error
    raise MarionetteException(message=message, status=status, stacktrace=stacktrace)
MarionetteException: cb is not a function
stacktrace:
	emulatorCmdResult@chrome://marionette/content/marionette-listener.js:709
	


======================================================================
FAIL: test_emulator_order (test_emulator.TestEmulatorChrome)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/tests/unit/test_emulator.py", line 22, in test_emulator_order
    """);
AssertionError: MarionetteException not raised

======================================================================
FAIL: test_emulator_order (test_emulator.TestEmulatorContent)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/jenkins/jobs/webapi-marionette-test/workspace/testing/marionette/client/marionette/tests/unit/test_emulator.py", line 22, in test_emulator_order
    """);
AssertionError: MarionetteException not raised

----------------------------------------------------------------------
Ran 4 tests in 4.673s

It looks like the test is running slowly enough on the CI VM that the callback is in fact fired before the test ends.

I'm going to disable the test for now to try and get the CI green.  I'm not sure there's any way of really fixing this, other than to add some test parameter to runEmulatorCmd that would cause the Python bits to sleep for a bit before returning.
(Assignee)

Comment 1

6 years ago
Disabled test: http://hg.mozilla.org/mozilla-central/rev/6ad95e65d74d
(Assignee)

Comment 2

4 years ago
Created attachment 786070 [details] [diff] [review]
Remove invalid test,

There's no way to guarantee that a callback won't return before marionetteScriptFinished is called, so let's delete this test.
Attachment #786070 - Flags: review?(mdas)
(Assignee)

Comment 3

4 years ago
I'm running this test on try, let's see what happens:  https://tbpl.mozilla.org/?tree=Try&rev=831b0ff3214e
(Assignee)

Comment 4

4 years ago
Created attachment 786578 [details] [diff] [review]
Enable test_emulator_order.py,

Turns out this test passes now on try, see https://tbpl.mozilla.org/?tree=Try&rev=831b0ff3214e
Attachment #786578 - Flags: review?(mdas)
(Assignee)

Updated

4 years ago
Attachment #786070 - Attachment is obsolete: true
Attachment #786070 - Flags: review?(mdas)
Attachment #786578 - Flags: review?(mdas) → review+
(Assignee)

Updated

4 years ago
Keywords: checkin-needed
https://hg.mozilla.org/integration/b2g-inbound/rev/87f584e4cbad
Assignee: nobody → jgriffin
Flags: in-testsuite+
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/87f584e4cbad
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla26
You need to log in before you can comment on or make changes to this bug.