1050511 - marionette-test mach target is broken

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect, P1)

Description

[James Graham [:jgraham]]

In several ways; let me know if this needs to be several bugs.

Initially I get errors like:
 0:06.46 LOG: MainThread ERROR Got test_start message before suite_start for test test_text_chrome.py
 0:06.46 LOG: MainThread ERROR test_end for test_text_chrome.py logged while not in progress. Logged with data: {"status": "SKIP", "message": "\"Bug 896046\"", "test": "test_text_chrome.py"}

Which suggests that a suite_start log action is missing.

Later some tests start running and I get errors like:

Traceback (most recent call last):
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette_test.py", line 171, in run
    testMethod()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/tests/unit/test_import_script.py", line 86, in test_import_script_and_clear
    self.assertRaises(JavascriptException, self.marionette.execute_script, "return testFunc();")
  File "/usr/lib/python2.7/unittest/case.py", line 475, in assertRaises
    callableObj(*args, **kwargs)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 1199, in execute_script
    filename=os.path.basename(frame[0]))
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 634, in _send_message
    self._handle_error(response)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 682, in _handle_error
    raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: JavascriptException: ReferenceError: testFunc is not defined
stacktrace:
	execute_script @case.py, line 475
	inline javascript, line 0
	src: "let __marionetteFunc = function(){return testFunc();};__marionetteFunc.apply(null, __marionetteParams);"

 0:10.14 TEST_START: MainThread test_click.py TestClick.testClickingALinkMadeUpOfNumbersIsHandledCorrectly
 0:10.44 TEST_END: MainThread ERROR, expected PASS
Traceback (most recent call last):
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette_test.py", line 171, in run
    testMethod()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/tests/unit/test_click.py", line 24, in testClickingALinkMadeUpOfNumbersIsHandledCorrectly
    lambda m: m.find_element(By.ID, 'username'))
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/wait.py", line 122, in until
    rv = condition(self.marionette)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/tests/unit/test_click.py", line 24, in <lambda>
    lambda m: m.find_element(By.ID, 'username'))
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 1274, in find_element
    response = self._send_message('findElement', 'value', **kwargs)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 634, in _send_message
    self._handle_error(response)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 668, in _handle_error
    raise errors.NoSuchElementException(message=message, status=status, stacktrace=stacktrace)
NoSuchElementException: NoSuchElementException: Unable to locate element: username

and a bunch more similar things.

Finally the whole harness dies with:
MarionetteException: MarionetteException: Could not delete session
stacktrace:
	TypeError: this.browser.removeTab is not a function



  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/mach_commands.py", line 124, in run_marionette_test
    topsrcdir=self.topsrcdir, address=address)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/mach_commands.py", line 70, in run_marionette
    runner = startTestRunner(MarionetteTestRunner, options, tests)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runtests.py", line 41, in startTestRunner
    runner.run_tests(tests)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runner/base.py", line 664, in run_tests
    self.run_test_sets()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runner/base.py", line 887, in run_test_sets
    self.run_test_set(in_process_tests)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runner/base.py", line 864, in run_test_set
    self.run_test(test['filepath'], test['expected'], test['oop'])
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runner/base.py", line 840, in run_test
    results = runner.run(suite)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runner/base.py", line 230, in run
    result = super(MarionetteTextTestRunner, self).run(test)
  File "/home/jgraham/develop/gecko-dev-1/testing/mozbase/moztest/moztest/adapters/unit.py", line 133, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
    return self.run(*args, **kwds)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette_test.py", line 208, in run
    self.cleanTest()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette_test.py", line 294, in cleanTest
    self._deleteSession()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette_test.py", line 306, in _deleteSession
    self.marionette.delete_session()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 826, in delete_session
    response = self._send_message('deleteSession', 'ok')
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 634, in _send_message
    self._handle_error(response)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 708, in _handle_error
    raise errors.MarionetteException(message=message, status=status, stacktrace=stacktrace)

Comment 1

[David Burns :automatedtester]

(In reply to James Graham [:jgraham] from comment #0)
> In several ways; let me know if this needs to be several bugs.
> 
> Initially I get errors like:
>  0:06.46 LOG: MainThread ERROR Got test_start message before suite_start for
> test test_text_chrome.py
>  0:06.46 LOG: MainThread ERROR test_end for test_text_chrome.py logged while
> not in progress. Logged with data: {"status": "SKIP", "message": "\"Bug
> 896046\"", "test": "test_text_chrome.py"}
> 
> Which suggests that a suite_start log action is missing.
> 

This seems to show ERROR whenever a test has been skipped.

Comment 2

[David Burns :automatedtester]

The removeTab issue is in bug 1018904

Comment 3

[Chris Manchester (limited bugmail, email directly)]

Attachment: Log skipped tests in a marionette run only after logging suite_start.

This should fix the first issue. I logged all the skips before suite_start, which is an error since bug 1036427.

Comment 4

[James Graham [:jgraham]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c5531819433c

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/c5531819433c

Comment 6

[Chris Manchester (limited bugmail, email directly)]

I set leave-open so this wouldn't be closed when my patch got merged to central, because it only addresses the first issue in the bug report. As far as I know the "testFunc is not defined" error is still present.

Comment 7

[Chris Manchester (limited bugmail, email directly)]

Attachment: Modify imports in python marionette test files to avoid a mismatch in assertRaises when running through mach.

Collusion between mach's effects on sys.path and marionette's folder structure means that when assertRaises checks exception types it ends up looking at different copies of the module, gets a mismatch, reraises, and reports a failure. This fixes the issue for me locally. Is there a better way?

Comment 8

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8502627 [details] [diff] [review]
Modify imports in python marionette test files to avoid a mismatch in assertRaises when running through mach.

This commit message needs to be updated, but you get the idea.

Comment 9

[Gregory Szorc [:gps]]

Comment on attachment 8502627 [details] [diff] [review]
Modify imports in python marionette test files to avoid a mismatch in assertRaises when running through mach.

Review of attachment 8502627 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/marionette/mach_commands.py
@@ +37,5 @@
>  
>  def run_marionette(tests, b2g_path=None, emulator=None, testtype=None,
>      address=None, binary=None, topsrcdir=None, **kwargs):
> +
> +    # See bug 1050511.

Nit: I always prefer expanding the reason in-line so others don't have to look at yet-another-system to understand code.

Comment 10

[Chris Manchester (limited bugmail, email directly)]

Attachment: Modify import of marionette harness to avoid a mismatch in assertRaises when running through mach.

Comment 11

[Chris Manchester (limited bugmail, email directly)]

Setting checkin-needed without a try run because this is npotb.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/51c57f6ffd34

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/51c57f6ffd34