marionette-test mach target is broken

RESOLVED FIXED in mozilla35

Status

defect
P1
normal
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: jgraham, Assigned: chmanchester)

Tracking

({pi-marionette-runner})

unspecified
mozilla35
x86_64
Linux
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

In several ways; let me know if this needs to be several bugs.

Initially I get errors like:
 0:06.46 LOG: MainThread ERROR Got test_start message before suite_start for test test_text_chrome.py
 0:06.46 LOG: MainThread ERROR test_end for test_text_chrome.py logged while not in progress. Logged with data: {"status": "SKIP", "message": "\"Bug 896046\"", "test": "test_text_chrome.py"}

Which suggests that a suite_start log action is missing.

Later some tests start running and I get errors like:

Traceback (most recent call last):
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette_test.py", line 171, in run
    testMethod()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/tests/unit/test_import_script.py", line 86, in test_import_script_and_clear
    self.assertRaises(JavascriptException, self.marionette.execute_script, "return testFunc();")
  File "/usr/lib/python2.7/unittest/case.py", line 475, in assertRaises
    callableObj(*args, **kwargs)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 1199, in execute_script
    filename=os.path.basename(frame[0]))
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 634, in _send_message
    self._handle_error(response)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 682, in _handle_error
    raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: JavascriptException: ReferenceError: testFunc is not defined
stacktrace:
	execute_script @case.py, line 475
	inline javascript, line 0
	src: "let __marionetteFunc = function(){return testFunc();};__marionetteFunc.apply(null, __marionetteParams);"

 0:10.14 TEST_START: MainThread test_click.py TestClick.testClickingALinkMadeUpOfNumbersIsHandledCorrectly
 0:10.44 TEST_END: MainThread ERROR, expected PASS
Traceback (most recent call last):
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette_test.py", line 171, in run
    testMethod()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/tests/unit/test_click.py", line 24, in testClickingALinkMadeUpOfNumbersIsHandledCorrectly
    lambda m: m.find_element(By.ID, 'username'))
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/wait.py", line 122, in until
    rv = condition(self.marionette)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/tests/unit/test_click.py", line 24, in <lambda>
    lambda m: m.find_element(By.ID, 'username'))
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 1274, in find_element
    response = self._send_message('findElement', 'value', **kwargs)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 634, in _send_message
    self._handle_error(response)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 668, in _handle_error
    raise errors.NoSuchElementException(message=message, status=status, stacktrace=stacktrace)
NoSuchElementException: NoSuchElementException: Unable to locate element: username

and a bunch more similar things.

Finally the whole harness dies with:
MarionetteException: MarionetteException: Could not delete session
stacktrace:
	TypeError: this.browser.removeTab is not a function



  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/mach_commands.py", line 124, in run_marionette_test
    topsrcdir=self.topsrcdir, address=address)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/mach_commands.py", line 70, in run_marionette
    runner = startTestRunner(MarionetteTestRunner, options, tests)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runtests.py", line 41, in startTestRunner
    runner.run_tests(tests)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runner/base.py", line 664, in run_tests
    self.run_test_sets()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runner/base.py", line 887, in run_test_sets
    self.run_test_set(in_process_tests)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runner/base.py", line 864, in run_test_set
    self.run_test(test['filepath'], test['expected'], test['oop'])
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runner/base.py", line 840, in run_test
    results = runner.run(suite)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/runner/base.py", line 230, in run
    result = super(MarionetteTextTestRunner, self).run(test)
  File "/home/jgraham/develop/gecko-dev-1/testing/mozbase/moztest/moztest/adapters/unit.py", line 133, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
    return self.run(*args, **kwds)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette_test.py", line 208, in run
    self.cleanTest()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette_test.py", line 294, in cleanTest
    self._deleteSession()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette_test.py", line 306, in _deleteSession
    self.marionette.delete_session()
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 826, in delete_session
    response = self._send_message('deleteSession', 'ok')
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 634, in _send_message
    self._handle_error(response)
  File "/home/jgraham/develop/gecko-dev-1/testing/marionette/client/marionette/marionette.py", line 708, in _handle_error
    raise errors.MarionetteException(message=message, status=status, stacktrace=stacktrace)
(In reply to James Graham [:jgraham] from comment #0)
> In several ways; let me know if this needs to be several bugs.
> 
> Initially I get errors like:
>  0:06.46 LOG: MainThread ERROR Got test_start message before suite_start for
> test test_text_chrome.py
>  0:06.46 LOG: MainThread ERROR test_end for test_text_chrome.py logged while
> not in progress. Logged with data: {"status": "SKIP", "message": "\"Bug
> 896046\"", "test": "test_text_chrome.py"}
> 
> Which suggests that a suite_start log action is missing.
> 

This seems to show ERROR whenever a test has been skipped.
The removeTab issue is in bug 1018904
This should fix the first issue. I logged all the skips before suite_start, which is an error since bug 1036427.
Attachment #8469637 - Flags: review?(james)
Assignee

Updated

5 years ago
Assignee: nobody → cmanchester
Status: NEW → ASSIGNED
Assignee

Updated

5 years ago
Assignee: cmanchester → nobody
Status: ASSIGNED → NEW
Attachment #8469637 - Flags: review?(james) → review+
Assignee

Updated

5 years ago
Keywords: leave-open
I set leave-open so this wouldn't be closed when my patch got merged to central, because it only addresses the first issue in the bug report. As far as I know the "testFunc is not defined" error is still present.
Collusion between mach's effects on sys.path and marionette's folder structure means that when assertRaises checks exception types it ends up looking at different copies of the module, gets a mismatch, reraises, and reports a failure. This fixes the issue for me locally. Is there a better way?
Attachment #8502627 - Flags: review?(gps)
Assignee

Updated

5 years ago
Assignee: nobody → cmanchester
Status: NEW → ASSIGNED
Comment on attachment 8502627 [details] [diff] [review]
Modify imports in python marionette test files to avoid a mismatch in assertRaises when running through mach.

This commit message needs to be updated, but you get the idea.
Comment on attachment 8502627 [details] [diff] [review]
Modify imports in python marionette test files to avoid a mismatch in assertRaises when running through mach.

Review of attachment 8502627 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/marionette/mach_commands.py
@@ +37,5 @@
>  
>  def run_marionette(tests, b2g_path=None, emulator=None, testtype=None,
>      address=None, binary=None, topsrcdir=None, **kwargs):
> +
> +    # See bug 1050511.

Nit: I always prefer expanding the reason in-line so others don't have to look at yet-another-system to understand code.
Attachment #8502627 - Flags: review?(gps) → review+
Assignee

Updated

5 years ago
Attachment #8502627 - Attachment is obsolete: true
Assignee

Updated

5 years ago
Attachment #8503276 - Flags: review+
Setting checkin-needed without a try run because this is npotb.
https://hg.mozilla.org/mozilla-central/rev/51c57f6ffd34
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla35
Assignee

Updated

5 years ago
Depends on: 1082196
You need to log in before you can comment on or make changes to this bug.