925469 - ./mach marionette-test has test failures where runtests doesnt

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[David Burns :automatedtester]

Attachment: log output

When I run 

    ./mach marionette-test --type=browser 

I get failures but when I run 

    python runtests.py tests/unit-tests.ini --binary ../../../../obj-ff-dbg/dist/Nightly.app/Contents/MacOS/firefox --type=browser

I get 100% pass rate

I have also tried 

    ../../../../mach python runtests.py tests/unit-tests.ini --binary ../../../../obj-ff-dbg/dist/Nightly.app/Contents/MacOS/firefox --type=browser 

and got 100% pass rate. 

I have put below the test failures and have attached the log output

OK
TEST-START test_cookies.py
testAddCookie (test_cookies.CookieTest) ... ok
testAddingACookieThatExpiredInThePast (test_cookies.CookieTest) ... ok
testDeleteAllCookie (test_cookies.CookieTest) ... ok
testDeleteCookie (test_cookies.CookieTest) ... ok
testGetAllCookies (test_cookies.CookieTest) ... ok
testShouldGetCookieByName (test_cookies.CookieTest) ... ok
testShouldNotDeleteCookiesWithASimilarName (test_cookies.CookieTest) ... ok

----------------------------------------------------------------------
Ran 7 tests in 1.463s

OK
TEST-START test_window_title.py
test_get_html_title (test_window_title.TestTitle) ... ok
test_get_chrome_title (test_window_title.TestTitleChrome) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.277s

OK
TEST-START test_window_type.py
test_get_window_type (test_window_type.TestWindowTypeChrome) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.103s

OK
TEST-START test_implicit_waits.py
testShouldImplicitlyWaitForASingleElement (test_implicit_waits.TestImplicitWaits) ... ok
testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled (test_implicit_waits.TestImplicitWaits) ... FAIL

======================================================================
FAIL: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dburns/development/mozilla/mozilla-inbound/testing/marionette/client/marionette/marionette_test.py", line 132, in run
    testMethod()
  File "/Users/dburns/development/mozilla/mozilla-inbound/testing/marionette/client/marionette/tests/unit/test_implicit_waits.py", line 28, in testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled
    self.fail("Should have thrown a NoSuchElementException")
TEST-UNEXPECTED-FAIL | test_implicit_waits.py test_implicit_waits.TestImplicitWaits.testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled | AssertionError: Should have thrown a NoSuchElementException
----------------------------------------------------------------------
Ran 2 tests in 4.419s

FAILED (failures=1)
TEST-START test_date_time_value.py
test_set_date (test_date_time_value.TestDateTime) ... ok
test_set_time (test_date_time_value.TestDateTime) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.385s

OK
TEST-START test_getactiveframe_oop.py
test_active_frame_oop (test_getactiveframe_oop.TestGetActiveFrameOOP) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.881s

OK

SUMMARY
-------
passed: 153
failed: 28
todo: 0

FAILED TESTS
-------
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_async_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_async_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_async_unique_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_unique_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_unload
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_permission
test_execute_script.py test_execute_script.TestExecuteChrome.test_execute_js_exception
test_execute_script.py test_execute_script.TestExecuteChrome.test_stack_trace
test_execute_script.py test_execute_script.TestExecuteContent.test_execute_js_exception
test_execute_script.py test_execute_script.TestExecuteContent.test_execute_permission
test_execute_script.py test_execute_script.TestExecuteContent.test_stack_trace
test_findelement.py test_findelement.TestElements.test_not_found
test_findelement.py test_findelement.TestElements.test_timeout
test_findelement_chrome.py test_findelement_chrome.TestElementsChrome.test_not_found
test_findelement_chrome.py test_findelement_chrome.TestElementsChrome.test_timeout
test_navigation.py test_navigation.TestNavigate.test_should_throw_a_timeoutexception_when_loading_page
test_navigation.py test_navigation.TestNavigate.test_shouldnt_error_if_nonexistent_url_used
test_timeouts.py test_timeouts.TestTimeouts.test_execute_async_timeout_settimeout
test_timeouts.py test_timeouts.TestTimeouts.test_pagetimeout_fail
test_timeouts.py test_timeouts.TestTimeouts.test_searchtimeout_found
test_timeouts.py test_timeouts.TestTimeouts.test_searchtimeout_notfound_settimeout
test_switch_frame.py test_switch_frame.TestSwitchFrame.test_stack_trace
test_switch_frame_chrome.py test_switch_frame_chrome.TestSwitchFrameChrome.test_stack_trace
test_implicit_waits.py test_implicit_waits.TestImplicitWaits.testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled

Comment 1

[Jonathan Griffin (:jgriffin)]

I can reproduce this on linux as well.

Comment 2

[Jonathan Griffin (:jgriffin)]

This is some kind of Python packaging problem wrt to Marionette. :(

Comment 3

[Jonathan Griffin (:jgriffin)]

The issue is with self.assertRaises.  Normally, Marionette exceptions appear in unittest's _AssertRaisesContext class as e.g., errors.JavascriptException, but when run in mach, it appears instead as marionette.errors.JavascriptException, which doesn't match the the type referenced by the test case.

I'm not quite sure how to resolve this elegantly.

Comment 4

[Jonathan Griffin (:jgriffin)]

Attachment: Import errors in __init__.py,

This fixes the problem for me locally; it imports errors via __init__.py so we can refer to them via the 'marionette' module both from within and outside of the package.

Comment 5

[Jonathan Griffin (:jgriffin)]

pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=b735145880ae

Comment 6

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8d7c1598c14d

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/8d7c1598c14d