Closed Bug 925469 Opened 6 years ago Closed 6 years ago

./mach marionette-test has test failures where runtests doesnt

Categories

(Testing :: Marionette, defect)

x86
macOS
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla27

People

(Reporter: automatedtester, Assigned: jgriffin)

Details

Attachments

(2 files)

Attached file log output
When I run 

    ./mach marionette-test --type=browser 

I get failures but when I run 

    python runtests.py tests/unit-tests.ini --binary ../../../../obj-ff-dbg/dist/Nightly.app/Contents/MacOS/firefox --type=browser

I get 100% pass rate

I have also tried 

    ../../../../mach python runtests.py tests/unit-tests.ini --binary ../../../../obj-ff-dbg/dist/Nightly.app/Contents/MacOS/firefox --type=browser 

and got 100% pass rate. 

I have put below the test failures and have attached the log output

OK
TEST-START test_cookies.py
testAddCookie (test_cookies.CookieTest) ... ok
testAddingACookieThatExpiredInThePast (test_cookies.CookieTest) ... ok
testDeleteAllCookie (test_cookies.CookieTest) ... ok
testDeleteCookie (test_cookies.CookieTest) ... ok
testGetAllCookies (test_cookies.CookieTest) ... ok
testShouldGetCookieByName (test_cookies.CookieTest) ... ok
testShouldNotDeleteCookiesWithASimilarName (test_cookies.CookieTest) ... ok

----------------------------------------------------------------------
Ran 7 tests in 1.463s

OK
TEST-START test_window_title.py
test_get_html_title (test_window_title.TestTitle) ... ok
test_get_chrome_title (test_window_title.TestTitleChrome) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.277s

OK
TEST-START test_window_type.py
test_get_window_type (test_window_type.TestWindowTypeChrome) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.103s

OK
TEST-START test_implicit_waits.py
testShouldImplicitlyWaitForASingleElement (test_implicit_waits.TestImplicitWaits) ... ok
testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled (test_implicit_waits.TestImplicitWaits) ... FAIL

======================================================================
FAIL: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dburns/development/mozilla/mozilla-inbound/testing/marionette/client/marionette/marionette_test.py", line 132, in run
    testMethod()
  File "/Users/dburns/development/mozilla/mozilla-inbound/testing/marionette/client/marionette/tests/unit/test_implicit_waits.py", line 28, in testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled
    self.fail("Should have thrown a NoSuchElementException")
TEST-UNEXPECTED-FAIL | test_implicit_waits.py test_implicit_waits.TestImplicitWaits.testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled | AssertionError: Should have thrown a NoSuchElementException
----------------------------------------------------------------------
Ran 2 tests in 4.419s

FAILED (failures=1)
TEST-START test_date_time_value.py
test_set_date (test_date_time_value.TestDateTime) ... ok
test_set_time (test_date_time_value.TestDateTime) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.385s

OK
TEST-START test_getactiveframe_oop.py
test_active_frame_oop (test_getactiveframe_oop.TestGetActiveFrameOOP) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.881s

OK

SUMMARY
-------
passed: 153
failed: 28
todo: 0

FAILED TESTS
-------
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_async_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_async_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_async_unique_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_unique_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_unload
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_permission
test_execute_script.py test_execute_script.TestExecuteChrome.test_execute_js_exception
test_execute_script.py test_execute_script.TestExecuteChrome.test_stack_trace
test_execute_script.py test_execute_script.TestExecuteContent.test_execute_js_exception
test_execute_script.py test_execute_script.TestExecuteContent.test_execute_permission
test_execute_script.py test_execute_script.TestExecuteContent.test_stack_trace
test_findelement.py test_findelement.TestElements.test_not_found
test_findelement.py test_findelement.TestElements.test_timeout
test_findelement_chrome.py test_findelement_chrome.TestElementsChrome.test_not_found
test_findelement_chrome.py test_findelement_chrome.TestElementsChrome.test_timeout
test_navigation.py test_navigation.TestNavigate.test_should_throw_a_timeoutexception_when_loading_page
test_navigation.py test_navigation.TestNavigate.test_shouldnt_error_if_nonexistent_url_used
test_timeouts.py test_timeouts.TestTimeouts.test_execute_async_timeout_settimeout
test_timeouts.py test_timeouts.TestTimeouts.test_pagetimeout_fail
test_timeouts.py test_timeouts.TestTimeouts.test_searchtimeout_found
test_timeouts.py test_timeouts.TestTimeouts.test_searchtimeout_notfound_settimeout
test_switch_frame.py test_switch_frame.TestSwitchFrame.test_stack_trace
test_switch_frame_chrome.py test_switch_frame_chrome.TestSwitchFrameChrome.test_stack_trace
test_implicit_waits.py test_implicit_waits.TestImplicitWaits.testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled
Assignee: nobody → jgriffin
I can reproduce this on linux as well.
This is some kind of Python packaging problem wrt to Marionette. :(
The issue is with self.assertRaises.  Normally, Marionette exceptions appear in unittest's _AssertRaisesContext class as e.g., errors.JavascriptException, but when run in mach, it appears instead as marionette.errors.JavascriptException, which doesn't match the the type referenced by the test case.

I'm not quite sure how to resolve this elegantly.
This fixes the problem for me locally; it imports errors via __init__.py so we can refer to them via the 'marionette' module both from within and outside of the package.
Attachment #816061 - Flags: review?(dburns)
Attachment #816061 - Flags: review?(dburns) → review+
You need to log in before you can comment on or make changes to this bug.