./mach marionette-test has test failures where runtests doesnt

RESOLVED FIXED in mozilla27

Status

Testing
Marionette
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: automatedtester, Assigned: jgriffin)

Tracking

unspecified
mozilla27
x86
Mac OS X
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

4 years ago
Created attachment 815540 [details]
log output

When I run 

    ./mach marionette-test --type=browser 

I get failures but when I run 

    python runtests.py tests/unit-tests.ini --binary ../../../../obj-ff-dbg/dist/Nightly.app/Contents/MacOS/firefox --type=browser

I get 100% pass rate

I have also tried 

    ../../../../mach python runtests.py tests/unit-tests.ini --binary ../../../../obj-ff-dbg/dist/Nightly.app/Contents/MacOS/firefox --type=browser 

and got 100% pass rate. 

I have put below the test failures and have attached the log output

OK
TEST-START test_cookies.py
testAddCookie (test_cookies.CookieTest) ... ok
testAddingACookieThatExpiredInThePast (test_cookies.CookieTest) ... ok
testDeleteAllCookie (test_cookies.CookieTest) ... ok
testDeleteCookie (test_cookies.CookieTest) ... ok
testGetAllCookies (test_cookies.CookieTest) ... ok
testShouldGetCookieByName (test_cookies.CookieTest) ... ok
testShouldNotDeleteCookiesWithASimilarName (test_cookies.CookieTest) ... ok

----------------------------------------------------------------------
Ran 7 tests in 1.463s

OK
TEST-START test_window_title.py
test_get_html_title (test_window_title.TestTitle) ... ok
test_get_chrome_title (test_window_title.TestTitleChrome) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.277s

OK
TEST-START test_window_type.py
test_get_window_type (test_window_type.TestWindowTypeChrome) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.103s

OK
TEST-START test_implicit_waits.py
testShouldImplicitlyWaitForASingleElement (test_implicit_waits.TestImplicitWaits) ... ok
testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled (test_implicit_waits.TestImplicitWaits) ... FAIL

======================================================================
FAIL: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dburns/development/mozilla/mozilla-inbound/testing/marionette/client/marionette/marionette_test.py", line 132, in run
    testMethod()
  File "/Users/dburns/development/mozilla/mozilla-inbound/testing/marionette/client/marionette/tests/unit/test_implicit_waits.py", line 28, in testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled
    self.fail("Should have thrown a NoSuchElementException")
TEST-UNEXPECTED-FAIL | test_implicit_waits.py test_implicit_waits.TestImplicitWaits.testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled | AssertionError: Should have thrown a NoSuchElementException
----------------------------------------------------------------------
Ran 2 tests in 4.419s

FAILED (failures=1)
TEST-START test_date_time_value.py
test_set_date (test_date_time_value.TestDateTime) ... ok
test_set_time (test_date_time_value.TestDateTime) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.385s

OK
TEST-START test_getactiveframe_oop.py
test_active_frame_oop (test_getactiveframe_oop.TestGetActiveFrameOOP) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.881s

OK

SUMMARY
-------
passed: 153
failed: 28
todo: 0

FAILED TESTS
-------
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_async_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_async_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_async_unique_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncChrome.test_execute_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_unique_timeout
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_async_unload
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_js_exception
test_execute_async_script.py test_execute_async_script.TestExecuteAsyncContent.test_execute_permission
test_execute_script.py test_execute_script.TestExecuteChrome.test_execute_js_exception
test_execute_script.py test_execute_script.TestExecuteChrome.test_stack_trace
test_execute_script.py test_execute_script.TestExecuteContent.test_execute_js_exception
test_execute_script.py test_execute_script.TestExecuteContent.test_execute_permission
test_execute_script.py test_execute_script.TestExecuteContent.test_stack_trace
test_findelement.py test_findelement.TestElements.test_not_found
test_findelement.py test_findelement.TestElements.test_timeout
test_findelement_chrome.py test_findelement_chrome.TestElementsChrome.test_not_found
test_findelement_chrome.py test_findelement_chrome.TestElementsChrome.test_timeout
test_navigation.py test_navigation.TestNavigate.test_should_throw_a_timeoutexception_when_loading_page
test_navigation.py test_navigation.TestNavigate.test_shouldnt_error_if_nonexistent_url_used
test_timeouts.py test_timeouts.TestTimeouts.test_execute_async_timeout_settimeout
test_timeouts.py test_timeouts.TestTimeouts.test_pagetimeout_fail
test_timeouts.py test_timeouts.TestTimeouts.test_searchtimeout_found
test_timeouts.py test_timeouts.TestTimeouts.test_searchtimeout_notfound_settimeout
test_switch_frame.py test_switch_frame.TestSwitchFrame.test_stack_trace
test_switch_frame_chrome.py test_switch_frame_chrome.TestSwitchFrameChrome.test_stack_trace
test_implicit_waits.py test_implicit_waits.TestImplicitWaits.testShouldStillFailToFindAnElementWhenImplicitWaitsAreEnabled
(Assignee)

Updated

4 years ago
Assignee: nobody → jgriffin
(Assignee)

Comment 1

4 years ago
I can reproduce this on linux as well.
(Assignee)

Comment 2

4 years ago
This is some kind of Python packaging problem wrt to Marionette. :(
(Assignee)

Comment 3

4 years ago
The issue is with self.assertRaises.  Normally, Marionette exceptions appear in unittest's _AssertRaisesContext class as e.g., errors.JavascriptException, but when run in mach, it appears instead as marionette.errors.JavascriptException, which doesn't match the the type referenced by the test case.

I'm not quite sure how to resolve this elegantly.
(Assignee)

Comment 4

4 years ago
Created attachment 816061 [details] [diff] [review]
Import errors in __init__.py,

This fixes the problem for me locally; it imports errors via __init__.py so we can refer to them via the 'marionette' module both from within and outside of the package.
Attachment #816061 - Flags: review?(dburns)
(Assignee)

Comment 5

4 years ago
pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=b735145880ae
(Reporter)

Updated

4 years ago
Attachment #816061 - Flags: review?(dburns) → review+
(Assignee)

Comment 6

4 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/8d7c1598c14d
Target Milestone: --- → mozilla27
https://hg.mozilla.org/mozilla-central/rev/8d7c1598c14d
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.