1297394 - Green up Marionette tests for Fennec

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Marionette tests on Fennec are scheduled since Bug 1284874. Before we set them to Tier 1 or 2, there are a few more failing tests to examine and fix or disable.

To give an idea of scope, here's the current list of failures based on  https://treeherder.mozilla.org/#/jobs?repo=try&revision=d613b73232b3bb6897487065cf886d5a756d4160:

test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_array
test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_boolean
test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_null
test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_number
test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_object
test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_string
test_execute_script.py test_execute_script.TestExecuteChrome.test_async_script_timeout
test_execute_script.py test_execute_script.TestExecuteChrome.test_exception
test_execute_script.py test_execute_script.TestExecuteChrome.test_globals
test_execute_script.py test_execute_script.TestExecuteChrome.test_invalid_chrome_handle
test_execute_script.py test_execute_script.TestExecuteChrome.test_lasting_side_effects
test_execute_script.py test_execute_script.TestExecuteChrome.test_no_callback
test_execute_script.py test_execute_script.TestExecuteChrome.test_no_return_value
test_execute_script.py test_execute_script.TestExecuteChrome.test_permission
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_array
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_boolean
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_number
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_object
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_string
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_web_element
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_web_element_array
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_web_element_nodelist
test_execute_script.py test_execute_script.TestExecuteChrome.test_sandbox_refresh_arguments
test_execute_script.py test_execute_script.TestExecuteChrome.test_sandbox_reuse
test_execute_script.py test_execute_script.TestExecuteChrome.test_stacktrace
test_execute_script.py test_execute_script.TestExecuteChrome.test_system_dead_object
test_execute_script.py test_execute_script.TestExecuteChrome.test_system_globals
test_execute_script.py test_execute_script.TestExecuteChrome.test_system_sandbox_wrappedjsobject
test_execute_script.py test_execute_script.TestExecuteChrome.test_unmarshal_element_collection
test_execute_script.py test_execute_script.TestExecuteChrome.test_wrappedjsobject
test_switch_frame.py test_switch_frame.TestSwitchFrame.test_should_be_able_to_switch_to_parent_from_second_level
 test_navigation.py test_navigation.TestNavigate.test_navigate_chrome_error
 test_navigation.py test_navigation.TestNavigate.test_should_not_error_if_nonexistent_url_used
 test_text.py test_text.TestText.test_should_report_key_code_of_arrow_keys_up_down_events
test_text.py test_text.TestText.testShouldTypeAnInteger
test_screenshot.py test_screenshot.Chrome.test_window
test_screenshot.py test_screenshot.Content.test_html_document_element
test_import_script.py test_import_script.TestImportScriptContent.test_imports_apply_globally
 test_typing.py test_typing.TestTyping.testShouldReportKeyCodeOfArrowKeysUpDownEvents
 test_typing.py test_typing.TestTyping.testCutAndPasteShortcuts
 test_typing.py test_typing.TestTyping.testWillSimulateAKeyPressWhenEnteringTextIntoTextAreas
 test_shadow_dom.py test_shadow_dom.TestShadowDom.test_shadow_dom
 test_click_scrolling.py test_click_scrolling.TestClickScrolling.test_should_not_scroll_elements_if_click_point_is_in_view
 test_addons.py test_addons.TestAddons.test_install_and_remove_temporary_unsigned_addon
 test_about_pages.py test_about_pages.TestAboutPages.test_back_forward
 test_about_pages.py test_about_pages.TestAboutPages.test_hang
 test_about_pages.py test_about_pages.TestAboutPages.test_navigate_non_remote_about_pages
 test_about_pages.py test_about_pages.TestAboutPages.test_navigate_shortcut_key
 test_about_pages.py test_about_pages.TestAboutPages.test_type_to_non_remote_tab
 test_about_pages.py test_about_pages.TestAboutPages.test_type_to_remote_tab
 test_window_type.py test_window_type.TestWindowTypeChrome.test_get_window_type
 test_profile_management.py test_profile_management.TestLog.test_in_app_restart_the_browser

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

David, do you have a sense of which of these tests should be fixed vs skipped on mobile?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So we do not have a chrome UI for Fennec given that all is powered by Java. So we should most likely skip all of those tests first, which try to interact in any way with the chrome UI. Not sure about all the test_execute_script.TestExecuteChrome.* failures which seem to be run at API level and should still work.

Comment 3

[David Burns :automatedtester]

Anything that is chrome related. For the most part we probably have some wrong assumptions on when in Chrome context.

Comment 4

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

A few other tests fail intermittenly:
TEST-UNEXPECTED-ERROR | test_visibility.py TestVisibility.testShouldModifyTheVisibilityOfAnElementDynamically | TimeoutException: Error loading page, timed out (checkLoad)
TEST-UNEXPECTED-ERROR | test_visibility.py TestVisibility.testShouldSayElementIsInvisibleWhenOverflowXIsHiddenAndOutOfViewport | TimeoutException: Error loading page, timed out (onDOMContentLoaded)
TEST-UNEXPECTED-ERROR | test_shadow_dom.py TestShadowDom.test_shadow_dom | TimeoutException: Error loading page, timed out (checkLoad)
TEST-UNEXPECTED-ERROR | test_shadow_dom.py TestShadowDom.test_shadow_dom_after_switch_away_from_shadow_root | TimeoutException: Error loading page, timed out (onDOMContentLoaded)

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Lots of tests which fail with "onDOMContentLoaded" happen most likely because tests in setUp() we try to load about:blank, while this is our default start page. So it will fail with Marionette right now.

We should use another page like about: to actually get rid of this problem. I will do something like that for bug 1156427 and test_navigate(). Lets see if that helps here.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Since my patch on bug 1156427 landed the two onDOMContentLoaded failures are gone now for test_navigation. At least a small step forward...

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Lots of tests are using ui elements which do not exist in Fennec:

https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=mn&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=3510219

We should simply mark them as desktop only.

Comment 8

[Nick Alexander :nalexander [he/him]]

myk, snorp, jchen: just letting you know about this ticket and the chain of tickets adding Marionette suppor to Fennec around it.  I think the best way to test interacting with the contents of GeckoView will be to run something like Selendroid against a GeckoView test Activity, connecting to Marionette instead of the bespoke Android WebView like Selendroid does.  Some of this probably already works, but hammering out details will be ... interesting.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Please note that I will reorganize the Marionette unit tests in the next couple of days via bug 1320632. With that I will identify chrome tests which we currently run for Fennec and which needs to be disabled for it. It will lower the number of failures we see on Treeherder.

Comment 10

[Myk Melez [:myk] [@mykmelez]]

(In reply to Nick Alexander :nalexander (leave until January 2017) from comment #8)
> myk, snorp, jchen: just letting you know about this ticket and the chain of
> tickets adding Marionette suppor to Fennec around it.

Thanks, Nick, for the heads-up! It sounds like don't need any actual info from me at this point, so I'm removing that flag, and I'll stay in the loop by being cc:ed on the bug.  Please re-add the flag if you do want some info from me, however.  (FWIW, your plan to drive a GeckoView test activity via Selendroid connecting to Marionette sounds reasonable in theory, although I'm not familiar enough with either project to speak to the devilish details.)

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Quick update on my latest fixes and skip patches... As of now we have 4/10 chunks green. With a fix for bug 1322993 we will get in an even much better shape.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Quick update here... with all the fixed I did over the last couple of weeks, we now have 6/10 chunks (later today maybe even 7/10) green. Only a couple of failures are remaining until we can decide bumping up the tier of Mn tests for Fennec.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Lets make it a tracking bug for greening up the tests. I will file a follow-up bug to actually bump to Tier-2 as next step.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Btw. with the following push we no longer have permanent test failures for Marionette jobs:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=cef9cfc92fec2c03a4780601e1c35c47e19ddd86

There are still some random oranges where Marionette kills the application due to no responses in navigate()/get(). I hope to address this completely with bug 1322277. Once this is done we can consider bumping up the tier level.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The last major intermittent failure has been fixed and got landed on autoland earlier today. Since then our tests are green. Once we merged the code to m-c I will request the bump to tier-2 on Treeherder.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I don't think we need this bug anytime longer. There are still intermittent failures left to fix but those are low volume. Also other work in re-enabling tests with possible harness changes can be done on individual bugs.

I will work on uplifting all remaining fixes to aurora and beta, so we can enable tier-2 also for the upcoming ESR release.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It got all uplifted to aurora and beta.