Closed Bug 1297394 Opened 3 years ago Closed 3 years ago

Green up Marionette tests for Fennec

Categories

(Testing :: Marionette, defect)

defect
Not set

Tracking

(firefox52 fixed, firefox53 fixed, firefox54 fixed)

RESOLVED FIXED
mozilla54
Tracking Status
firefox52 --- fixed
firefox53 --- fixed
firefox54 --- fixed

People

(Reporter: maja_zf, Assigned: whimboo)

References

Details

Marionette tests on Fennec are scheduled since Bug 1284874. Before we set them to Tier 1 or 2, there are a few more failing tests to examine and fix or disable.

To give an idea of scope, here's the current list of failures based on  https://treeherder.mozilla.org/#/jobs?repo=try&revision=d613b73232b3bb6897487065cf886d5a756d4160:

test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_array
test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_boolean
test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_null
test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_number
test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_object
test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_string
test_execute_script.py test_execute_script.TestExecuteChrome.test_async_script_timeout
test_execute_script.py test_execute_script.TestExecuteChrome.test_exception
test_execute_script.py test_execute_script.TestExecuteChrome.test_globals
test_execute_script.py test_execute_script.TestExecuteChrome.test_invalid_chrome_handle
test_execute_script.py test_execute_script.TestExecuteChrome.test_lasting_side_effects
test_execute_script.py test_execute_script.TestExecuteChrome.test_no_callback
test_execute_script.py test_execute_script.TestExecuteChrome.test_no_return_value
test_execute_script.py test_execute_script.TestExecuteChrome.test_permission
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_array
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_boolean
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_number
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_object
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_string
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_web_element
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_web_element_array
test_execute_script.py test_execute_script.TestExecuteChrome.test_return_web_element_nodelist
test_execute_script.py test_execute_script.TestExecuteChrome.test_sandbox_refresh_arguments
test_execute_script.py test_execute_script.TestExecuteChrome.test_sandbox_reuse
test_execute_script.py test_execute_script.TestExecuteChrome.test_stacktrace
test_execute_script.py test_execute_script.TestExecuteChrome.test_system_dead_object
test_execute_script.py test_execute_script.TestExecuteChrome.test_system_globals
test_execute_script.py test_execute_script.TestExecuteChrome.test_system_sandbox_wrappedjsobject
test_execute_script.py test_execute_script.TestExecuteChrome.test_unmarshal_element_collection
test_execute_script.py test_execute_script.TestExecuteChrome.test_wrappedjsobject
test_switch_frame.py test_switch_frame.TestSwitchFrame.test_should_be_able_to_switch_to_parent_from_second_level
 test_navigation.py test_navigation.TestNavigate.test_navigate_chrome_error
 test_navigation.py test_navigation.TestNavigate.test_should_not_error_if_nonexistent_url_used
 test_text.py test_text.TestText.test_should_report_key_code_of_arrow_keys_up_down_events
test_text.py test_text.TestText.testShouldTypeAnInteger
test_screenshot.py test_screenshot.Chrome.test_window
test_screenshot.py test_screenshot.Content.test_html_document_element
test_import_script.py test_import_script.TestImportScriptContent.test_imports_apply_globally
 test_typing.py test_typing.TestTyping.testShouldReportKeyCodeOfArrowKeysUpDownEvents
 test_typing.py test_typing.TestTyping.testCutAndPasteShortcuts
 test_typing.py test_typing.TestTyping.testWillSimulateAKeyPressWhenEnteringTextIntoTextAreas
 test_shadow_dom.py test_shadow_dom.TestShadowDom.test_shadow_dom
 test_click_scrolling.py test_click_scrolling.TestClickScrolling.test_should_not_scroll_elements_if_click_point_is_in_view
 test_addons.py test_addons.TestAddons.test_install_and_remove_temporary_unsigned_addon
 test_about_pages.py test_about_pages.TestAboutPages.test_back_forward
 test_about_pages.py test_about_pages.TestAboutPages.test_hang
 test_about_pages.py test_about_pages.TestAboutPages.test_navigate_non_remote_about_pages
 test_about_pages.py test_about_pages.TestAboutPages.test_navigate_shortcut_key
 test_about_pages.py test_about_pages.TestAboutPages.test_type_to_non_remote_tab
 test_about_pages.py test_about_pages.TestAboutPages.test_type_to_remote_tab
 test_window_type.py test_window_type.TestWindowTypeChrome.test_get_window_type
 test_profile_management.py test_profile_management.TestLog.test_in_app_restart_the_browser
David, do you have a sense of which of these tests should be fixed vs skipped on mobile?
Flags: needinfo?(dburns)
So we do not have a chrome UI for Fennec given that all is powered by Java. So we should most likely skip all of those tests first, which try to interact in any way with the chrome UI. Not sure about all the test_execute_script.TestExecuteChrome.* failures which seem to be run at API level and should still work.
Anything that is chrome related. For the most part we probably have some wrong assumptions on when in Chrome context.
Flags: needinfo?(dburns)
A few other tests fail intermittenly:
TEST-UNEXPECTED-ERROR | test_visibility.py TestVisibility.testShouldModifyTheVisibilityOfAnElementDynamically | TimeoutException: Error loading page, timed out (checkLoad)
TEST-UNEXPECTED-ERROR | test_visibility.py TestVisibility.testShouldSayElementIsInvisibleWhenOverflowXIsHiddenAndOutOfViewport | TimeoutException: Error loading page, timed out (onDOMContentLoaded)
TEST-UNEXPECTED-ERROR | test_shadow_dom.py TestShadowDom.test_shadow_dom | TimeoutException: Error loading page, timed out (checkLoad)
TEST-UNEXPECTED-ERROR | test_shadow_dom.py TestShadowDom.test_shadow_dom_after_switch_away_from_shadow_root | TimeoutException: Error loading page, timed out (onDOMContentLoaded)
Lots of tests which fail with "onDOMContentLoaded" happen most likely because tests in setUp() we try to load about:blank, while this is our default start page. So it will fail with Marionette right now.

We should use another page like about: to actually get rid of this problem. I will do something like that for bug 1156427 and test_navigate(). Lets see if that helps here.
Depends on: 1156427
Since my patch on bug 1156427 landed the two onDOMContentLoaded failures are gone now for test_navigation. At least a small step forward...
Lots of tests are using ui elements which do not exist in Fennec:

https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=mn&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=3510219

We should simply mark them as desktop only.
myk, snorp, jchen: just letting you know about this ticket and the chain of tickets adding Marionette suppor to Fennec around it.  I think the best way to test interacting with the contents of GeckoView will be to run something like Selendroid against a GeckoView test Activity, connecting to Marionette instead of the bespoke Android WebView like Selendroid does.  Some of this probably already works, but hammering out details will be ... interesting.
Flags: needinfo?(snorp)
Flags: needinfo?(nchen)
Flags: needinfo?(myk)
Please note that I will reorganize the Marionette unit tests in the next couple of days via bug 1320632. With that I will identify chrome tests which we currently run for Fennec and which needs to be disabled for it. It will lower the number of failures we see on Treeherder.
Depends on: 1320632
(In reply to Nick Alexander :nalexander (leave until January 2017) from comment #8)
> myk, snorp, jchen: just letting you know about this ticket and the chain of
> tickets adding Marionette suppor to Fennec around it.

Thanks, Nick, for the heads-up! It sounds like don't need any actual info from me at this point, so I'm removing that flag, and I'll stay in the loop by being cc:ed on the bug.  Please re-add the flag if you do want some info from me, however.  (FWIW, your plan to drive a GeckoView test activity via Selendroid connecting to Marionette sounds reasonable in theory, although I'm not familiar enough with either project to speak to the devilish details.)
Flags: needinfo?(myk)
Flags: needinfo?(nchen)
Depends on: 1243415
Depends on: 1322993
Depends on: 1323169
Depends on: 1323185
Depends on: 1323189
Quick update on my latest fixes and skip patches... As of now we have 4/10 chunks green. With a fix for bug 1322993 we will get in an even much better shape.
Depends on: 1323451
Depends on: 1323831
Depends on: 1324725
Depends on: 1326047
Depends on: 1328696
Depends on: 1329683
Depends on: 1325738
Depends on: 1330282
Depends on: 1330321
Depends on: 1330323
Depends on: 1330585
Quick update here... with all the fixed I did over the last couple of weeks, we now have 6/10 chunks (later today maybe even 7/10) green. Only a couple of failures are remaining until we can decide bumping up the tier of Mn tests for Fennec.
Depends on: 1330598
Depends on: 1330616
Depends on: 1330642
Depends on: 1331037
Lets make it a tracking bug for greening up the tests. I will file a follow-up bug to actually bump to Tier-2 as next step.
Flags: needinfo?(snorp)
Summary: Green up Marionette tests on Fennec and bump up from Treeherder Tier 3 → Green up Marionette tests for Fennec
Depends on: 1333069
Depends on: 1293855
Depends on: 1333209
Btw. with the following push we no longer have permanent test failures for Marionette jobs:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=cef9cfc92fec2c03a4780601e1c35c47e19ddd86

There are still some random oranges where Marionette kills the application due to no responses in navigate()/get(). I hope to address this completely with bug 1322277. Once this is done we can consider bumping up the tier level.
Depends on: 1322277
Depends on: 1334031
Depends on: 1334032
Depends on: 1334033
Depends on: 1334034
Depends on: 1334095
Depends on: 1334141
Depends on: 1334142
Depends on: 1334149
Depends on: 1335480
Depends on: 1335554
The last major intermittent failure has been fixed and got landed on autoland earlier today. Since then our tests are green. Once we merged the code to m-c I will request the bump to tier-2 on Treeherder.
I don't think we need this bug anytime longer. There are still intermittent failures left to fix but those are low volume. Also other work in re-enabling tests with possible harness changes can be done on individual bugs.

I will work on uplifting all remaining fixes to aurora and beta, so we can enable tier-2 also for the upcoming ESR release.
Assignee: nobody → hskupin
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla54
It got all uplifted to aurora and beta.
You need to log in before you can comment on or make changes to this bug.