Closed Bug 1297394 Opened 8 years ago Closed 8 years ago

Green up Marionette tests for Fennec

Categories

(Remote Protocol :: Marionette, defect)

Product:
Remote Protocol
Component:
Marionette
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox52 fixed, firefox53 fixed, firefox54 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla54
Tracking Flags:
Tracking Status
firefox52 --- fixed
firefox53 --- fixed
firefox54 --- fixed

People

(Reporter: impossibus, Assigned: whimboo)

References

Details

Marionette tests on Fennec are scheduled since Bug 1284874. Before we set them to Tier 1 or 2, there are a few more failing tests to examine and fix or disable. To give an idea of scope, here's the current list of failures based on https://treeherder.mozilla.org/#/jobs?repo=try&revision=d613b73232b3bb6897487065cf886d5a756d4160: test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_array test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_boolean test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_null test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_number test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_object test_execute_script.py test_execute_script.TestExecuteChrome.test_argument_string test_execute_script.py test_execute_script.TestExecuteChrome.test_async_script_timeout test_execute_script.py test_execute_script.TestExecuteChrome.test_exception test_execute_script.py test_execute_script.TestExecuteChrome.test_globals test_execute_script.py test_execute_script.TestExecuteChrome.test_invalid_chrome_handle test_execute_script.py test_execute_script.TestExecuteChrome.test_lasting_side_effects test_execute_script.py test_execute_script.TestExecuteChrome.test_no_callback test_execute_script.py test_execute_script.TestExecuteChrome.test_no_return_value test_execute_script.py test_execute_script.TestExecuteChrome.test_permission test_execute_script.py test_execute_script.TestExecuteChrome.test_return_array test_execute_script.py test_execute_script.TestExecuteChrome.test_return_boolean test_execute_script.py test_execute_script.TestExecuteChrome.test_return_number test_execute_script.py test_execute_script.TestExecuteChrome.test_return_object test_execute_script.py test_execute_script.TestExecuteChrome.test_return_string test_execute_script.py test_execute_script.TestExecuteChrome.test_return_web_element test_execute_script.py test_execute_script.TestExecuteChrome.test_return_web_element_array test_execute_script.py test_execute_script.TestExecuteChrome.test_return_web_element_nodelist test_execute_script.py test_execute_script.TestExecuteChrome.test_sandbox_refresh_arguments test_execute_script.py test_execute_script.TestExecuteChrome.test_sandbox_reuse test_execute_script.py test_execute_script.TestExecuteChrome.test_stacktrace test_execute_script.py test_execute_script.TestExecuteChrome.test_system_dead_object test_execute_script.py test_execute_script.TestExecuteChrome.test_system_globals test_execute_script.py test_execute_script.TestExecuteChrome.test_system_sandbox_wrappedjsobject test_execute_script.py test_execute_script.TestExecuteChrome.test_unmarshal_element_collection test_execute_script.py test_execute_script.TestExecuteChrome.test_wrappedjsobject test_switch_frame.py test_switch_frame.TestSwitchFrame.test_should_be_able_to_switch_to_parent_from_second_level test_navigation.py test_navigation.TestNavigate.test_navigate_chrome_error test_navigation.py test_navigation.TestNavigate.test_should_not_error_if_nonexistent_url_used test_text.py test_text.TestText.test_should_report_key_code_of_arrow_keys_up_down_events test_text.py test_text.TestText.testShouldTypeAnInteger test_screenshot.py test_screenshot.Chrome.test_window test_screenshot.py test_screenshot.Content.test_html_document_element test_import_script.py test_import_script.TestImportScriptContent.test_imports_apply_globally test_typing.py test_typing.TestTyping.testShouldReportKeyCodeOfArrowKeysUpDownEvents test_typing.py test_typing.TestTyping.testCutAndPasteShortcuts test_typing.py test_typing.TestTyping.testWillSimulateAKeyPressWhenEnteringTextIntoTextAreas test_shadow_dom.py test_shadow_dom.TestShadowDom.test_shadow_dom test_click_scrolling.py test_click_scrolling.TestClickScrolling.test_should_not_scroll_elements_if_click_point_is_in_view test_addons.py test_addons.TestAddons.test_install_and_remove_temporary_unsigned_addon test_about_pages.py test_about_pages.TestAboutPages.test_back_forward test_about_pages.py test_about_pages.TestAboutPages.test_hang test_about_pages.py test_about_pages.TestAboutPages.test_navigate_non_remote_about_pages test_about_pages.py test_about_pages.TestAboutPages.test_navigate_shortcut_key test_about_pages.py test_about_pages.TestAboutPages.test_type_to_non_remote_tab test_about_pages.py test_about_pages.TestAboutPages.test_type_to_remote_tab test_window_type.py test_window_type.TestWindowTypeChrome.test_get_window_type test_profile_management.py test_profile_management.TestLog.test_in_app_restart_the_browser
David, do you have a sense of which of these tests should be fixed vs skipped on mobile?
Flags: needinfo?(dburns)
So we do not have a chrome UI for Fennec given that all is powered by Java. So we should most likely skip all of those tests first, which try to interact in any way with the chrome UI. Not sure about all the test_execute_script.TestExecuteChrome.* failures which seem to be run at API level and should still work.
Anything that is chrome related. For the most part we probably have some wrong assumptions on when in Chrome context.
Flags: needinfo?(dburns)
A few other tests fail intermittenly: TEST-UNEXPECTED-ERROR | test_visibility.py TestVisibility.testShouldModifyTheVisibilityOfAnElementDynamically | TimeoutException: Error loading page, timed out (checkLoad) TEST-UNEXPECTED-ERROR | test_visibility.py TestVisibility.testShouldSayElementIsInvisibleWhenOverflowXIsHiddenAndOutOfViewport | TimeoutException: Error loading page, timed out (onDOMContentLoaded) TEST-UNEXPECTED-ERROR | test_shadow_dom.py TestShadowDom.test_shadow_dom | TimeoutException: Error loading page, timed out (checkLoad) TEST-UNEXPECTED-ERROR | test_shadow_dom.py TestShadowDom.test_shadow_dom_after_switch_away_from_shadow_root | TimeoutException: Error loading page, timed out (onDOMContentLoaded)
Lots of tests which fail with "onDOMContentLoaded" happen most likely because tests in setUp() we try to load about:blank, while this is our default start page. So it will fail with Marionette right now. We should use another page like about: to actually get rid of this problem. I will do something like that for bug 1156427 and test_navigate(). Lets see if that helps here.
Depends on: 1156427
Since my patch on bug 1156427 landed the two onDOMContentLoaded failures are gone now for test_navigation. At least a small step forward...
Lots of tests are using ui elements which do not exist in Fennec: https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=mn&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=3510219 We should simply mark them as desktop only.
myk, snorp, jchen: just letting you know about this ticket and the chain of tickets adding Marionette suppor to Fennec around it. I think the best way to test interacting with the contents of GeckoView will be to run something like Selendroid against a GeckoView test Activity, connecting to Marionette instead of the bespoke Android WebView like Selendroid does. Some of this probably already works, but hammering out details will be ... interesting.
Flags: needinfo?(snorp)
Flags: needinfo?(nchen)
Flags: needinfo?(myk)
Please note that I will reorganize the Marionette unit tests in the next couple of days via bug 1320632. With that I will identify chrome tests which we currently run for Fennec and which needs to be disabled for it. It will lower the number of failures we see on Treeherder.
Depends on: 1320632
(In reply to Nick Alexander :nalexander (leave until January 2017) from comment #8) > myk, snorp, jchen: just letting you know about this ticket and the chain of > tickets adding Marionette suppor to Fennec around it. Thanks, Nick, for the heads-up! It sounds like don't need any actual info from me at this point, so I'm removing that flag, and I'll stay in the loop by being cc:ed on the bug. Please re-add the flag if you do want some info from me, however. (FWIW, your plan to drive a GeckoView test activity via Selendroid connecting to Marionette sounds reasonable in theory, although I'm not familiar enough with either project to speak to the devilish details.)
Flags: needinfo?(myk)
Flags: needinfo?(nchen)
Depends on: 1243415
Depends on: 1322993
Depends on: 1323169
Depends on: 1323185
Depends on: 1323189
Quick update on my latest fixes and skip patches... As of now we have 4/10 chunks green. With a fix for bug 1322993 we will get in an even much better shape.
Depends on: 1323451
Depends on: 1323831
Depends on: 1324725
Depends on: 1326047
Depends on: 1328696
Depends on: 1329683
Depends on: 1325738
Depends on: 1330282
Depends on: 1330321
Depends on: 1330323
Depends on: 1330585
Quick update here... with all the fixed I did over the last couple of weeks, we now have 6/10 chunks (later today maybe even 7/10) green. Only a couple of failures are remaining until we can decide bumping up the tier of Mn tests for Fennec.
Depends on: 1330598
Depends on: 1330616
Depends on: 1330642
Depends on: 1331037
Lets make it a tracking bug for greening up the tests. I will file a follow-up bug to actually bump to Tier-2 as next step.
Flags: needinfo?(snorp)
Summary: Green up Marionette tests on Fennec and bump up from Treeherder Tier 3 → Green up Marionette tests for Fennec
Blocks: 1332205
Depends on: 1333069
Depends on: 1293855
Depends on: 1333209
Btw. with the following push we no longer have permanent test failures for Marionette jobs: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=cef9cfc92fec2c03a4780601e1c35c47e19ddd86 There are still some random oranges where Marionette kills the application due to no responses in navigate()/get(). I hope to address this completely with bug 1322277. Once this is done we can consider bumping up the tier level.
Depends on: 1322277
Depends on: 1334031
Depends on: 1334032
Depends on: 1334033
Depends on: 1334034
Depends on: 1334095
Depends on: 1334141
Depends on: 1334142
Depends on: 1334149
Depends on: 1335480
Depends on: 1335554
The last major intermittent failure has been fixed and got landed on autoland earlier today. Since then our tests are green. Once we merged the code to m-c I will request the bump to tier-2 on Treeherder.
I don't think we need this bug anytime longer. There are still intermittent failures left to fix but those are low volume. Also other work in re-enabling tests with possible harness changes can be done on individual bugs. I will work on uplifting all remaining fixes to aurora and beta, so we can enable tier-2 also for the upcoming ESR release.
Assignee: nobody → hskupin
Status: NEW → RESOLVED
Closed: 8 years ago
status-firefox52: --- → affected
status-firefox53: --- → affected
status-firefox54: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla54
It got all uplifted to aurora and beta.
status-firefox52: affected → fixed
status-firefox53: affected → fixed
Product: Testing → Remote Protocol
You need to log in before you can comment on or make changes to this bug.