Closed Bug 1453105 Opened 2 years ago Closed 2 years ago
[wpt-sync] Sync PR 10401 - Updating Web
Driver execute[Async]Script tests for unhandled user prompts
Sync web-platform-tests PR 10401 into mozilla-central (this bug is closed when the sync is complete). PR: https://github.com/w3c/web-platform-tests/pull/10401 Details from upstream follow. Jim Evans <email@example.com> wrote: > Updating WebDriver execute[Async]Script tests for unhandled user prompts > > When the WebDriver specification is modified with the contents of PR 1248 > (https://github.com/w3c/webdriver/pull/1248), the behavior of > implementations changes to handle user prompts on the next command after > an executeScript or executeAsyncScript command. This PR modifies the > tests for user prompts for those commands to match the spec behavior.
Component: web-platform-tests → geckodriver
Pushed to try (stability) https://treeherder.mozilla.org/#/jobs?repo=try&revision=20b1ca035403be374bd17d9a41701fa2e3318ea1
The following tests were disabled based on stability try push: webdriver/tests/execute_async_script/test_handle_prompt_twice webdriver/tests/execute_async_script/user_prompts.py webdriver/tests/execute_script/test_handle_prompt_twice webdriver/tests/execute_script/user_prompts.py
Ran 2 tests and 14 subtests OK : 2 PASS : 3[linux64-opt], 4[linux32-debug,linux32-opt,linux64-debug] FAIL : 10[linux32-debug,linux32-opt,linux64-debug], 11[linux64-opt]
Pushed to try (stability) https://treeherder.mozilla.org/#/jobs?repo=try&revision=a4f6f727e758199ca30c66e1e7ed489e9435ce04
The following tests were disabled based on stability try push: webdriver/tests/execute_script/user_prompts.py webdriver/tests/execute_async_script/test_handle_prompt_twice webdriver/tests/execute_async_script/user_prompts.py
James, this is strange: > + [test_handle_prompt_twice] > + disabled: > + if not debug and not webrender and e10s and (os == "linux") and (version == "Ubuntu 16.04") and (processor == "x86_64") and (bits == 64): wpt-sync Bug 1453105 > > - [test_handle_prompt_twice] > - expected: FAIL This test is marked as expected to fail, and it fails on try. Why is it reported as unexpected fail, and now gets disabled?
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/mozilla-inbound/rev/3f72c9dde8d6 [wpt PR 10401] - Updating WebDriver execute[Async]Script tests for unhandled user prompts , a=testonly https://hg.mozilla.org/integration/mozilla-inbound/rev/31f6f6815fe1 [wpt PR 10401] - Update wpt metadata, a=testonly
Presumably the problem here was that we saw unstable results from the test, and so opted to disable it. Of course our intermittent detection isn't infallible, so it may be that it's a very low frequency problem, or actually caused by issues in another test.
Hm, indeed. It's not a perma expected fail so it could still be that we have seen some unexpected pass, which should have caused to mark the test as failed. I will keep an eye on those tests. Thanks.
You need to log in before you can comment on or make changes to this bug.