Closed Bug 1453105 Opened 7 years ago Closed 7 years ago

[wpt-sync] Sync PR 10401 - Updating WebDriver execute[Async]Script tests for unhandled user prompts

Categories

(Testing :: geckodriver, enhancement, P4)

Product:
Testing
Component:
geckodriver
Type:
enhancement

Tracking

(firefox61 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla61
Tracking Flags:
Tracking Status
firefox61 --- fixed

People

(Reporter: wpt-sync, Unassigned)

References

(
URL
)

Details

(Whiteboard: [wptsync downstream])

Sync web-platform-tests PR 10401 into mozilla-central (this bug is closed when the sync is complete). PR: https://github.com/w3c/web-platform-tests/pull/10401 Details from upstream follow. Jim Evans <james.h.evans.jr@gmail.com> wrote: > Updating WebDriver execute[Async]Script tests for unhandled user prompts > > When the WebDriver specification is modified with the contents of PR 1248 > (https://github.com/w3c/webdriver/pull/1248), the behavior of > implementations changes to handle user prompts on the next command after > an executeScript or executeAsyncScript command. This PR modifies the > tests for user prompts for those commands to match the spec behavior.
Component: web-platform-tests → geckodriver
The following tests were disabled based on stability try push: webdriver/tests/execute_async_script/test_handle_prompt_twice webdriver/tests/execute_async_script/user_prompts.py webdriver/tests/execute_script/test_handle_prompt_twice webdriver/tests/execute_script/user_prompts.py
Ran 2 tests and 14 subtests OK : 2 PASS : 3[linux64-opt], 4[linux32-debug,linux32-opt,linux64-debug] FAIL : 10[linux32-debug,linux32-opt,linux64-debug], 11[linux64-opt]
The following tests were disabled based on stability try push: webdriver/tests/execute_script/user_prompts.py webdriver/tests/execute_async_script/test_handle_prompt_twice webdriver/tests/execute_async_script/user_prompts.py
James, this is strange: > + [test_handle_prompt_twice] > + disabled: > + if not debug and not webrender and e10s and (os == "linux") and (version == "Ubuntu 16.04") and (processor == "x86_64") and (bits == 64): wpt-sync Bug 1453105 > > - [test_handle_prompt_twice] > - expected: FAIL This test is marked as expected to fail, and it fails on try. Why is it reported as unexpected fail, and now gets disabled?
Flags: needinfo?(james)
Pushed by wptsync@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/3f72c9dde8d6 [wpt PR 10401] - Updating WebDriver execute[Async]Script tests for unhandled user prompts , a=testonly https://hg.mozilla.org/integration/mozilla-inbound/rev/31f6f6815fe1 [wpt PR 10401] - Update wpt metadata, a=testonly
https://hg.mozilla.org/mozilla-central/rev/3f72c9dde8d6 https://hg.mozilla.org/mozilla-central/rev/31f6f6815fe1
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox61: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Presumably the problem here was that we saw unstable results from the test, and so opted to disable it. Of course our intermittent detection isn't infallible, so it may be that it's a very low frequency problem, or actually caused by issues in another test.
Flags: needinfo?(james)
Hm, indeed. It's not a perma expected fail so it could still be that we have seen some unexpected pass, which should have caused to mark the test as failed. I will keep an eye on those tests. Thanks.
You need to log in before you can comment on or make changes to this bug.