Closed Bug 1453105 Opened 2 years ago Closed 2 years ago

[wpt-sync] Sync PR 10401 - Updating WebDriver execute[Async]Script tests for unhandled user prompts

Categories

(Testing :: geckodriver, enhancement, P4)

enhancement

Tracking

(firefox61 fixed)

RESOLVED FIXED
mozilla61
Tracking Status
firefox61 --- fixed

People

(Reporter: wptsync, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug, )

Details

(Whiteboard: [wptsync downstream])

Sync web-platform-tests PR 10401 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/w3c/web-platform-tests/pull/10401
Details from upstream follow.

Jim Evans <james.h.evans.jr@gmail.com> wrote:
>   Updating WebDriver execute[Async]Script tests for unhandled user prompts 
>  
>  When the WebDriver specification is modified with the contents of PR 1248
>  (https://github.com/w3c/webdriver/pull/1248), the behavior of
>  implementations changes to handle user prompts on the next command after
>  an executeScript or executeAsyncScript command. This PR modifies the
>  tests for user prompts for those commands to match the spec behavior.
Component: web-platform-tests → geckodriver
The following tests were disabled based on stability try push:
 webdriver/tests/execute_async_script/test_handle_prompt_twice
webdriver/tests/execute_async_script/user_prompts.py
webdriver/tests/execute_script/test_handle_prompt_twice
webdriver/tests/execute_script/user_prompts.py
Ran 2 tests and 14 subtests
OK     : 2
PASS   : 3[linux64-opt], 4[linux32-debug,linux32-opt,linux64-debug]
FAIL   : 10[linux32-debug,linux32-opt,linux64-debug], 11[linux64-opt]
The following tests were disabled based on stability try push:
 webdriver/tests/execute_script/user_prompts.py
webdriver/tests/execute_async_script/test_handle_prompt_twice
webdriver/tests/execute_async_script/user_prompts.py
James, this is strange:

> +  [test_handle_prompt_twice]
> +    disabled:
> +      if not debug and not webrender and e10s and (os == "linux") and (version == "Ubuntu 16.04") and (processor == "x86_64") and (bits == 64): wpt-sync Bug 1453105
>  
> -  [test_handle_prompt_twice]
> -    expected: FAIL

This test is marked as expected to fail, and it fails on try. Why is it reported as unexpected fail, and now gets disabled?
Flags: needinfo?(james)
Pushed by wptsync@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3f72c9dde8d6
[wpt PR 10401] -  Updating WebDriver execute[Async]Script tests for unhandled user prompts , a=testonly
https://hg.mozilla.org/integration/mozilla-inbound/rev/31f6f6815fe1
[wpt PR 10401] - Update wpt metadata, a=testonly
https://hg.mozilla.org/mozilla-central/rev/3f72c9dde8d6
https://hg.mozilla.org/mozilla-central/rev/31f6f6815fe1
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Presumably the problem here was that we saw unstable results from the test, and so opted to disable it. Of course our intermittent detection isn't infallible, so it may be that it's a very low frequency problem, or actually caused by issues in another test.
Flags: needinfo?(james)
Hm, indeed. It's not a perma expected fail so it could still be that we have seen some unexpected pass, which should have caused to mark the test as failed. I will keep an eye on those tests. Thanks.
You need to log in before you can comment on or make changes to this bug.