1721012 - Permanent /webdriver/tests/bidi/new_session/connect.py | test_websocket_url_connect[capabilities0] - KeyError: 'webSocketUrl' - when Gecko 92 merges to Beta on 2021-08-09

Status: VERIFIED FIXED

(Remote Protocol :: Marionette, defect, P2)

Description

[Sandor Molnar[:smolnar]]

• Central-as-beta-simulation
  • How to run these simulations
  • Failure log

[task 2021-07-17T12:09:37.733Z] 12:09:37     INFO - STDOUT: session = <Session 10d905d4-3d9c-49ee-910e-5ef8864100a4>
[task 2021-07-17T12:09:37.733Z] 12:09:37     INFO - STDOUT:     @pytest.mark.asyncio
[task 2021-07-17T12:09:37.734Z] 12:09:37     INFO - STDOUT:     @pytest.mark.capabilities({"webSocketUrl": True})
[task 2021-07-17T12:09:37.735Z] 12:09:37     INFO - STDOUT:     async def test_websocket_url_connect(session):
[task 2021-07-17T12:09:37.736Z] 12:09:37     INFO - STDOUT: >       websocket_url = session.capabilities["webSocketUrl"]
[task 2021-07-17T12:09:37.736Z] 12:09:37     INFO - STDOUT: E       KeyError: 'webSocketUrl'
[task 2021-07-17T12:09:37.737Z] 12:09:37     INFO - STDOUT: session    = <Session 10d905d4-3d9c-49ee-910e-5ef8864100a4>
[task 2021-07-17T12:09:37.738Z] 12:09:37     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/bidi/new_session/connect.py
[task 2021-07-17T12:09:37.738Z] 12:09:37     INFO - STDOUT: :11: KeyError
[task 2021-07-17T12:09:37.739Z] 12:09:37     INFO - STDOUT: =========================== short test summary info ============================
[task 2021-07-17T12:09:37.740Z] 12:09:37     INFO - 
[task 2021-07-17T12:09:37.741Z] 12:09:37     INFO - TEST-UNEXPECTED-FAIL | /webdriver/tests/bidi/new_session/connect.py | test_websocket_url_connect[capabilities0] - KeyError: 'webSocketUrl'
[task 2021-07-17T12:09:37.741Z] 12:09:37     INFO - session = <Session 10d905d4-3d9c-49ee-910e-5ef8864100a4>
[task 2021-07-17T12:09:37.741Z] 12:09:37     INFO - 
[task 2021-07-17T12:09:37.741Z] 12:09:37     INFO -     @pytest.mark.asyncio
[task 2021-07-17T12:09:37.741Z] 12:09:37     INFO -     @pytest.mark.capabilities({"webSocketUrl": True})
[task 2021-07-17T12:09:37.741Z] 12:09:37     INFO -     async def test_websocket_url_connect(session):
[task 2021-07-17T12:09:37.742Z] 12:09:37     INFO - >       websocket_url = session.capabilities["webSocketUrl"]
[task 2021-07-17T12:09:37.742Z] 12:09:37     INFO - E       KeyError: 'webSocketUrl'
[task 2021-07-17T12:09:37.742Z] 12:09:37     INFO - 
[task 2021-07-17T12:09:37.742Z] 12:09:37     INFO - session    = <Session 10d905d4-3d9c-49ee-910e-5ef8864100a4>
[task 2021-07-17T12:09:37.742Z] 12:09:37     INFO - 
[task 2021-07-17T12:09:37.742Z] 12:09:37     INFO - tests/web-platform/tests/webdriver/tests/bidi/new_session/connect.py:11: KeyError
[task 2021-07-17T12:09:37.742Z] 12:09:37     INFO - TEST-INFO | expected ERROR
[task 2021-07-17T12:09:37.755Z] 12:09:37     INFO - ...
[task 2021-07-17T12:09:37.755Z] 12:09:37     INFO - TEST-OK | /webdriver/tests/bidi/new_session/connect.py | took 21763ms```

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The failure here happens when accessing session.capabilities["webSocketUrl"]. Given that on beta the Remote Agent is disabled, Marionette doesn't return this capability to the client. The current manifest of the file looks like:

https://searchfox.org/mozilla-central/source/testing/web-platform/meta/webdriver/tests/bidi/new_session/connect.py.ini

Problem here is most likely that we expect a FAIL instead of an ERROR. I'll have a look soon.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Indeed the ERROR state is only reached when something goes wrong during setup or teardown. In this case the above mentioned failure happens within the test, and as such it's expecting to fail.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1721012 - [wdspec] Expect 'bidi/new_session/connect.py | test_websocket_url_connect" to fail when BiDi is disabled.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2df3d7d91c43
[wdspec] Expect 'bidi/new_session/connect.py | test_websocket_url_connect" to fail when BiDi is disabled. r=webdriver-reviewers,jdescottes DONTBUILD

Comment 5

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/2df3d7d91c43

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Verified fixed with today's central-as-beta simulation.