1870816 - Perma late beta webdriver/tests/bidi/network/continue_with_auth/action.py | test_provideCredentials - AssertionError when Gecko 122 merges to Late Beta on 2024-01-05

Status: RESOLVED FIXED

(Remote Protocol :: WebDriver BiDi, defect, P1)

Description

[Noemi Erli[:noemi_erli]]

Central-as-beta simulation:
How to run these simulations:
Failure log

[task 2023-12-19T12:15:18.728Z] 12:15:18     INFO - TEST-PASS | /webdriver/tests/bidi/network/continue_with_auth/action.py | test_default 
[task 2023-12-19T12:15:18.729Z] 12:15:18     INFO - TEST-UNEXPECTED-FAIL | /webdriver/tests/bidi/network/continue_with_auth/action.py | test_provideCredentials - AssertionError
[task 2023-12-19T12:15:18.729Z] 12:15:18     INFO - setup_blocked_request = <function setup_blocked_request.<locals>.setup_blocked_request at 0x7f83f77ac310>
[task 2023-12-19T12:15:18.730Z] 12:15:18     INFO - subscribe_events = <function subscribe_events.<locals>.subscribe_events at 0x7f83f77acb80>
[task 2023-12-19T12:15:18.730Z] 12:15:18     INFO - bidi_session = <webdriver.bidi.client.BidiSession object at 0x7f83f77d7130>
[task 2023-12-19T12:15:18.730Z] 12:15:18     INFO - url = <function url.<locals>.url at 0x7f83f77ac040>
[task 2023-12-19T12:15:18.731Z] 12:15:18     INFO - 
[task 2023-12-19T12:15:18.731Z] 12:15:18     INFO -     async def test_provideCredentials(
[task 2023-12-19T12:15:18.731Z] 12:15:18     INFO -         setup_blocked_request, subscribe_events, bidi_session, url
[task 2023-12-19T12:15:18.731Z] 12:15:18     INFO -     ):
[task 2023-12-19T12:15:18.731Z] 12:15:18     INFO -         # Setup unique username / password because browsers cache credentials.
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -         username = "test_provideCredentials"
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -         password = "test_provideCredentials_password"
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -         request = await setup_blocked_request("authRequired", username=username, password=password)
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -     
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -         # Additionally subscribe to network.responseCompleted
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -         await subscribe_events(events=[RESPONSE_COMPLETED_EVENT])
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -     
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -         # Track all received network.responseCompleted events in the events array
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -         events = []
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -     
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -         async def on_event(method, data):
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -             events.append(data)
[task 2023-12-19T12:15:18.732Z] 12:15:18     INFO -     
[task 2023-12-19T12:15:18.733Z] 12:15:18     INFO -         remove_listener = bidi_session.add_event_listener(
[task 2023-12-19T12:15:18.733Z] 12:15:18     INFO -             RESPONSE_COMPLETED_EVENT, on_event
[task 2023-12-19T12:15:18.733Z] 12:15:18     INFO -         )
[task 2023-12-19T12:15:18.733Z] 12:15:18     INFO -     
[task 2023-12-19T12:15:18.733Z] 12:15:18     INFO -         credentials = AuthCredentials(username=username, password=password)
[task 2023-12-19T12:15:18.733Z] 12:15:18     INFO -         await bidi_session.network.continue_with_auth(
[task 2023-12-19T12:15:18.733Z] 12:15:18     INFO -             request=request, action="provideCredentials", credentials=credentials
[task 2023-12-19T12:15:18.733Z] 12:15:18     INFO -         )
[task 2023-12-19T12:15:18.733Z] 12:15:18     INFO -     
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO -         wait = AsyncPoll(bidi_session, message="Didn't receive response completed events")
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO -         await wait.until(lambda _: len(events) >= 2)
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO -         assert len(events) == 2
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO -     
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO - >       assert_response_event(
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO -             events[0],
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO -             expected_response={
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO -                 "status": 401,
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO -                 "statusText": "Unauthorized",
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO -             },
[task 2023-12-19T12:15:18.734Z] 12:15:18     INFO -         )
[task 2023-12-19T12:15:18.735Z] 12:15:18     INFO - 

Julian, could you please take a look at this?
Thanks.

Comment 1

[Julian Descottes [:jdescottes]]

This might be related to network.auth.use_redirect_for_retries which is only true for early beta or earlier: https://searchfox.org/mozilla-central/source/modules/libpref/init/StaticPrefList.yaml#11333-11337

Comment 2

[Julian Descottes [:jdescottes]]

I confirm this comes from the pref mentioned above. We always get the expected number of "network.responseCompleted" events, but the status differs depending on the preference.

With the pref enabled the failed attempts have a 401 status, and only the final one has a 200 status code. With the pref disabled, all responseCompleted events have a 200 status.

This reminds me that the spec actually expects a single responseCompleted event, which might have to change in the future (see https://bugzilla.mozilla.org/show_bug.cgi?id=1826196#c2)

So I think for now I should relax the test to only wait until we get at least 1 responseCompleted event with status OK, and stop asserting the number of events and the status of previous responseCompleted events.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1826196

Comment 4

[Julian Descottes [:jdescottes]]

Attachment: Bug 1870816 - [wdspec] Do not assert each responseCompleted event in continue with auth tests

Comment 5

[Julian Descottes [:jdescottes]]

Regular try push: https://treeherder.mozilla.org/jobs?repo=try&revision=7d0a81ddabcf9c9c2d52ef90aa607cb6042da26c (no idea why all xpcshell tests are orange)
Late beta simulation: https://treeherder.mozilla.org/jobs?repo=try&revision=398efdd32ec2b11954236e515432e73876685aea

Comment 6

[Julian Descottes [:jdescottes]]

Comment on attachment 9369426 [details]
Bug 1870816 - [wdspec] Do not assert each responseCompleted event in continue with auth tests

Beta/Release Uplift Approval Request

• User impact if declined: Fixes permafailure for late beta
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Test-only change
• String changes made/needed:
• Is Android affected?: No

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4d6eabdc6228
[wdspec] Do not assert each responseCompleted event in continue with auth tests r=webdriver-reviewers,whimboo

Comment 8

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/43742 for changes under testing/web-platform/tests

Comment 9

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/4d6eabdc6228

Comment 10

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 11

[Donal Meehan [:dmeehan]]

Comment on attachment 9369426 [details]
Bug 1870816 - [wdspec] Do not assert each responseCompleted event in continue with auth tests

Approved for 122.0b3

Comment 12

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/6902caac4bec

Comment 13

[Atila Butkovits]

Verified fixed in today's beta simulation

Comment 14

[Atila Butkovits]

(In reply to Atila Butkovits from comment #13)

Verified fixed in today's beta simulation

Please disregard comment 13.