Closed Bug 1859871 Opened 8 months ago Closed 8 months ago

Intermittent /webdriver/tests/bidi/browsing_context/load/load.py | test_iframe - assert 3 == 2

Categories

(Remote Protocol :: WebDriver BiDi, defect, P5)

Product:
Remote Protocol
Component:
WebDriver BiDi
Type:
defect
Points:
1

Tracking

(firefox121 fixed)

Status:
RESOLVED FIXED
Milestone:
121 Branch
Tracking Flags:
Tracking Status
firefox121 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: jdescottes)

References

Details

(Keywords: intermittent-failure, Whiteboard: [webdriver:m9])

Attachments

(1 file)

Bug 1859871 - [wdspec] Use wait=complete navigation in bidi browsingContext.load test
48 bytes, text/x-phabricator-request
Details | Review

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=433012404&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WjklaC8qTcqe5WxmF1_qNQ/runs/0/artifacts/public/logs/live_backing.log

[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO - TEST-PASS | /webdriver/tests/bidi/browsing_context/load/load.py | test_subscribe 
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO - TEST-PASS | /webdriver/tests/bidi/browsing_context/load/load.py | test_timestamp 
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO - TEST-UNEXPECTED-FAIL | /webdriver/tests/bidi/browsing_context/load/load.py | test_iframe - assert 3 == 2
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO - bidi_session = <webdriver.bidi.client.BidiSession object at 0x0000023E66424130>
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO - subscribe_events = <function subscribe_events.<locals>.subscribe_events at 0x0000023E664B61F0>
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO - new_tab = {'context': '442fdd28-785a-45e5-9078-bc35f083a4d6'}
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO - test_page = 'https://web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Cdiv%3Efoo%3C%2Fdiv%3E&mime=text%2Fhtml&charset=UTF-8'
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO - test_page_same_origin_frame = 'https://web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%...253Efoo%253C%252Fdiv%253E%26mime%3Dtext%252Fhtml%26charset%3DUTF-8%27%3E%3C%2Fiframe%3E&mime=text%2Fhtml&charset=UTF-8'
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO - 
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -     async def test_iframe(
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -         bidi_session, subscribe_events, new_tab, test_page, test_page_same_origin_frame
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -     ):
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -         events = []
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -     
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -         async def on_event(method, data):
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -             events.append(data)
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -     
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -         remove_listener = bidi_session.add_event_listener(CONTEXT_LOAD_EVENT, on_event)
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -         await subscribe_events(events=[CONTEXT_LOAD_EVENT])
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -     
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -         result = await bidi_session.browsing_context.navigate(
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -             context=new_tab["context"], url=test_page_same_origin_frame
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -         )
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -     
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -         wait = AsyncPoll(
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -             bidi_session, message="Didn't receive context load events for frames"
[task 2023-10-18T18:09:51.668Z] 18:09:51     INFO -         )
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -         await wait.until(lambda _: len(events) >= 2)
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - >       assert len(events) == 2
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - E       assert 3 == 2
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - E         +3
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - E         -2
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - 
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - bidi_session = <webdriver.bidi.client.BidiSession object at 0x0000023E66424130>
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - events     = [{'context': 'ae4ca1dc-5f54-4f87-b9ba-31622d6790c5',
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -   'navigation': '5108edec-bdd2-46d7-b51d-93ec6daf4702',
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -   'timestamp': 1697652584320,
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -   'url': 'https://web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Cdiv%3Efoo%3C%2Fdiv%3E&mime=text%2Fhtml&charset=UTF-8'},
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -  {'context': '21474836481',
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -   'navigation': 'b0a47256-952c-4409-8c72-3aaa4d985f1e',
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -   'timestamp': 1697652584327,
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -   'url': 'https://web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Cdiv%3Efoo%3C%2Fdiv%3E&mime=text%2Fhtml&charset=UTF-8'},
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -  {'context': '442fdd28-785a-45e5-9078-bc35f083a4d6',
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -   'navigation': '696cf4c2-9808-4af7-8ddb-cf69bc6b2161',
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -   'timestamp': 1697652584330,
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -   'url': 'https://web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Ciframe+src%3D%27https%3A%2F%2Fweb-platform.test%3A8443%2Fwebdriver%2Ftests%2Fsupport%2Finline.py%3Fdoc%3D%253C%2521doctype%2Bhtml%253E%250A%253Cmeta%2Bcharset%253DUTF-8%253E%250A%253Cdiv%253Efoo%253C%252Fdiv%253E%26mime%3Dtext%252Fhtml%26charset%3DUTF-8%27%3E%3C%2Fiframe%3E&mime=text%2Fhtml&charset=UTF-8'}]
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - new_tab    = {'context': '442fdd28-785a-45e5-9078-bc35f083a4d6'}
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - on_event   = <function test_iframe.<locals>.on_event at 0x0000023E66497CA0>
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - remove_listener = <function BidiSession.add_event_listener.<locals>.<lambda> at 0x0000023E66497F70>
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - result     = {'navigation': '696cf4c2-9808-4af7-8ddb-cf69bc6b2161',
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO -  'url': 'https://web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Ciframe+src%3D%27https%3A%2F%2Fweb-platform.test%3A8443%2Fwebdriver%2Ftests%2Fsupport%2Finline.py%3Fdoc%3D%253C%2521doctype%2Bhtml%253E%250A%253Cmeta%2Bcharset%253DUTF-8%253E%250A%253Cdiv%253Efoo%253C%252Fdiv%253E%26mime%3Dtext%252Fhtml%26charset%3DUTF-8%27%3E%3C%2Fiframe%3E&mime=text%2Fhtml&charset=UTF-8'}
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - subscribe_events = <function subscribe_events.<locals>.subscribe_events at 0x0000023E664B61F0>
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - test_page  = 'https://web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Cdiv%3Efoo%3C%2Fdiv%3E&mime=text%2Fhtml&charset=UTF-8'
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - test_page_same_origin_frame = 'https://web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Ciframe+src%3D%27https%3A%2F%2Fweb-platform.test%3A8443%2Fwebdriver%2Ftests%2Fsupport%2Finline.py%3Fdoc%3D%253C%2521doctype%2Bhtml%253E%250A%253Cmeta%2Bcharset%253DUTF-8%253E%250A%253Cdiv%253Efoo%253C%252Fdiv%253E%26mime%3Dtext%252Fhtml%26charset%3DUTF-8%27%3E%3C%2Fiframe%3E&mime=text%2Fhtml&charset=UTF-8'
[task 2023-10-18T18:09:51.669Z] 18:09:51     INFO - wait       = <tests.support.sync.AsyncPoll object at 0x0000023E664418B0>
[task 2023-10-18T18:09:51.670Z] 18:09:51     INFO - 
[task 2023-10-18T18:09:51.670Z] 18:09:51     INFO - tests\web-platform\tests\webdriver\tests\bidi\browsing_context\load\load.py:96: AssertionError
[task 2023-10-18T18:09:51.672Z] 18:09:51     INFO - ....
[task 2023-10-18T18:09:51.672Z] 18:09:51     INFO - TEST-OK | /webdriver/tests/bidi/browsing_context/load/load.py | took 13680ms
Duplicate of this bug: 1859872

It seems we are catching the load event of the very first test of the file test_not_unsubscribed. This test is actually not waiting for a load event but is using top_context whereas the other ones are using new_tab. So this first navigation can lead to a load event appearing in any other test.

We should probably navigate with wait=complete to avoid this.

Assignee: nobody → jdescottes
Status: NEW → ASSIGNED
Pushed by jdescottes@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/41cafc4f4828
[wdspec] Use wait=complete navigation in bidi browsingContext.load test r=webdriver-reviewers,Sasha
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/42695 for changes under testing/web-platform/tests

https://hg.mozilla.org/mozilla-central/rev/41cafc4f4828

Status: ASSIGNED → RESOLVED
Closed: 8 months ago
status-firefox121: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 121 Branch
Upstream PR merged by moz-wptsync-bot
Points: --- → 1
Whiteboard: [webdriver:m9]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: