1962306 - Intermittent MacOS /webdriver/tests/bidi/browsing_context/navigation_failed/navigation_failed.py | <something> - expected FAIL

Status: REOPENED

(Remote Protocol :: WebDriver BiDi, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=505259273&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VYe-oW1hTamvYOInYCzZXg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2025-04-24T02:06:20.963Z] 02:06:20     INFO - TEST-PASS | /webdriver/tests/bidi/browsing_context/navigation_failed/navigation_failed.py | test_with_content_blocking_header_in_top_context[Content-Security-Policy, default-src 'self'] 
[task 2025-04-24T02:06:20.963Z] 02:06:20     INFO - TEST-UNEXPECTED-PASS | /webdriver/tests/bidi/browsing_context/navigation_failed/navigation_failed.py | test_with_content_blocking_header_in_top_context[Cross-Origin-Embedder-Policy, require-corp] - expected FAIL
[task 2025-04-24T02:06:20.963Z] 02:06:20     INFO - TEST-INFO | expected FAIL
[task 2025-04-24T02:06:20.964Z] 02:06:20     INFO - 
[task 2025-04-24T02:06:20.964Z] 02:06:20     INFO - TEST-UNEXPECTED-PASS | /webdriver/tests/bidi/browsing_context/navigation_failed/navigation_failed.py | test_with_x_frame_options_header[SAMEORIGIN] - expected FAIL
[task 2025-04-24T02:06:20.964Z] 02:06:20     INFO - TEST-INFO | expected FAIL
[task 2025-04-24T02:06:20.965Z] 02:06:20     INFO - 
[task 2025-04-24T02:06:20.965Z] 02:06:20     INFO - TEST-UNEXPECTED-PASS | /webdriver/tests/bidi/browsing_context/navigation_failed/navigation_failed.py | test_with_x_frame_options_header[DENY] - expected FAIL
[task 2025-04-24T02:06:20.965Z] 02:06:20     INFO - TEST-INFO | expected FAIL
[task 2025-04-24T02:06:21.026Z] 02:06:21     INFO - ......
[task 2025-04-24T02:06:21.026Z] 02:06:21     INFO - TEST-OK | /webdriver/tests/bidi/browsing_context/navigation_failed/navigation_failed.py | took 25704ms

Comment 1

[amarc]

Tier 1 failure here

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Looks like we have some unexpected pass here for the navigation failed event. I had a look in the meta file of that test and none of these failing sub tests are actually marked as expected fail:

https://searchfox.org/mozilla-central/source/testing/web-platform/meta/webdriver/tests/bidi/browsing_context/navigation_failed/navigation_failed.py.ini

Checking hgweb the entries are shown which means searchfox isn't up-to-date yet:

https://hg-edge.mozilla.org/mozilla-central/file/tip/testing/web-platform/meta/webdriver/tests/bidi/browsing_context/navigation_failed/navigation_failed.py.ini

So this actually regressed by the last wpt sync from bug 1961860.

Sasha, can you please take a look?

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1961860

Comment 4

[Alexandra Borovova [:Sasha]]

Attachment: Bug 1962306 - Update metadata for browsing_context/navigation_failed/navigation_failed.py.

Comment 5

[Alexandra Borovova [:Sasha]]

So looks like it's a similar situation to bug 1958736. During the wptsync run there was only one worker with mac 14.7.1 which failed, and the rest were 14.7 where the tests passed, so wpt-update marked it as failing for 14.7.1. But running it again on try it looks like failures are pretty consistent: https://treeherder.mozilla.org/jobs?repo=try&revision=dca669c8b6ac4a44a803834edbd1ad1729980a0d, so we can just remove it. So I would suggest we mark it as pass/fail for now.
In terms of why it fails, the tests fail because navigationFailed event doesn't come in time. So it looks like it might be again similar to bug 1958736 being related to system timeout multiplier to be different/wrong.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d41fc175cb67
Update metadata for browsing_context/navigation_failed/navigation_failed.py. r=webdriver-reviewers,whimboo

Comment 7

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/d41fc175cb67

Comment 8

[Intermittent Failures Robot]

8 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 2
• try: 1

Table

	headless	no_variant
macosx1470-64--/opt	2	5
macosx1470-64--/debug	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1962306&startday=2025-04-21&endday=2025-04-27&tree=all

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The landed patch is not really a fix but only updated the expected status. We still need to figure why we pass/fail on MacOS for some of the tests:

• [test_with_content_blocking_header_in_top_context[Content-Security-Policy, default-src 'self']]
• [test_with_content_blocking_header_in_top_context[Cross-Origin-Embedder-Policy, require-corp]]
• [test_with_x_frame_options_header[SAMEORIGIN]]
• [test_with_x_frame_options_header[DENY]]

Comment 10

[Intermittent Failures Robot]

3 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-beta: 1

Table

	headless	no_variant
macosx1470-64--/opt		1
macosx1470-64-devedition--/opt		1
windows11-64-24h2--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1962306&startday=2025-04-28&endday=2025-05-04&tree=all