Closed Bug 1868353 Opened 2 years ago Closed 2 years ago

[wpt-sync] Sync PR 43514 - [wdspec] Introduce shared constants for bidi network tests

Categories

(Remote Protocol :: WebDriver BiDi, task, P4)

Product:
Remote Protocol
Component:
WebDriver BiDi
Type:
task
Points:
1

Tracking

(firefox122 fixed)

Status:
RESOLVED FIXED
Milestone:
122 Branch
Tracking Flags:
Tracking Status
firefox122 --- fixed

People

(Reporter: wpt-sync, Assigned: jdescottes)

References

(
URL
)

Details

(Whiteboard: [webdriver:m9] [wptsync downstream])

Sync web-platform-tests PR 43514 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/43514
Details from upstream follow.

Julian Descottes <jdescottes@mozilla.com> wrote:

[wdspec] Introduce shared constants for bidi network tests

Extracted and expanded from PR #42667

This PR started moving some URLs and network event names to network's init.py.
To avoid mixing this with the rest of the changes, let's have a dedicated PR for this straightforward refactor.

Component: web-platform-tests → WebDriver BiDi
Product: Testing → Remote Protocol

CI Results

Ran 10 Firefox configurations based on mozilla-central, and Firefox, Chrome, and Safari on GitHub CI

Total 17 tests and 5 subtests

Status Summary

Firefox

OK : 17
PASS : 325
FAIL : 10

Chrome

OK : 14
PASS : 160
FAIL : 58
TIMEOUT: 3

Safari

OK : 17
ERROR : 335

Links

Gecko CI (Treeherder)
GitHub PR Head
GitHub PR Base

Details

New Tests That Don't Pass

Hi Julian, do you have an idea why that many tests started to fail for Firefox in our downstream sync?

Flags: needinfo?(jdescottes)

I'm not sure why this gets reported as a new failure, they have been failing (on mc and upstream) since end of october.

The tests in fail_request/invalid.py.ini have been FAIL for a while for firefox. We don't implement this command at all. See the wpt.fyi history which shows a FAIL status since the end of October.

Same goes for the test_response_headers ones, they are marked FAIL since the sync done in Bug 1860524 (https://hg.mozilla.org/integration/autoland/rev/44dcb2922209). But for those 2 I don't remember if this is an expected issue.

Edit: for test_response_headers, the failures make sense, filed https://bugzilla.mozilla.org/show_bug.cgi?id=1868576

Sasha: do you understand why those old failures are reported as new here?

Flags: needinfo?(jdescottes) → needinfo?(aborovova)

Maybe the test sync bot cannot differentiate between "new" tests and updated tests? Because all the tests listed above have been slightly updated (to use constants instead of hardcoded strings), but they are not new.

Assignee: wptsync → jdescottes
Status: NEW → ASSIGNED
Points: --- → 1
Whiteboard: [wptsync downstream] → [webdriver:m9] [wptsync downstream]

(In reply to Julian Descottes [:jdescottes] from comment #4)

I'm not sure why this gets reported as a new failure, they have been failing (on mc and upstream) since end of october.

The tests in fail_request/invalid.py.ini have been FAIL for a while for firefox. We don't implement this command at all. See the wpt.fyi history which shows a FAIL status since the end of October.

Same goes for the test_response_headers ones, they are marked FAIL since the sync done in Bug 1860524 (https://hg.mozilla.org/integration/autoland/rev/44dcb2922209). But for those 2 I don't remember if this is an expected issue.

Edit: for test_response_headers, the failures make sense, filed https://bugzilla.mozilla.org/show_bug.cgi?id=1868576

Sasha: do you understand why those old failures are reported as new here?

Yes, I think it's considered new because it was updated and at the moment the bot doesn't process the previous data from try/central of what the status of the test was, so it doesn't know that the test was failing already. We're going to try to change this next year with starting to process the previous test run results, so we hopefully should be able to improve these notifications and also generate more accurate metadata.

Flags: needinfo?(aborovova)
Pushed by wptsync@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/caa0efc58e56 [wpt PR 43514] - [wdspec] Introduce shared constants for bidi network tests, a=testonly https://hg.mozilla.org/integration/autoland/rev/d3a60095b81d [wpt PR 43514] - Update wpt metadata, a=testonly
Blocks: 1870032

https://hg.mozilla.org/mozilla-central/rev/caa0efc58e56
https://hg.mozilla.org/mozilla-central/rev/d3a60095b81d

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox122: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 122 Branch
You need to log in before you can comment on or make changes to this bug.