1831263 - (webgpu-v1-cts-windows) Webgpu CI jobs are not running in configs that can do anything but fail

Status: RESOLVED FIXED

(Core :: Graphics: WebGPU, defect, P1)

Description

[Kelsey Gilbert [:jgilbert]]

Quis custodiet ipsos custodes?

This is the most recent try run from bug :
https://treeherder.mozilla.org/jobs?repo=try&revision=adefaafc0b5d68929d5b0a8d3369fd523a154b59
Despite the extensive changes in that bug, apparently all's well. This was too suspicious to me.

Here's that same run but when we ask for a gpu:
https://treeherder.mozilla.org/jobs?repo=try&revision=fd60d0fa06e61a00633f0fca75faba6a474e7d65
Linux is still not getting a gpu, and so is successfully continuing to expect to fail all tests.
Windows is getting a GPU now though, and we finally start to see daylight:

[task 2023-05-04T08:06:39.943Z] 08:06:39 INFO - TEST-START | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:default:*
[task 2023-05-04T08:06:39.971Z] 08:06:39 INFO - Setting pref dom.webgpu.enabled to true
[task 2023-05-04T08:06:41.745Z] 08:06:41 INFO -
[task 2023-05-04T08:06:41.745Z] 08:06:41 INFO - TEST-UNEXPECTED-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:default:* | : - expected FAIL
[task 2023-05-04T08:06:41.745Z] 08:06:41 INFO - TEST-INFO | expected FAIL
[task 2023-05-04T08:06:41.746Z] 08:06:41 INFO - TEST-OK | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:default:* | took 1803ms

Also chunking probably doesn't work right now for this job, since I asked for 4 and I'm getting 2 still.

Similarly, mochitest-webgpu Linux jobs look like:

[task 2023-05-04T04:07:56.305Z] 04:07:56 INFO - TEST-START | dom/webgpu/mochitest/test_device_creation.html
[task 2023-05-04T04:07:56.490Z] 04:07:56 INFO - GECKO(1976) | Validation error without device target: No suitable adapter found
[task 2023-05-04T04:07:56.525Z] 04:07:56 INFO - GECKO(1976) | MEMORY STAT | vsize 2570MB | residentFast 146MB | heapAllocated 8MB
[task 2023-05-04T04:07:56.681Z] 04:07:56 INFO - TEST-OK | dom/webgpu/mochitest/test_device_creation.html | took 375ms

mochitest-webgpu does not appear to be running on Windows right now.

Comment 1

[Kelsey Gilbert [:jgilbert]]

Attachment: Bug 1831263 - Ask for gpu for W-webgpu jobs but drop Linux; run M-webgpu on Windows.

Comment 2

[Jim Blandy :jimb]

Kelsey and Erich are working on re-evaluating what our test expectations should be, so that when we land it we won't get massive oranges for unexpected passes.

Comment 3

[Kelsey Gilbert [:jgilbert]]

Attachment: wpt-re-mark.py

This is the script I wrote to automate re-marking tests, since wpt-update doesn't seem to be working, and Erich's previous experience was that it was slow anyway.
Feed it "wptreport.json"s one at a time, generally from the list of artifacts from a W(webgpuN) job, and it will re-mark tests based on the unexpected values in the report json.
It generally tries to preserve existing comments and line orders, with some narrow exceptions.

It has a framework to support amending expectations with e.g. if os == "linux": FAIL/PASS later on, but consider that aspect a WIP.

If this pans out, this will land somewhere more permanent, possibly in-tree.

Comment 4

[Kelsey Gilbert [:jgilbert]]

Erich will try to mark these failures in a one-off way, while finishing automating marking will likely land separately.

Comment 5

[Kelsey Gilbert [:jgilbert]]

I think we should keep the old title.
"Webgpu CI jobs are not running in configs that can do anything but fail" is the defect/bug, "Narrow WebGPU CTS testing in CI to Windows (for now)" is the solution/patch.

Comment 6

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1831263: fix(webgpu): track forgotten strong refs. in shader compilation r=#webgpu-reviewers,nical

Comment 7

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1831263: fix(webgpu): remove strong ref. cycle b/w `Compilation{Info,Message}` r=#webgpu-reviewers

Depends on D179812

Comment 8

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1831263: refactor(webgpu): align `Device::InitSwapChain`'s `IntSize` arg. name b/w header and impl. r=#webgpu-reviewers

Depends on D179813

Comment 9

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1831263: build(ci): restrict WebGPU to known working envs. (read: Windows) r=#webgpu-reviewers

Depends on D179814

Comment 10

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1831263: test(webgpu): simplify when tests runs r=#webgpu-reviewers

Co-Authored-By: Kelsey Gilbert <[e-mail]>

Depends on D179815

Comment 11

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1831263: build(ci): chunk WebGPU WPT tests more r=#webgpu-reviewers

Depends on D179816

Comment 12

[Erich Gubler [:ErichDonGubler]]

Seeing if I can land the first patches only depending on #webgpu-reviewers with this Try build. 🤞🏻 EDIT: It's green, ship it!

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/09f78a71f86a
fix(webgpu): track forgotten strong refs. in shader compilation r=webgpu-reviewers,jgilbert
https://hg.mozilla.org/integration/autoland/rev/485b73c565ed
fix(webgpu): remove strong ref. cycle b/w `Compilation{Info,Message}` r=webgpu-reviewers,jgilbert
https://hg.mozilla.org/integration/autoland/rev/d64d0a03298f
refactor(webgpu): align `Device::InitSwapChain`'s `IntSize` arg. name b/w header and impl. r=webgpu-reviewers,jgilbert

Comment 14

[Erich Gubler [:ErichDonGubler]]

Since D179816 has approval, I'm going to try reordering it to be earlier in the patch stack, so it can land. Checking that everything is still green in CI with this Try push. EDIT: It's green! Landing.

Comment 15

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/09f78a71f86a
https://hg.mozilla.org/mozilla-central/rev/485b73c565ed
https://hg.mozilla.org/mozilla-central/rev/d64d0a03298f

Comment 16

[Kelsey Gilbert [:jgilbert]]

This only partially landed.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1ceebfa927ec
test(webgpu): simplify when tests runs r=taskgraph-reviewers,bhearsum

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/efcb29ce73d6
build(ci): restrict WebGPU to known working envs. (read: Windows) r=webgpu-reviewers,jgilbert
https://hg.mozilla.org/integration/autoland/rev/dbc987c2d0cd
build(ci): chunk WebGPU WPT tests more r=webgpu-reviewers,jgilbert

Comment 19

[Atila Butkovits]

Backed outfor causing webgpu failures at mozilla/webgpu/chunked/1/cts.https.

Backout link: https://hg.mozilla.org/integration/autoland/rev/9b04e12f5edd9f428f3c9bd0d6b4b9a99cdc646e

Push where failures started: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=3e7d709ab2a3a86ff58f4406faa05e738ca49ec9&selectedTaskRun=ZNkrNVr_Qz6q5LlNst4UoA.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=418319498&repo=autoland&lineNumber=1868

Comment 20

[Norisz Fay [:noriszfay]]

Backed out changeset 1ceebfa927ec for causing TypeError related webgpu failures

Backout link

Push with failures

Failure log

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/714383968fdb
test(webgpu): simplify when tests runs r=taskgraph-reviewers,bhearsum

Comment 22

[Erich Gubler [:ErichDonGubler]]

Investigating backouts. I'm greatly surprised that the last single revision got backed out, given the positive Try push I noted in comment 14.

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/50babf946f47
build(ci): restrict WebGPU to known working envs. (read: Windows) r=webgpu-reviewers,jgilbert
https://hg.mozilla.org/integration/autoland/rev/d777b0086154
build(ci): chunk WebGPU WPT tests more r=webgpu-reviewers,jgilbert

Comment 24

[Iulian Moraru]

Backed out for causing webgpu and wpt failures.

• Push with failures

• Example of failure log for wpt

• Example of failure log for wpt

• Failure log for webgpu

• Backout link

Comment 25

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/714383968fdb

Comment 26

[Kelsey Gilbert [:jgilbert]]

This still has parts that haven't landed.

Comment 27

[Erich Gubler [:ErichDonGubler]]

Currently investigating why web-platform-tests jobs generated by Taskcluster's decision task are including WebGPU tests. One can observe this with Try pushes like this one, which, like backout pushes above, fail because /_mozilla/webgpu tests are still being included (despite the intent to not do so with bug 1829715, CC :jgilbert). This is the single biggest source of CI failures with current pushes.

Comment 28

[Erich Gubler [:ErichDonGubler]]

Looks like we were never actually filtering out webgpu tests from wpt. This Try push's wpt5 on Linux 18.04 x64 WebRender debug doesn't have any of the changes that haven't yet landed and been backed out, but apparently contains /_mozilla/webgpu test runs. 😭

Sooo...we get to figure out how to actually do that filtering correctly now, since we're specifically accepting that the virtualization: virtual environment specified in the web-platform-tests is broken, and we need virtual-with-gpu instead.

Asked :jgraham about this in Firefox CI on Matrix.

Comment 29

[Erich Gubler [:ErichDonGubler]]

Talked with :jgraham. Current plan is to expose an inverse filtering operation based on WPT tag, i.e., an --exclude-tag=… option or something similar. I'm going to try writing the PR myself, but if that doesn't work as I expect, I can either get support or hand off to :jgraham in his next work day.

Comment 30

[Erich Gubler [:ErichDonGubler]]

Attachment: WIP: Bug 1831263: feat(wpt): add `--exclude-tag`

Comment 31

[Erich Gubler [:ErichDonGubler]]

Attachment: WIP: Bug 1831263: test(wpt): use `--exclude-tag=webgpu` instead of `--exclude=…` for `web-platform-tests` r?#webgpu-reviewers

Depends on D180993

Comment 32

[Phabricator Automation]

Comment on attachment 9339159 [details]
WIP: Bug 1831263: feat(wpt): add --exclude-tag

Revision D180993 was moved to bug 1838742. Setting attachment 9339159 [details] to obsolete.

Comment 33

[Phabricator Automation]

Comment on attachment 9339160 [details]
WIP: Bug 1831263: test(wpt): use --exclude-tag=webgpu instead of --exclude=… for web-platform-tests r?#webgpu-reviewers

Revision D180994 was moved to bug 1838742. Setting attachment 9339160 [details] to obsolete.

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3b0228cca008
build(ci): chunk WebGPU WPT tests more r=webgpu-reviewers,jgilbert

Comment 35

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/3b0228cca008

Comment 36

[Erich Gubler [:ErichDonGubler]]

LOL, shoulda used leave-open.

Comment 37

[Erich Gubler [:ErichDonGubler]]

Attachment: fixup! Bug 1831263: build(ci): restrict WebGPU to known working envs. (read: Windows) r=#webgpu-reviewers

Depends on D179815

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/aa9ee0e3e4e9
build(ci): restrict WebGPU to known working envs. (read: Windows) r=webgpu-reviewers,jgilbert

Comment 39

[Cosmin Sabou [:CosminS]]

Backed out for webgpu unexpected passes.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=G7792yC7T06hpjZYPsC6mA.0&resultStatus=testfailed%2Cbusted%2Cexception&revision=aa9ee0e3e4e9225fc0df7f62a09066fcf68df911

Failure log:

• https://treeherder.mozilla.org/logviewer?job_id=420019674&repo=autoland
• https://treeherder.mozilla.org/logviewer?job_id=420019822&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/b3731aa59f4b75e213c7516d9d202053e59e7028

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e2747753fb5e
build(ci): restrict WebGPU to known working envs. (read: Windows) r=webgpu-reviewers,jgilbert

Comment 41

[Sandor Molnar[:smolnar]]

Backed out for causing webgpu unexpected fails.

Backout link: https://hg.mozilla.org/integration/autoland/rev/37ce560e4c4242d065c31dc5d81368a994123371

Push with failures

Failure log -> TEST-UNEXPECTED-FAIL | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,better_than_supported:* | :limit="maxBufferSize" - assert_unreached:

Comment 42

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/179a87036c2b
build(ci): restrict WebGPU to known working envs. (read: Windows) r=webgpu-reviewers,jgilbert

Comment 43

[Norisz Fay [:noriszfay]]

Backed out for causing webgpu failures

Backout link

Push with failures

Failure log

Comment 44

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/54f3fddcb9bf
build(ci): restrict WebGPU to known working envs. (read: Windows) r=webgpu-reviewers,jgilbert

Comment 45

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/54f3fddcb9bf