1720941 - (webgpu-cts) Integrate WebGPU CTS into CI

Status: RESOLVED FIXED

(Core :: Graphics: WebGPU, task)

Description

[Dzmitry Malyshau [:kvark]]

Previously filed in 1601036, but since then WebGPU CTS was no longer updated in WPT.
We need to re-integrate it into the CI now.

Comment 2

[Erich Gubler [:ErichDonGubler]]

It's annoying that I can't edit the description of a bug I didn't file in Bugzilla! 😕 If I could, I'd use this:

As part of the WebGPU project, we'd like to get the WebGPU Conformance Test Suite (CTS) working against Firefox's implementation. This will be an essential part of stabilizing WebGPU for Firefox.

I've assigned this task to myself (after unwittingly creating a dupe at 1812349), per team discussion with :jimb and :jgilbert. Live reporting on the progress of this task is being done at HackMD.

Comment 4

[Kelsey Gilbert [:jgilbert]]

@egubler, to be clear, completion of this bug doesn't depend on all these other bugs, right?
I believe it might be useful to track these in a separate meta-bug, and see-also that bug here.

Comment 5

[Erich Gubler [:ErichDonGubler]]

:jgilbert: I think you're spot on. I'll move them to be dependencies of the new webgpu-v1-cts-blockers bug instead.

Comment 6

[Erich Gubler [:ErichDonGubler]]

This bug is currently set to be implemented by using the WPT target that WebGPU's CTS repo offers. However, dom.webgpu.enabled et al do not currently live-update correctly, which is necessary for execution in our WPT runner (which sets prefs. on-the-fly). I've filed 1814745 to capture this issue.

Comment 7

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1720941, part 1: build(ci): integrate WebGPU CTS into CI r=#webgpu-reviewers

Sample output from this script:

Removing old vendored files at `<gecko>\dom\webgpu\tests\cts\checkout`…
Making a vendored copy of checked-in files from `<snip>\gpuweb\cts` (AKA `<cts>`) at `<gecko>\dom\webgpu\tests\cts\checkout`…
Removing old contents of `<cts>\out-wpt`…
Generating WPT test cases into `<cts>\out-wpt` with `npm run wpt`…

> @webgpu/cts@0.1.0 wpt
> grunt wpt

Running "set-quiet-mode" task
Running tasks........................
Build completed! Running checks/tests...........
Refining the output of `<cts>\out-wpt\cts.https.html` with `npm run gen_wpt_cts_html …`…

> @webgpu/cts@0.1.0 gen_wpt_cts_html
> node tools/gen_wpt_cts_html

Removing old contents of `<cts>\out`…
Generating standalone runner files into `<cts>\out` with `npm run standalone`, so we can steal some runtime files for WPT test execution…

> @webgpu/cts@0.1.0 standalone
> grunt standalone

Running "set-quiet-mode" task
Running tasks............
Build completed! Running checks/tests...........
Stealing standalone runtime files from `<cts>\out` for `<cts>\out-wpt`…
  …copying from `<cts>\out\external` to `<cts>\out-wpt\external`…
  …copying from `<cts>\out\common\internal` to `<cts>\out-wpt\common\internal`…
  …copying from `<cts>\out\common\util` to `<cts>\out-wpt\common\util`…
  …Done stealing!
Removing old contents of `<gecko>\testing\web-platform\tests\webgpu`…
Copying contents of `<cts>\out-wpt` to `<gecko>\testing\web-platform\tests\webgpu`…
All done! Now get your CTS _ON_! :)

Comment 8

[Erich Gubler [:ErichDonGubler]]

Attachment: WIP: Bug 1720941: build(ci): update WebGPU CTS test cases resolved by unblocking CI's usage of `dom.webgpu.enabled`

Depends on D169951

Comment 9

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1720941, part 2: tests(ci): now do the actual vendoring r=#webgpu-reviewers

Depends on D169952

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a211d76a5e01
part 1: build(ci): integrate WebGPU CTS into CI r=webgpu-reviewers,taskgraph-reviewers,jmaher,jimb
https://hg.mozilla.org/integration/autoland/rev/f7fed5fde8b7
part 2: tests(ci): now do the actual vendoring r=webgpu-reviewers,taskgraph-reviewers,jmaher,jimb

Comment 11

[Marian-Vasile Laza]

Backed out 2 changesets (Bug 1720941) for web-platform tests failures.
Backout link
Push with failures <--> wpt6
Failure Log
Also wpt8 Failure Log

Comment 12

[Erich Gubler [:ErichDonGubler]]

Actively investigating as of yesterday. Will report here later.

Comment 13

[Erich Gubler [:ErichDonGubler]]

It seems that I erroneously script-changed some portions of the test metadata while making adaptations for Mac, i.e.:

[:isAsync=true;format="depth24plus-stencil8";faceAndOpType="frontFailOp";op="_undef_"]
    expected:
      expected: FAIL

…which is definitely wrong. Landing with this fixed. 😅

---

This took a lot longer to diagnose than I would have liked. I wish that the Python error being raised here mentioned a source path and line number. That would have expedited debugging to take seconds, instead of hours. I've filed this bug: 1823802

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fb5712afe8b1
part 1: build(ci): integrate WebGPU CTS into CI r=webgpu-reviewers,taskgraph-reviewers,jmaher,jimb
https://hg.mozilla.org/integration/autoland/rev/292a7aff8df7
part 2: tests(ci): now do the actual vendoring r=webgpu-reviewers,taskgraph-reviewers,jmaher,jimb

Comment 15

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/fb5712afe8b1
https://hg.mozilla.org/mozilla-central/rev/292a7aff8df7