Closed Bug 1720941 (webgpu-cts) Opened 3 years ago Closed 1 year ago

Integrate WebGPU CTS into CI

Categories

(Core :: Graphics: WebGPU, task)

task

Tracking

()

RESOLVED FIXED
113 Branch
Tracking Status
firefox113 --- fixed

People

(Reporter: kvark, Assigned: ErichDonGubler, Mentored)

References

(Depends on 1 open bug, Blocks 1 open bug, Regressed 5 open bugs, )

Details

Attachments

(2 files, 1 obsolete file)

Previously filed in 1601036, but since then WebGPU CTS was no longer updated in WPT.
We need to re-integrate it into the CI now.

Assignee: nobody → jgilbert

It's annoying that I can't edit the description of a bug I didn't file in Bugzilla! 😕 If I could, I'd use this:

As part of the WebGPU project, we'd like to get the WebGPU Conformance Test Suite (CTS) working against Firefox's implementation. This will be an essential part of stabilizing WebGPU for Firefox.

I've assigned this task to myself (after unwittingly creating a dupe at 1812349), per team discussion with :jimb and :jgilbert. Live reporting on the progress of this task is being done at HackMD.

Assignee: jgilbert → egubler
Blocks: webgpu-v1
No longer blocks: webgpu-mvp
Mentor: jgilbert, jimb
Status: NEW → ASSIGNED
Depends on: 1812352, 1812353
Alias: webgpu-cts
Duplicate of this bug: 1812349
Depends on: 1814088
Depends on: 1814090
Depends on: 1814091
Depends on: 1814094
Depends on: 1814098
Depends on: 1814101

@egubler, to be clear, completion of this bug doesn't depend on all these other bugs, right?
I believe it might be useful to track these in a separate meta-bug, and see-also that bug here.

Flags: needinfo?(egubler)

:jgilbert: I think you're spot on. I'll move them to be dependencies of the new webgpu-v1-cts-blockers bug instead.

Flags: needinfo?(egubler)
Depends on: 1814745

This bug is currently set to be implemented by using the WPT target that WebGPU's CTS repo offers. However, dom.webgpu.enabled et al do not currently live-update correctly, which is necessary for execution in our WPT runner (which sets prefs. on-the-fly). I've filed 1814745 to capture this issue.

Sample output from this script:

Removing old vendored files at `<gecko>\dom\webgpu\tests\cts\checkout`…
Making a vendored copy of checked-in files from `<snip>\gpuweb\cts` (AKA `<cts>`) at `<gecko>\dom\webgpu\tests\cts\checkout`…
Removing old contents of `<cts>\out-wpt`…
Generating WPT test cases into `<cts>\out-wpt` with `npm run wpt`…

> @webgpu/cts@0.1.0 wpt
> grunt wpt

Running "set-quiet-mode" task
Running tasks........................
Build completed! Running checks/tests...........
Refining the output of `<cts>\out-wpt\cts.https.html` with `npm run gen_wpt_cts_html …`…

> @webgpu/cts@0.1.0 gen_wpt_cts_html
> node tools/gen_wpt_cts_html

Removing old contents of `<cts>\out`…
Generating standalone runner files into `<cts>\out` with `npm run standalone`, so we can steal some runtime files for WPT test execution…

> @webgpu/cts@0.1.0 standalone
> grunt standalone

Running "set-quiet-mode" task
Running tasks............
Build completed! Running checks/tests...........
Stealing standalone runtime files from `<cts>\out` for `<cts>\out-wpt`…
  …copying from `<cts>\out\external` to `<cts>\out-wpt\external`…
  …copying from `<cts>\out\common\internal` to `<cts>\out-wpt\common\internal`…
  …copying from `<cts>\out\common\util` to `<cts>\out-wpt\common\util`…
  …Done stealing!
Removing old contents of `<gecko>\testing\web-platform\tests\webgpu`…
Copying contents of `<cts>\out-wpt` to `<gecko>\testing\web-platform\tests\webgpu`…
All done! Now get your CTS _ON_! :)
Attachment #9317919 - Attachment description: WIP: Bug 1720941: build(ci): integrate WebGPU CTS into CI → WIP: Bug 1720941, part 1: build(ci): integrate WebGPU CTS into CI
Attachment #9317923 - Attachment description: WIP: Bug 1720941: more chunking to compensate for WebGPU CTS → WIP: Bug 1720941, part 2: tests(ci): now do the actual vendoring
Attachment #9317921 - Attachment is obsolete: true
Depends on: 1814222
Attachment #9317919 - Attachment description: WIP: Bug 1720941, part 1: build(ci): integrate WebGPU CTS into CI → Bug 1720941, part 1: build(ci): integrate WebGPU CTS into CI r=#webgpu-reviewers
Attachment #9317923 - Attachment description: WIP: Bug 1720941, part 2: tests(ci): now do the actual vendoring → Bug 1720941, part 2: tests(ci): now do the actual vendoring r=#webgpu-reviewers
See Also: → 1822630
Pushed by egubler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a211d76a5e01
part 1: build(ci): integrate WebGPU CTS into CI r=webgpu-reviewers,taskgraph-reviewers,jmaher,jimb
https://hg.mozilla.org/integration/autoland/rev/f7fed5fde8b7
part 2: tests(ci): now do the actual vendoring r=webgpu-reviewers,taskgraph-reviewers,jmaher,jimb

Backed out 2 changesets (Bug 1720941) for web-platform tests failures.
Backout link
Push with failures <--> wpt6
Failure Log
Also wpt8 Failure Log

Flags: needinfo?(egubler)

Actively investigating as of yesterday. Will report here later.

Flags: needinfo?(egubler)

It seems that I erroneously script-changed some portions of the test metadata while making adaptations for Mac, i.e.:

[:isAsync=true;format="depth24plus-stencil8";faceAndOpType="frontFailOp";op="_undef_"]
    expected:
      expected: FAIL

…which is definitely wrong. Landing with this fixed. 😅


This took a lot longer to diagnose than I would have liked. I wish that the Python error being raised here mentioned a source path and line number. That would have expedited debugging to take seconds, instead of hours. I've filed this bug: 1823802

See Also: → 1823802
See Also: 1823802
See Also: → 1823802
Pushed by egubler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fb5712afe8b1
part 1: build(ci): integrate WebGPU CTS into CI r=webgpu-reviewers,taskgraph-reviewers,jmaher,jimb
https://hg.mozilla.org/integration/autoland/rev/292a7aff8df7
part 2: tests(ci): now do the actual vendoring r=webgpu-reviewers,taskgraph-reviewers,jmaher,jimb
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → 113 Branch
Regressions: 1824042
Regressions: 1824038
Regressions: 1825419
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: