1909753 - Update WebGPU CTS (2024-07)

Status: RESOLVED FIXED

(Core :: Graphics: WebGPU, task, P1)

Comment 1

[Jim Blandy :jimb]

[edit: wrong bug]

Comment 2

[Jim Blandy :jimb]

Wrong bug!

Comment 3

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 0.1 - test(webgpu): eliminate subtests no longer being reported r=#webgpu-reviewers

Comment 4

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 0.2 - test(webgpu): remove `expected` values for `webgpu:api,validation,render_pass,render_pass_descriptor:attachments,one_color_attachment:*` r=#webgpu-reviewers

Comment 5

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 0.3 - demote non-`PASS`ing tests to tier 2 r=#webgpu-reviewers

Not sure how we hadn't caught these before. 😅

Comment 6

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 0.4 - align `run-on-projects` in `webgpu-long` task with main `webgpu` task r=#webgpu-reviewers

Comment 7

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 1 - update WebGPU CTS to 6c58a195839034d9f72bdc125cfe8dabcce7336d r=#webgpu-reviewers

Comment 8

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.1 - eliminate tests removed during upgrade r=#webgpu-reviewers

Comment 9

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.2 - migrate `adapter,requestAdapterInfo` tests to `adapter,info` r=#webgpu-reviewers

Comment 10

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, step 2.3 - add test cases from new CTS that pass r=#webgpu-reviewers

• webgpu:shader,execution,expression,call,builtin,textureNumLevels:depth:*, implemented by gpuweb/cts#3855.
• webgpu:shader,execution,expression,call,builtin,textureLoad:storage_texel_formats:*, introduced by gpuweb/cts#3849.
• webgpu:shader,execution,expression,unary,address_of_and_indirection:deref_swizzle:*, introduced by gpuweb/cts#3869.
• webgpu:shader,validation,expression,call,builtin,pack2x16float:*, introduced by TODO.
• webgpu:shader,validation,expression,call,builtin,texture{Dimensions,Num{Layers,Levels,Samples}}:*, introduced by gpuweb/cts#3689.
• webgpu:shader,validation,statement,continue:*, introduced by gpuweb/cts#3688.
• webgpu:shader,validation,statement,{if,loop,return,switch,while}:*, introduced by gpuweb/cts#3645 and moved around by gpuweb/cts#3742.
• webgpu:shader,validation,types,pointer:missing_type:*, introduced by gpuweb/cts#3696.

This list is likely incomplete, because I really needed to make this tractable. Sorry! I upkept the above for my own sanity, and think it's interesting for record-keeping, but not necessary useful for a reviewer.

Comment 11

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.4 - accept "regression" in `webgpu:shader,validation,types,textures:sampled_texture_types:*` r=#webgpu-reviewers

This appears to be because of new trailing comma subcases (which aren't represented in the WPT test path) from gpuweb/cts#3696, rather than an actual regression.

Comment 12

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.5 - trivially moved tests, no action required r=#webgpu-reviewers

• webpgpu:api,validation,buffer,mapping:getMappedRange,disjoinRanges_many was renamed to disjointRanges_many (see also @jimb's PR making the move at gpuweb/cts#3838)
• webgpu:shader,validation,decl,ptr_spelling:* was merged with webgpu:shader,validation,types,pointer:* in gpuweb/cts#3714.
• webgpu:shader,validation,{decl,var_access_mode,parse,var_and_let}:* → webgpu:shader,validation,decl,var:*, merged in [gpuweb/cts#3713](https:// github.com/gpuweb/cts/pull/3713).
  • …:initializer_type:* had its structure changed to accommodate the merge.
• webgpu:shader,validation,parse,align:* → webgpu:shader,validation,shader_io,align:*, moved by gpuweb/cts#3712,
• webgpu:shader,validation,parse,{break,break_if,compound,const_assert,continuing,discard,statement_behavior}, moved by gpuweb/cts#3742
  • webgpu:shader,validation,parse,break_if:non_bool_param:* → webgpu:shader,validation,statement,break_if:condition_type:*, moved by gpuweb/cts#3671, affected by gpuweb/cts#3690.
• webgpu:shader,validation,parse_binary_ops:* → webgpu:shader,validation,expression,binary,parse:*

This list is likely incomplete, because I really needed to make this tractable. Sorry! I upkept the above for my own sanity, and think it's interesting for record-keeping, but not necessary useful for a reviewer.

Comment 13

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.6 - new `backlog` tests r=#webgpu-reviewers

This (non-exhaustively) adds or implements the following test paths:

• webgpu:api,validation,render_pipeline,depth_stencil_state:depth_bias:*, introduced by gpuweb/cts#3863. Tracked with bug 1911312.

• webgpu:shader,execution,expression,access,array,index:matrix:*, introduced by gpuweb/cts#3738.

• webgpu:shader,execution,expression,call,builtin,atomics,atomicSub:sub_i32_min:*, introduced by gpuweb/cts#3728.

• webgpu:shader,execution,expression,call,builtin,fwidth{,Coarse,Fine}:f32:*, implemented by gpuweb/cts#3686.

• webgpu:shader,execution,expression,call,builtin,textureNumLayers:arrayed:*, implemented by gpuweb/cts#3853.

• webgpu:shader,execution,expression,call,builtin,textureNumLevels:sampled:*, implemented by gpuweb/cts#3855.

• webgpu:shader,execution,expression,call,builtin,textureNumSamples:sampled:*, implemented by gpuweb/cts#3856.

• webgpu:shader,execution,expression,call,builtin,textureLoad:*, with:

  • {arrayed,sampled_1d,sampled_3d} implemented by gpuweb/cts#3852
  • sampled_2d implemented by gpuweb/cts#3861.

• webgpu:shader,execution,expression,call,builtin,textureSample:*, apparently changed by gpuweb/cts#3877.

• webgpu:shader,execution,expression,call,builtin,textureStore:* had the following tests added:

  • By gpuweb/cts#3781:
    • out_of_bounds
    • out_of_bounds_array
  • By gpuweb/cts#3841:
    • bgra8unorm_swizzle
    • texel_formats

  It's unclear why these are failing, but I suspect part 2.7.

• webgpu:shader,execution,expression,constructor,zero_value:vector_prefix:*, introduced by gpuweb/cts#3734.

• webgpu:shader,validation,decl,var:address_space_access_mode:*, introduced by gpuweb/cts#3695.

• webgpu:shader,validation,decl,var:shader_stage:*, introduced by gpuweb/cts#3736.

• webgpu:shader,validation,expression,access,vector:* was modified in gpuweb/cts#3708:

  • renamed …:vector:* to concrete
  • added abstract

• webgpu:shader,validation,expression,binary,*:

  • …,bitwise_shift:*:
    • …:partial_eval_errors:*, introduced by gpuweb/cts#3796
    • …:{shift_left_concrete,shift_right_concrete}:* had new cases introduced in gpuweb/cts#3726.
    • …:{shift_left_abstract,shift_right_abstract}:*, introduced by gpuweb/cts#3833.
  • …,{add_sub_mul:*,and_or_xor:*,bitwise_shift:scalar_vector:*,div_rem:scalar_vector:*} modified by gpuweb/cts#3648.

• webgpu:shader,validation,expression,call,builtin,*:

  • …,dot:*, introduced in gpuweb/cts#3579.
  • …,fma:*, introduced by gpuweb/cts#3577.
  • …,ldexp:*:
    • …:{args,must_use,values}:* introduced by gpuweb/cts#3615.
    • …:partial_values:* introduced by gpuweb/cts#3798.
  • …,mix:*, introduced by TODO.
  • …,modf:*, introduced by TODO.
  • …,pow:*, introduced by TODO.
  • …,refract:*, introduced by TODO.
  • …,saturate:*, introduced by TODO.
  • …,texture{Dimensions,Num{Layers,Levels,Samples}}:*, introduced by gpuweb/cts#3689.

• webgpu:shader,validation,functions,alias_analysis:swizzles:*, introduced by gpuweb/cts#3869.

• webgpu:shader,validation,shader_io,align:parsing:duplicate, introduced by gpuweb/cts#3692.

• webgpu:shader,validation,statement,{for,loop,return}:* cases introduced by gpuweb/cts#3645 and moved around by gpuweb/cts#3742.

• webgpu:shader,validation,types,pointer:{access_mode,address_space,type}:*, introduced by gpuweb/cts#3649.

This list is incomplete, because I really needed to make this tractable. Sorry! I upkept the above for my own sanity, and think it's interesting for record-keeping, but not necessary useful for a reviewer.

Comment 14

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.7 - accept implemented `webgpu:api,operation,uncapturederror:*` r=#webgpu-reviewers

• webgpu:api,operation,uncapturederror:constructor:* was unimplemented, and then removed.
• webgpu:api,operation,uncapturederror:iff_uncaptured:* was finally implemented (which trivially PASSed before), and fails in the out-of-memory case.

Comment 15

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.8 - accept WebGPU CTS regressions to `FAIL` due to `@interpolate(…, either)` r=#webgpu-reviewers

These failures (non-exhaustively) include:

• webgpu:api,operation,uncapturederror:*
• webgpu:api,rendering,depth_clip_clamp:*
• webgpu:api,validation,render_pipeline,inter_stage:interpolation_sampling:*

This list is incomplete, because I really needed to make this tractable. Sorry! I upkept the above for my own sanity, and think it's interesting for record-keeping, but not necessary useful for a reviewer.

These started breaking in gpuweb/cts#3814. Neither we nor Chrome stable (127.0.6533.89) pass this test ATOW, so 🤷🏻‍♂️ whatevs!

Comment 16

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.9 - accept WebGPU CTS regressions to `FAIL` due to upstream `trackForCleanup all buffers and textures` PR r=#webgpu-reviewers

Quite a few tests regressed, with the only recent change being gpuweb/cts#3821. These failures (non-exhaustively) include:

• webgpu:api,validation,buffer,create:{size,usage}:*
• webgpu:compat,api,validation,encoding,programmable,pipeline_bind_group_compat:twoDifferentTextureViews,{render_pass,compute_pass},{used,unused}:*

This list is incomplete, because I really needed to make this tractable. Sorry! I upkept the above for my own sanity, and think it's interesting for record-keeping, but not necessary useful for a reviewer

Comment 17

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.10 - move intermittent `TIMEOUT`/`NOTRUN` tests to `webgpu-long` r=#webgpu-reviewers

Comment 18

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.11 - accept WebGPU CTS regressions that `FAIL` after `compat` tests use `expectValidationErrorInCompatibilityMode` r=#webgpu-reviewers

I found the following:

• webgpu:compat,api,validation,encoding,programmable,pipeline_bind_group_compat:twoDifferentTextureViews,{render_pass,compute_pass},used:*
  • These started actually failing in part 2.9, but are also affected here.
• webgpu:compat,api,validation,render_pipeline,fragment_state:colorState:*

…and there may be more, but I really needed to make this tractable. Sorry! I upkept the above for my own sanity, and think it's interesting for record-keeping, but not necessary useful for a reviewer.

Comment 19

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.12 migrate `webgpu:compat,api,validation,shader_module,shader_module:*` to `webgpu:compat,api,validation,render_pipeline,shader_module:*` r=#webgpu-reviewers

This movement was introduced in gpuweb/cts#3769.

These would have been included in part 2.7, except that a breakage via @interpolate(…, either) was introduced (see also part 2.7).

Comment 20

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.13 split `webgpu-long` task into 2 chunks r=#webgpu-reviewers

With so many tests starting to enter the webgpu-long track, the tasks are starting to take > 70 minutes on Windows. Sounds like a good time to split 'em up!

There's enough of these coming into webgpu-long that I expect we'll be making it worth CI time on all platforms shortly.

Comment 21

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.14 add `web-platform-tests-webgpu-backlog-log` tests r=#webgpu-reviewers

Comment 22

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.15 tag observedly long-running WebGPU CTS tests as `webgpu-long` r=#webgpu-reviewers

If I saw a test that took longer than 1 min. to execute, than I marked it as webgpu-long. Works pretty well; one can see that non-*-long WebGPU CI task chunks are executing very consistently now. 😀

Comment 23

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.16 demote long tests discovered to be failing to `backlog` r=#webgpu-reviewers

Comment 24

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.17 promote tests whose structure didn't change, but now pass r=#webgpu-reviewers

Comment 25

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1909753, part 2.18 misc. `backlog` changes r=#webgpu-reviewers

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cb8a738d5589
part 0.1 - test(webgpu): eliminate subtests no longer being reported r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/e0d45e295c6d
part 0.2 - test(webgpu): remove `expected` values for `webgpu:api,validation,render_pass,render_pass_descriptor:attachments,one_color_attachment:*` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/1a80f8baa124
part 0.3 - demote non-`PASS`ing tests to tier 2 r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/5ac683064d3d
part 0.4 - align `run-on-projects` in `webgpu-long` task with main `webgpu` task r=webgpu-reviewers,teoxoy

Comment 27

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/cb8a738d5589
https://hg.mozilla.org/mozilla-central/rev/e0d45e295c6d
https://hg.mozilla.org/mozilla-central/rev/1a80f8baa124
https://hg.mozilla.org/mozilla-central/rev/5ac683064d3d

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f296491b5136
part 1 - update WebGPU CTS to 6c58a195839034d9f72bdc125cfe8dabcce7336d r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/2868446de6df
part 2.1 - eliminate tests removed during upgrade r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/353c8cbdfa7a
part 2.2 - migrate `adapter,requestAdapterInfo` tests to `adapter,info` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/860d96bdcd69
step 2.3 - add test cases from new CTS that pass r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/4ea84dd05c32
part 2.4 - accept "regression" in `webgpu:shader,validation,types,textures:sampled_texture_types:*` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/e7b536d026a2
part 2.5 - trivially moved tests, no action required r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/cc8e4c0d5276
part 2.6 - new `backlog` tests r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/5424ce25eebe
part 2.7 - accept implemented `webgpu:api,operation,uncapturederror:*` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/46842e355b83
part 2.8 - accept WebGPU CTS regressions to `FAIL` due to `@interpolate(…, either)` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/8df332be3ac3
part 2.9 - accept WebGPU CTS regressions to `FAIL` due to upstream `trackForCleanup all buffers and textures` PR r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/0f0a6f299665
part 2.10 - move intermittent `TIMEOUT`/`NOTRUN` tests to `webgpu-long` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/c9348783d3bf
part 2.11 - accept WebGPU CTS regressions that `FAIL` after `compat` tests use `expectValidationErrorInCompatibilityMode` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/e8f44cadcc6a
part 2.12 migrate `webgpu:compat,api,validation,shader_module,shader_module:*` to `webgpu:compat,api,validation,render_pipeline,shader_module:*` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/3d77ae1a68ea
part 2.13 split `webgpu-long` task into 2 chunks r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/5e668e6fa9e1
part 2.14 add `web-platform-tests-webgpu-backlog-log` tests r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/160efb2294aa
part 2.15 tag observedly long-running WebGPU CTS tests as `webgpu-long` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/545b3601f6ff
part 2.16 demote long tests discovered to be failing to `backlog` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/aeff4370e885
part 2.17 promote tests whose structure didn't change, but now pass r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/590598212025
part 2.18 misc. `backlog` changes r=webgpu-reviewers,teoxoy

Comment 29

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/f296491b5136
https://hg.mozilla.org/mozilla-central/rev/2868446de6df
https://hg.mozilla.org/mozilla-central/rev/353c8cbdfa7a
https://hg.mozilla.org/mozilla-central/rev/860d96bdcd69
https://hg.mozilla.org/mozilla-central/rev/4ea84dd05c32
https://hg.mozilla.org/mozilla-central/rev/e7b536d026a2
https://hg.mozilla.org/mozilla-central/rev/cc8e4c0d5276
https://hg.mozilla.org/mozilla-central/rev/5424ce25eebe
https://hg.mozilla.org/mozilla-central/rev/46842e355b83
https://hg.mozilla.org/mozilla-central/rev/8df332be3ac3
https://hg.mozilla.org/mozilla-central/rev/0f0a6f299665
https://hg.mozilla.org/mozilla-central/rev/c9348783d3bf
https://hg.mozilla.org/mozilla-central/rev/e8f44cadcc6a
https://hg.mozilla.org/mozilla-central/rev/3d77ae1a68ea
https://hg.mozilla.org/mozilla-central/rev/5e668e6fa9e1
https://hg.mozilla.org/mozilla-central/rev/160efb2294aa
https://hg.mozilla.org/mozilla-central/rev/545b3601f6ff
https://hg.mozilla.org/mozilla-central/rev/aeff4370e885
https://hg.mozilla.org/mozilla-central/rev/590598212025