1907637 - Extend timeout on WebGPU CTS tests to x5.0

Status: RESOLVED FIXED

(Core :: Graphics: WebGPU, task, P1)

Description

[Erich Gubler [:ErichDonGubler]]

:teoxoy and I were suspicious that WebGPU CTS tests would Just Run To Completion™ if we extended the WPT timeout windows by a sufficient amount. The Chromium team publishes (very) rough timing data for their WebGPU test runs

It turns out, if we match our current WPT long timeouts' to the worst case observed in Chromium by setting a --timeout-multipler=5.0 in invocations to wptrunner, the vast majority of tests Just Run Right™. So, we're filing a patch stack to expose and use the timeout multiplier properly, adjust expected WPT test outcomes for WebGPU CTS, and then promote perma-PASSing tests on all platforms to tier 2 (as we've been doing with other CTS tests).

Comment 1

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1907637 - feat(wpt): expose `--timeout-multiplier` in `mozharness` r=#webgpu-reviewers

Comment 2

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1907637 - test(webgpu): set `--timeout-multiplier=5.0` for CTS r=#webgpu-reviewers!

Comment 3

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1907637 - test(webgpu): `update-expected --preset reset-all` for `try:46d3500fb3c3346d4fc7b66e24cfb1d17a4231ce` r=#webgpu-reviewers!

Comment 4

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1907637 - test(webgpu): promote long tests to tier 2 r=#webgpu-reviewers!

Comment 5

[Teodor Tanasoaia [:teoxoy]]

Attachment: before-and-after.png

Comment 6

[Erich Gubler [:ErichDonGubler]]

:teoxoy's prototypical Try push that proved that we got very interesting results, with ~400 intermittents on Windows[^1]: try:46d3500fb3c3

First Mega Serious™ Try push that actually tries to do this in a landable way: try:4ec08f6bd1a0

[^1]: Other platforms have similar results, but Windows is our team's current focus. 🙂

Comment 7

[Erich Gubler [:ErichDonGubler]]

try:4ec08f6bd1a0 is timing out in tier 2 with some chunks exceeding the task window, but that should be easily resolvable with more chunks and/or more time.

Comment 8

[Erich Gubler [:ErichDonGubler]]

Try push 2: try:8cd549b49657

Comment 9

[Erich Gubler [:ErichDonGubler]]

Try push 3, with adjustments for apparent timeouts from push 2 and a 5 reruns: try:280f3b7323eb

Comment 10

[Erich Gubler [:ErichDonGubler]]

Attachment: Bug 1907637 - style(webgpu): fix up `WebGPUChild.cpp` r=#webgpu-reviewers!

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6be3328e38a0
feat(wpt): expose `--timeout-multiplier` in `mozharness` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/ec4906410b70
test(webgpu): set `--timeout-multiplier=5.0` for CTS r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/58dc9170f7e5
test(webgpu): `update-expected --preset reset-all` for `try:46d3500fb3c3346d4fc7b66e24cfb1d17a4231ce` r=webgpu-reviewers,teoxoy

Comment 12

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/6be3328e38a0
https://hg.mozilla.org/mozilla-central/rev/ec4906410b70
https://hg.mozilla.org/mozilla-central/rev/58dc9170f7e5

Comment 13

[Erich Gubler [:ErichDonGubler]]

Oof, looks like Try push 4 is still failing, with the abort from timeout moving to a new chunk. Using ErichDonGubler/fx-scripts#1 to analyze its first webgpu-backlog5 log with an invocation like this:

use wpt.nu

http get 'https://treeherder.mozilla.org/jobs?repo=try&author=egubler%40mozilla.com&selectedTaskRun=MJNEIlPjT3-wvAmSmZUN4A.0'
  | wpt aggregate-timings webgpu 
  | explore`

…it can be observed that the webgpu:shader,execution,memory_model,weak test is getting cut off while still executing 15 minutes later. 😬 That's a lot of test time inside a single WPT file! Still figuring out how we can ultimately get that landed, but I think we don't have to resolve that to finish this particular bug.

Let's try abandon promotions in that particular file, for now, and create a separate follow-up bug to make it work. It's only 6 out of the ~88 tests we currently have slated for promotion in the latest revision I've got pushed here.

Comment 14

[Erich Gubler [:ErichDonGubler]]

Try push 5: try:57ff5cc81725

Comment 15

[Erich Gubler [:ErichDonGubler]]

RE: comment 13: I've filed bug 1908916.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ceb8667d6fc2
test(webgpu): promote long tests to tier 2 r=webgpu-reviewers,teoxoy

Comment 17

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/ceb8667d6fc2