Bug 1822630 Comment 0 Edit History

Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.

Our current vendoring of `gpuweb/cts` is far from perfect WRT its generation of WPT tests to run in CI. There are two related pain points we'd like to resolve with them specifically:

1. Chunked test files contain a fixed number of tests, evenly divided from the set generated by CTS upstream scripting. Each individual chunk requires its own set of metadata for the tests it contains. Both of these facts create a particularly painful problem: when we vendor in updates to the CTS, additions and removals of tests causes:
    1. The chunk that new tests land in _and all subsequent chunks_ to have lines moved between them. This makes it difficult for a human to visually identify tests that have, in fact, been added or removed, instead of being moved between chunks.
    1. Expectation metadata for tests must also be _manually_ moved to match generated changes. This is tedious, because almost every test in the CTS currently fails. 
1. :ErichDonGubler's [initial stab at WebGPU CTS](https://bugzilla.mozilla.org/show_bug.cgi?id=1720941) left [an action item](https://phabricator.services.mozilla.com/D169951#inline-949646) for improving the distribution of task times per :jmaher's request, and we'd like to honor that here. To wit, the WPT tests we generate for WebGPU CTS (viz., `wpt*` jobs in Taskcluster runs like [these](https://treeherder.mozilla.org/jobs?repo=try&revision=970e42abaf6bb3f9fd3dc3a974c5918667c41932)) need to:

    1. Stay within 50 minutes of execution time for optimized builds.
    1. Stay within 30 minutes of execution time for debug builds.
Our current vendoring of [`gpuweb/cts`](https://github.com/gpuweb/cts) is far from perfect WRT the test we generate for CI. There are two related pain points we'd like to resolve around them with this bug:

1. Chunked test files contain a fixed number of tests, evenly divided from the set generated by CTS upstream scripting. Each individual chunk requires its own set of metadata for the tests it contains. Both of these facts create a particularly painful problem: when we vendor in updates to the CTS, additions and removals of tests causes:
    1. The chunk that new tests land in _and all subsequent chunks_ have to have lines moved between them. This generates a high volume of diff noise, making it difficult for a human to visually identify tests that have, in fact, been added or removed (rather than being moved between chunks).
    1. Expectation metadata for tests must also be _manually_ moved to match generated changes, because of the above. This is tedious, because a high percentage of tests still fail, ATOW.
1. :ErichDonGubler's [initial stab at WebGPU CTS](https://bugzilla.mozilla.org/show_bug.cgi?id=1720941) left [an action item](https://phabricator.services.mozilla.com/D169951#inline-949646) for improving the distribution of task times per :jmaher's request, and we'd like to honor that here. To wit, the WPT tests we generate for WebGPU CTS (viz., `wpt*` jobs in Taskcluster runs like [these](https://treeherder.mozilla.org/jobs?repo=try&revision=970e42abaf6bb3f9fd3dc3a974c5918667c41932)) need to:

    1. Stay within 50 minutes of execution time for optimized builds.
    1. Stay within 30 minutes of execution time for debug builds.

Back to Bug 1822630 Comment 0