Bug 1863146 Comment 0 Edit History

Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.

Revision 1 by
on 
Meta bug for points at which we update WebGPU CTS in `mozilla-central`.

## General instructions for updating WebGPU CTS

1. Get a checkout of WebGPU CTS locally. You will want <https://github.com/mozilla/gpuweb-cts> as the repository to update (in the `fx-ci` branch), and <https://github.com/gpuweb/cts> as the upstream to pull from (on the `main` branch).
1. Determine which commit you will be updating to. Let's call this `NEW_CTS_COMMIT` in further steps.
1. Ensure that your local repository's work tree has `NEW_CTS_COMMIT` checked out (i.e., `git switch $NEW_CTS_COMMIT`) and has no untracked files (i.e., `git clean -xdf`).
1. With an up-to-date checkout of `mozilla-central`, navigate to `dom/webgpu/tests/cts/vendor` and follow the instructions printed when you execute `cargo run -- --help` to actually implement the re-vendoring.
1. Converge test expectations again. (This is, unfortunately, the most laborious part. 😅)
    1. Start CI run(s) using `mach try --preset webgpu`. You will be using the results of these runs as essential information before the end of this first-level step.
    1. Analyze changes to tests in `dom/webgpu/tests/cts/checkout/src/webgpu/`. You might also include breakages from CI runs in your analysis, if they are available.
        * If tests have not significantly changed:
            1. Feed reports from your first CI runs into `moz-webgpu-cts process-reports`, starting with `--preset=reset-contradictory …` for your first invocation. Then, use `--preset=merge` to converge expectations again, feeding in reports from as many follow-up CI runs as needed.
            1. Once expectations have converged, file all changes as a single patch.
        * If tests have been significantly changed, more work may be needed. It might be most effective to recreate expectations from scratch.
            1. In your first patch, delete the contents of `testing/web-platform/mozilla/meta/webgpu`.
            1. Feed reports from your CI runs into `moz-webgpu-cts process-reports …`, using  `--preset=reset-all` as your first invocation. Then, use `--preset=merge` to converge expectations again, feeding in reports from as many follow-up CI runs as needed.
            1. Once expectations have converged, file new expectations as a second patch.
1. File your patches, get a review!
Revision 2 by
on 
Meta bug for points at which we update WebGPU CTS in `mozilla-central`.

## General instructions for updating WebGPU CTS

1. Get a checkout of WebGPU CTS locally. You will want <https://github.com/mozilla/gpuweb-cts> as the repository to update (in the `fx-ci` branch), and <https://github.com/gpuweb/cts> as the upstream to pull from (on the `main` branch).
1. Determine which commit you will be updating to. Let's call this `NEW_CTS_COMMIT` in further steps.
1. Ensure that your local repository's work tree has `NEW_CTS_COMMIT` checked out (i.e., `git switch $NEW_CTS_COMMIT`) and has no untracked files (i.e., `git clean -xdf`).
1. With an up-to-date checkout of `mozilla-central`, navigate to `dom/webgpu/tests/cts/vendor` and follow the instructions printed when you execute `cargo run -- --help` to actually implement the re-vendoring.
1. Converge test expectations again. (This is, unfortunately, the most laborious part. 😅) This will be a loop where you:
    1. Start CI run(s) using `mach try --preset webgpu`.
    1. Download reports from your Try pushes and process them.
        1. It's recommended you download these with `treeherder-dl`, using an invocation like `treeherder-dl --artifact 'public/test_info/wptreport.json' try:<revision>`
        1. Update the `expected` property of metadata using `moz-webgpu-cts update-expected --preset reset-all`.
        1. Update the `implementation-status` property of metadata using `moz-webgpu-cts update-backlog promote-perma-passing`.
        1. If tests need to be demoted to tier 3, you may manually add `implementation-status: backlog` to their metadata.
1. File your patches, get a review!
Revision 3 by
on 
Meta bug for points at which we update WebGPU CTS in `mozilla-central`.

## General instructions for updating WebGPU CTS

1. Get a checkout of WebGPU CTS locally. You will want <https://github.com/mozilla/gpuweb-cts> as the repository to update (in the `fx-ci` branch), and <https://github.com/gpuweb/cts> as the upstream to pull from (on the `main` branch).
1. Determine which commit you will be updating to. Let's call this `NEW_CTS_COMMIT` in further steps.
1. Set the `fx-ci-next` branch in the `mozilla/gpuweb-cts` repo on GitHub to `NEW_CTS_COMMIT`, for transparency with folks using the repo.
1. Ensure that your local repository's work tree has `NEW_CTS_COMMIT` checked out (i.e., `git switch $NEW_CTS_COMMIT`) and has no untracked files (i.e., `git clean -xdf`).
1. With an up-to-date checkout of `mozilla-central`, navigate to `dom/webgpu/tests/cts/vendor` and follow the instructions printed when you execute `cargo run -- --help` to actually implement the re-vendoring.
1. Converge test expectations again. (This is, unfortunately, the most laborious part. 😅) This will be a loop where you:
    1. Start CI run(s) using `mach try --preset webgpu`.
    1. Download reports from your Try pushes and process them.
        1. It's recommended you download these with `treeherder-dl`, using an invocation like `treeherder-dl --artifact 'public/test_info/wptreport.json' try:<revision>`
        1. Update the `expected` property of metadata using `moz-webgpu-cts update-expected --preset reset-all`.
        1. Update the `implementation-status` property of metadata using `moz-webgpu-cts update-backlog promote-perma-passing`.
        1. If tests need to be demoted to tier 3, you may manually add `implementation-status: backlog` to their metadata.
1. File your patches, get a review!
1. Once landed, update the `fx-ci` branch in the `mozilla/gpuweb-cts` on GitHub to match `NEW_CTS_COMMIT`.
Revision 4 by
on 
Meta bug for points at which we update WebGPU CTS in `mozilla-central`.

## General instructions for updating WebGPU CTS

1. CC yourself on this bug. When Updatebot files a bug against it (configured in `dom/webgpu/tests/cts/moz.yaml`), take the bug and assign yourself to it.
1. Now, you need to reconcile test expectations with the new version of the CTS. (This is, unfortunately, the most laborious part. 😅) This will be a loop where you:
    1. Start CI run(s) using the preset configured for Updatebot. Updatebot files the first one for you!
    1. Download reports from your Try pushes and process them.
        1. It's recommended you download these with [`treeherder-dl`](https://github.com/erichdongubler-mozilla/treeherder-dl), using an invocation like `treeherder-dl --artifact 'public/test_info/wptreport.json' try:<revision>`
        1. Update the `expected` property of metadata using `moz-webgpu-cts update-expected`. If you are running this on the second run or later, you can use `--preset merge` to converge intermittent expectations more quickly.
        1. Update the `implementation-status` property of metadata using `moz-webgpu-cts update-backlog sync perma-passing`.
        1. Update the bug's patch stack with your latest results.

Back to Bug 1863146 Comment 0