1753302 - Make WebGPU accessible via OffscreenCanvas

Status: RESOLVED FIXED

(Core :: Graphics: WebGPU, enhancement, P3)

Comment 1

[Andrew Osmond [:aosmond] (he/him)]

Current implementation:

In CanvasContext::CanvasContext, we allocate an external image ID for the canvas:
https://searchfox.org/mozilla-central/rev/90e5ec580b4334c54829d83d79222d0b3ca4c5df/dom/webgpu/CanvasContext.cpp#35

In WebGPUParent::RecvDeviceCreateSwapChain, we register the external image ID with a RenderTextureHost used by WR:
https://searchfox.org/mozilla-central/rev/90e5ec580b4334c54829d83d79222d0b3ca4c5df/dom/webgpu/ipc/WebGPUParent.cpp#541

In WebGPUParent::RecvSwapChainPresent, we request a pixel readback from the underlying library:
https://searchfox.org/mozilla-central/rev/90e5ec580b4334c54829d83d79222d0b3ca4c5df/dom/webgpu/ipc/WebGPUParent.cpp#708

In PresentCallback, we copy the yielded pixel rows into the exposed RenderTextureHost pixels:
https://searchfox.org/mozilla-central/rev/90e5ec580b4334c54829d83d79222d0b3ca4c5df/dom/webgpu/ipc/WebGPUParent.cpp#577

In CanvasContext::UpdateWebRenderLocalCanvasData, called during the display list generation, yields the external image ID:
https://searchfox.org/mozilla-central/rev/90e5ec580b4334c54829d83d79222d0b3ca4c5df/dom/webgpu/CanvasContext.cpp#132

In CanvasContext::Configure, we invalidate the canvas element, the only time we explicitly do so (meaning we can only get the first frame at best):
https://searchfox.org/mozilla-central/rev/90e5ec580b4334c54829d83d79222d0b3ca4c5df/dom/webgpu/CanvasContext.cpp#90

There are a few problems here. One, we don't refresh the canvas when there are frame updates. Two, we are very racy and WR might end up sampling a half copied pixel readback. My initial thought would be to switch things to work more like OffscreenCanvas does for worker threads (ImageBridge + AsyncImagePipeline) for both the main thread and worker thread. We could then update without a display list rebuild / depending on the main thread. Presumably we should have some sort of front/back buffer to allow consistent frame updates as well.

Comment 2

[Andrew Osmond [:aosmond] (he/him)]

In WebRenderCommandBuilder::EmptyTransaction, we also request a readback:

https://searchfox.org/mozilla-central/rev/90e5ec580b4334c54829d83d79222d0b3ca4c5df/gfx/layers/wr/WebRenderCommandBuilder.cpp#1534

I don't think we tell WR the texture updated anywhere though, so caching may prevent us from picking up the change.

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

To get something working on/off the main thread, and fix some of the race conditions that exist today, we can:

• Use async pipelines for display instead of an external image ID; this will allow us to produce frames without relying upon a display list update (including off the main thread)
• Create a new memory-backed TextureHost that can be serialized to a unique ID that the content process can send back to the parent, instead of the pixel data
• When creating buffers on the swap chain update / PresentCallback path, we could use recycled buffers to avoid the allocation cost; we also don't want to update the live buffer used by WR to avoid frame consistency issues
• When PresentCallback completes, it needs to notify the owning PWebGPU so that the child can receive the serialized TextureHost to pipe it into the async pipeline / PImageBridge to schedule a composition

Typically today, we only serialize native platform handles but it would be useful to have an abstraction for memory backed textures, when we don't even need the pixel data in the content process most of the time. This could also be useful for WebGL as well when we cannot use a native platform handle to avoid the copy to/from the content process. We may need to add abstractions to be able to read the pixel data back for these buffers however.

In an ideal world we would not even rely upon the content process do mirror back the IDs, and trigger the scheduling of the composition, but I think this best fits into the architecture we have today.

Comment 4

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1753302 - Part 1. Integrate WebGPU with OffscreenCanvas plumbing.

Comment 5

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1753302 - Part 2. Expose WebGPU on DOM workers.

Comment 6

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1753302 - Part 3. Add basic worker thread tests for WebGPU.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/36f8c9b4065d
Part 1. Integrate WebGPU with OffscreenCanvas plumbing. r=gfx-reviewers,jgilbert
https://hg.mozilla.org/integration/autoland/rev/1a0de7cfcc18
Part 2. Expose WebGPU on DOM workers. r=emilio
https://hg.mozilla.org/integration/autoland/rev/688cbe362065
Part 3. Add basic worker thread tests for WebGPU. r=gfx-reviewers,jgilbert

Comment 8

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/36f8c9b4065d
https://hg.mozilla.org/mozilla-central/rev/1a0de7cfcc18
https://hg.mozilla.org/mozilla-central/rev/688cbe362065