1736177 - Add initial implementation of OffscreenCanvas + WebGL

Status: RESOLVED FIXED

(Core :: Graphics: CanvasWebGL, defect, P3)

Description

[Andrew Osmond [:aosmond] (he/him)]

The OffscreenCanvas WebIDL has fallen behind the standard. We need to update it to be more current before we can hope to ship this feature.

Comment 1

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 1. Rename CanvasRenderingContextHelper::GetContext to GetOrCreateContext.

This patch contains no functional changes but sets us up for refactoring
in a later part in this series. It also helps silence a bunch of
compiler warnings.

Comment 2

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 2. Add variants of CanvasRenderingContextHelper::ToBlob.

This patch contains no functional changes but sets us up for refactoring
in a later part in this series.

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 3. Update OffscreenCanvas WebIDL to be more current.

Comment 4

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 4. Expose missing WebGL interfaces to workers.

Comment 5

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 5. Implement OffscreenCanvas::ConvertToBlob.

Comment 6

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 6. Refactor WebRender display list building integration.

This patch separates out a class, nsICanvasRenderingDisplay, for use on
the display list building pipeline. nsICanvasRenderingContextInternal
inherits from this and all of the canvas contexts implement the APIs the
same as today for display purposes. This will make it easier to provide
a display only class for OffscreenCanvas.

It also standardizes on SupportsWeakPtr/WeakPtr instead of mixing in C++
standard library versions.

Comment 7

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 7. Remove some main thread dependencies from (Client)WebGLContext.

Workers require all runnables to implement nsIDiscardableRunnable. This
is because the worker may want to terminate after going out of scope,
and want to discard any pending events. It should be safe to simply drop
any WebGL related events since we are going to tear down the context /
actors as well.

Also ensure that we record dirty events even without an associated
HTMLCanvasElement so that the OffscreenCanvas is notified properly.

Comment 8

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 8. Remove unused visibility observer from HTMLCanvasElement.

Comment 9

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 9. Add plumbing to snapshot worker canvas contexts from the main thread.

Right now, if we wanted to get a snapshot of an OffscreenCanvas context
on a worker thread from the main thread, we would need to block the main
thread on the worker thread, and from the worker thread, issue a sync
IPC call get the front buffer snapshot.

This patch adds an alternative which allows the main thread to go
directly to the canvas owning thread in the compositor process to get
the snapshot, thereby bypassing the worker thread in content process
entirely. All it needs as the unique ID of the CanvasManagerChild
instance, and the protocol ID of the WebGLChild instance.

This will be used for Firefox screenshots, New Tab tiles, and printing.

Comment 10

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 10. Add OffscreenCanvasDisplayHelper implementation.

This helper class manages the necessary state to facilitate the display
pipeline for an OffscreenCanvas created through HTMLCanvasElement's
transferControlToOffscreen.

Comment 11

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 11. Plumb in OffscreenCanvasDisplayHelper.

This patch integrates OffscreenCanvasDisplayHelper with
HTMLCanvasElement, OffscreenCanvas and nsDisplayCanvas to allow
asynchronous display of an OffscreenCanvas.

Comment 12

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 12. Make HTMLCanvasElement's toDataURL to work with OffscreenCanvas.

Comment 13

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1736177 - Part 12. Fix/mark more OffscreenCanvas related tests as passing.

Comment 17

[Phabricator Automation]

Comment on attachment 9250061 [details]
Bug 1736177 - Part 12. Make HTMLCanvasElement's toDataURL to work with OffscreenCanvas.

Revision D130788 was moved to bug 1744854. Setting attachment 9250061 [details] to obsolete.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/db2347ee8147
Part 1. Rename CanvasRenderingContextHelper::GetContext to GetOrCreateContext. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/70aff7fcd001
Part 2. Add variants of CanvasRenderingContextHelper::ToBlob. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/4a34124d5f4e
Part 3. Update OffscreenCanvas WebIDL to be more current. r=dom-worker-reviewers,jgilbert,smaug
https://hg.mozilla.org/integration/autoland/rev/f400c75c5829
Part 4. Expose missing WebGL interfaces to workers. r=dom-worker-reviewers,jgilbert,smaug
https://hg.mozilla.org/integration/autoland/rev/86f6f6d86c6c
Part 5. Implement OffscreenCanvas::ConvertToBlob. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/bf588f8ffcf1
Part 6. Refactor WebRender display list building integration. r=kvark,jgilbert
https://hg.mozilla.org/integration/autoland/rev/d55f1ee88bb9
Part 7. Remove some main thread dependencies from (Client)WebGLContext. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/6d6c78c31c28
Part 8. Remove unused visibility observer from HTMLCanvasElement. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/2610d4464ad5
Part 9. Add plumbing to snapshot worker canvas contexts from the main thread. r=jgilbert,ipc-reviewers,nika
https://hg.mozilla.org/integration/autoland/rev/49f8b4205a46
Part 10. Add OffscreenCanvasDisplayHelper implementation. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/5be8557c4721
Part 11. Plumb in OffscreenCanvasDisplayHelper. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/828633114de2
Part 12. Fix/mark more OffscreenCanvas related tests as passing. r=jgilbert

Comment 19

[Natalia Csoregi [:nataliaCs]]

Backed out for bp-hybrid failures and others, along with Bug 1738971

• backout: https://hg.mozilla.org/integration/autoland/rev/d38a89ced939d330f8f9cd88926297e7c158ce72

• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=828633114de21a2199e9b267e1f2c80a53edbc79

• failure logs:

  • chrome/common/ipc_message_utils.h:116:33: error: no member named 'Write' in 'IPC::ParamTraits<mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits>>'
  • TEST-UNEXPECTED-FAIL | dom/media/webrtc/tests/mochitests/test_peerConnection_captureStream_canvas_webgl.html | Test timed out. -
  • TEST-UNEXPECTED-FAIL | dom/serviceworkers/test/test_serviceworker_interfaces.html | undefined: If this is failing: DANGER, are you sure you want to expose the new interface WebGLQuery to all webpages as a property on self? Do not make a change to this file without a review from a DOM peer for that specific change!!! (or a JS peer for changes to ecmaGlobals)

Comment 20

[Andrew Osmond [:aosmond] (he/him)]

I think I've gotten the outstanding issues resolved except for the WebRTC one which seems to be out of left field. But it is failing consistently and it is WebGL, so it is plausible I broke something.

Comment 21

[Jim Mathies [:jimm]]

Jib, any ideas here?

Comment 22

[Andrew Osmond [:aosmond] (he/him)]

It appears that it is part 7 that caused it. Perhaps it is related to the changes in ClientWebGLContext::MarkCanvasDirty, as the behaviour is slightly different for setting mCapturedFrameInvalidated to true, and the test case is capture related.

Comment 23

[Andrew Osmond [:aosmond] (he/him)]

Let's see if this does it:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9934a5b81ab4e74f1de8047990b2d9be6f845ac2

Comment 24

[Andrew Osmond [:aosmond] (he/him)]

Looks like that's it.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5ad078db7d7b
Part 1. Rename CanvasRenderingContextHelper::GetContext to GetOrCreateContext. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/6cbaee34e535
Part 2. Add variants of CanvasRenderingContextHelper::ToBlob. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/25ca519aa6fc
Part 3. Update OffscreenCanvas WebIDL to be more current. r=dom-worker-reviewers,jgilbert,smaug
https://hg.mozilla.org/integration/autoland/rev/f5814b14480c
Part 4. Expose missing WebGL interfaces to workers. r=dom-worker-reviewers,jgilbert,smaug
https://hg.mozilla.org/integration/autoland/rev/e8c475e9e8e4
Part 5. Implement OffscreenCanvas::ConvertToBlob. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/26e52340910f
Part 6. Refactor WebRender display list building integration. r=kvark,jgilbert
https://hg.mozilla.org/integration/autoland/rev/2859a91f3a73
Part 7. Remove some main thread dependencies from (Client)WebGLContext. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/047bcbf3769f
Part 8. Remove unused visibility observer from HTMLCanvasElement. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/7e5591aad25a
Part 9. Add plumbing to snapshot worker canvas contexts from the main thread. r=jgilbert,ipc-reviewers,nika
https://hg.mozilla.org/integration/autoland/rev/7b946576bafa
Part 10. Add OffscreenCanvasDisplayHelper implementation. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/bc1b6e82d67f
Part 11. Plumb in OffscreenCanvasDisplayHelper. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/a441516deed7
Part 12. Fix/mark more OffscreenCanvas related tests as passing. r=jgilbert

Comment 26

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/5ad078db7d7b
https://hg.mozilla.org/mozilla-central/rev/6cbaee34e535
https://hg.mozilla.org/mozilla-central/rev/25ca519aa6fc
https://hg.mozilla.org/mozilla-central/rev/f5814b14480c
https://hg.mozilla.org/mozilla-central/rev/e8c475e9e8e4
https://hg.mozilla.org/mozilla-central/rev/26e52340910f
https://hg.mozilla.org/mozilla-central/rev/2859a91f3a73
https://hg.mozilla.org/mozilla-central/rev/047bcbf3769f
https://hg.mozilla.org/mozilla-central/rev/7e5591aad25a
https://hg.mozilla.org/mozilla-central/rev/7b946576bafa
https://hg.mozilla.org/mozilla-central/rev/bc1b6e82d67f
https://hg.mozilla.org/mozilla-central/rev/a441516deed7

Comment 27

[Ruth John]

Adding dev-doc-needed keyword so I can track this for MDN updates