Open Bug 1390089 (offscreen-canvas) Opened 5 years ago Updated 19 days ago

[meta] OffscreenCanvas

Categories

(Core :: Graphics: CanvasWebGL, enhancement, P3)

enhancement

Tracking

()

People

(Reporter: jujjyl, Assigned: aosmond)

References

(Depends on 17 open bugs)

Details

(Keywords: meta, parity-chrome, Whiteboard: [gfx-noted])

Bug 709490 tracked initial implementation of OffscreenCanvas, after which it is still behind a pref. We don't seem to have a metabug tracking the remaining work to be able to ship unpreffed, so here is one.
There are also bug 1241255 and bug 1241256 for tooling, but I don't think those are requirements to be able to ship unpreffed.
Depends on: 1391301
Whiteboard: [gfx-noted]
Depends on: 1409789
Depends on: 1422862
Alias: offscreen-canvas
Depends on: 1443674
Depends on: 1443671
Depends on: 1449398
Depends on: 1529995
Depends on: 1564784

This shows up as a popular feature on the Web in crawls for Chrome APIs that Firefox doesn't implement.

Webcompat Priority: --- → ?
Keywords: parity-chrome
Depends on: 1593870
Depends on: 1596772

(In reply to Jukka Jylänki from comment #0)

Bug 709490 tracked initial implementation of OffscreenCanvas, after which it
is still behind a pref. We don't seem to have a metabug tracking the
remaining work to be able to ship unpreffed, so here is one.

I'd like to point out that user gesture requirements equivalent to Chrome's "User Activation v2" are important for some OffscreenCanvas use cases. In our case we can host our entire game engine in a web worker, forwarding input events over postMessage, and posting back for certain DOM APIs not available in workers (e.g. requestFullscreen). If user activation is limited to synchronous callbacks in input events, this blocks the use of any input-restricted APIs (like requestFullscreen), which would block our use of OffscreenCanvas. With User Activation v2, the restrictions are relaxed enough that it keeps working and means we can use OffscreenCanvas. I'm not sure what the status of user gestures are in Firefox but it might be worth reviewing this for OffscreenCanvas.

(In reply to ashley from comment #4)

I'd like to point out that user gesture requirements equivalent to Chrome's "User Activation v2" are important for some OffscreenCanvas use cases.

How did you determine that?

Which specification describes user gesture requirements for OffscreenCanvas?

In our case we can host our entire game engine in a web worker, forwarding input events over postMessage, and posting back for certain DOM APIs not available in workers (e.g. requestFullscreen).

Used a similar approach at https://plnkr.co/edit/4Tb91b?p=info to workaround Firefox, Nightly having not implemented OffscreenCanvasRenderingContext2D for the purpose of manipulating pixels in a Worker ("However there are currently no reliable way to render a video on an OffscreenCanvas." https://discourse.wicg.io/t/proposal-offscreenvideo/3952).

The image drawn onto <canvas> using drawImage() when "2d" context is set or transferFromImageBitmap() when ImageBitmapRenderingContext is not limited by user activation, the issue is the drawn image is not visible (tranparent; width and height might not be set at the <canvas>) on the <canvas>, see https://bugzilla.mozilla.org/show_bug.cgi?id=1564784#c5.

If user activation is limited to synchronous callbacks in input events, this blocks the use of any input-restricted APIs (like requestFullscreen), which would block our use of OffscreenCanvas. With User Activation v2, the restrictions are relaxed enough that it keeps working and means we can use OffscreenCanvas. I'm not sure what the status of user gestures are in Firefox but it might be worth reviewing this for OffscreenCanvas.

Chromium implementation of user activation is WIP and AFAICT not based on interoperability with any other implementation of an API; e.g., https://bugs.chromium.org/p/chromium/issues/detail?id=1014171.

Not sure why the image is not displayed when OffscreenCanvas or when ImageBitmapRenderingContext are used, even though canvas.toDataURL() outputs a data URL of the image (.toBlob() does not).

(In reply to ashley from comment #4)

I'd like to point out that user gesture requirements equivalent to Chrome's "User Activation v2" are important for some OffscreenCanvas use cases.

This code does not require user activation https://plnkr.co/edit/0By56O?p=preview. Not certain how user activation criteria can be applied to an OffscreenCanvas within a Worker, particularly at Firefox, Nightly.

The observable bug with the code in the above-linked plnkr is that transferFromImageBitmap() "does not work": width and height of <canvas> on main thread does not change, image is not visible on the <canvas>

Main thread

    <canvas style="border:1px solid purple"></canvas>
    <script>
      const canvas = document.querySelector("canvas");
      const offscreen = canvas.transferControlToOffscreen();
      worker.postMessage({offscreen}, [offscreen]);
   </script>

Worker

    const bitmap = await createImageBitmap(request);
    const {width, height} = bitmap;
    const osc = e.data.offscreen;
    osc.width = width;
    osc.height = height;
    const osctx = osc.getContext("bitmaprenderer");
    // the image at main thread is visible at Chromium after 2 requestAnimationFrame calls
    // the image is not visible at Firefox 70 or Nightly 72
    osctx.transferFromImageBitmap(bitmap); 

I think you've misunderstood. Of course user gestures are not part of the OffscreenCanvas spec. It is a different spec (at least, I think it is specified), but it is relevant to OffscreenCanvas use cases, which is why I highlighted it.

(In reply to ashley from comment #7)

I think you've misunderstood. Of course user gestures are not part of the OffscreenCanvas spec. It is a different spec (at least, I think it is specified), but it is relevant to OffscreenCanvas use cases, which is why I highlighted it.

If not mistaken gathered the gist of the comment as to not neglect consideration of user activation relevant to OffscreenCanavas, correct?

From perspective here, given that the concept of user activation is arbitrary as to the implementation authors decisions, where interoparability might not be expressed as the primary aim of the discrete implemenation umbrella or specific API, was attempting to convey that given the various observable bugs in Firefox implementation of OffscreenCanvas, if user activation is not an issue now in Firefox, is it beneficial to include such considerations contemporary to observable bugs?

Do bugs currently exist relevant to OffscreenCanvas and user activation at Firefox?

I just wanted to make a heads up to the Firefox developers.

(In reply to ashley from comment #9)

I just wanted to make a heads up to the Firefox developers.

Yes, gather that. User activation relevant to Native File System implementation at Chrome reveals at least several cases that could produce unexpected results. Not certain if any single user activation policy could be implemented in the wild to manage all possible use (and edge) cases and implementer interests for any API.

BTW excellent effort on Imagedata extensions. Cheers.

See Also: → 1598237
Depends on: 1598237
See Also: 1598237
No longer depends on: 1598237
See Also: → 1598237
Depends on: 1598237
See Also: 1598237
Depends on: 1598998
Depends on: 1602420
Depends on: 1603339
Crash Signature: [@ mozilla::ClientWebGLContext::CreateHostContext]

Is there a timeline of release for OffScreenCanvas support in FF?

Roughly 2020H2.

Depends on: 1642216

we have an offscreencanvas crash issue windows FF.

Description: OffscreenCanvas crashes on Windows
Steps to Reproduce:
Running below url on Windows 10 with 78.0.2 (64-bit)
https://web.autocad.com/?webgl2=true

set gfx.offscreencanvas.enabled = true in firefox about:config
Open https://web.autocad.com/?webgl2=true then Login/Create a new account
Open any drawing in file manager
Firefox crashes after the page has run for a couple of seconds.
Chrome is working without any problem.

Crash report:
https://crash-stats.mozilla.org/report/index/330920ed-0b09-4e51-b59b-63ca70200720
https://crash-stats.mozilla.org/report/index/9b8524f1-5300-44b8-a6ca-de5e70200713

Depends on: 1657375

Moved the crash signature to a bug on its own, Bug 1657375 to allow tracking it better.

Crash Signature: [@ mozilla::ClientWebGLContext::CreateHostContext]

Any updates on when this is planned to be implemented? If it's still planned to be implemented in 2020H2 that would be awesome.

Related to bug 1645136

Any progress? Here Firefox plans to ship WebCodecs API only in workers context
https://github.com/w3c/webcodecs/issues/211#issuecomment-859965775
But you can't draw decoded image without OffscreenCanvas in worker.

Last time I checked it year ago behind the flag on Ubuntu it crashed.

Can I get an update on prioritization for this work?

Offscreen Canvas does still seem to be super crashy, so I understand why it's flagged off, but this is a key technology for a number of ML workloads, and it's a gap that is very hard to fill with other technologies that Gecko supports.

(In reply to Richard Newman [:rnewman] from comment #23)

Can I get an update on prioritization for this work?

Offscreen Canvas does still seem to be super crashy, so I understand why it's flagged off, but this is a key technology for a number of ML workloads, and it's a gap that is very hard to fill with other technologies that Gecko supports.

"Ideally this year"

Assignee: nobody → aosmond
Depends on: 1734649
Depends on: 1735874
Depends on: 1735937
Depends on: 1736177

Note: I am not Jeff Gilbert. I am a different entity with a similar name and interest in OffscreenCanvas (small universe).

Cross linking OffscreenCanvas support in webkit to monitor progress there: https://bugs.webkit.org/show_bug.cgi?id=183720

Webcompat Priority: ? → ---
Depends on: 1738971
Depends on: 1740127
Depends on: 1744854
Depends on: 1740388
Depends on: 1745984
Depends on: 1746059
Depends on: 1746093
Depends on: 1746095
Depends on: 1746110
Depends on: 1746260
Depends on: 1746276
Depends on: 1746538
Depends on: 1746750
Depends on: 1072107
Depends on: 1746962
Depends on: 1747476
Depends on: 1747520
Depends on: 1747431
Depends on: 2D-OffscreenCanvas
Depends on: 1749323
Severity: normal → --
Depends on: 1747176
Depends on: 1750980
Depends on: 1751205
Depends on: 1751915
Depends on: 1753573
Depends on: 1753657

I just want to bring awareness that OffscreenCanvas is required to bring compatibility to a number of WebExtension use cases due to the ongoing switch to forcing service workers instead of background pages. Bug 1573659 and Bug 1578286

(In reply to sdaniele3 from comment #27)

I just want to bring awareness that OffscreenCanvas is required to bring compatibility to a number of WebExtension use cases due to the ongoing switch to forcing service workers instead of background pages. Bug 1573659 and Bug 1578286

Oh good to know, thanks! Currently we have turned it on in nightly for certain domains as an origin trial. The major gap remaining is text rendering with Canvas2D. We intend to close that this year and hopefully it can get turned on for extensions as well :).

Depends on: 1756186
Depends on: 1756205
Depends on: 1755698
Depends on: 1755578
Depends on: 1756687
Depends on: 1756864
Depends on: 1754226
Depends on: 1753364
Depends on: 1757504
Depends on: 1757493
Depends on: 1754399
Depends on: 1757820
Depends on: 1755580
Depends on: 1757898
Depends on: 1757759
Blocks: 1703668
Depends on: 1759706
Depends on: 1755458
Depends on: 1753302
Depends on: 1760786
Depends on: 1760907
Depends on: 1761329
Depends on: 1763801
Depends on: 1765323
Depends on: 1765367
Depends on: 1755704
Depends on: 1771154
No longer blocks: 1703668
Depends on: 1776215
Depends on: 1775392
Depends on: 1777279
Depends on: 1777287
Depends on: 1777411
Depends on: 1778537
Depends on: 1778681
Depends on: 1778694
Depends on: 1778711
Depends on: 1778765
Depends on: 1779009
Depends on: 1779096
Depends on: 1779098
Depends on: 1779167
Depends on: 1779315
Depends on: 1779519
Depends on: 1779521
Depends on: 1779835
Depends on: 1780986
Depends on: 1781177
Depends on: 1781831
Depends on: 1782552
Depends on: 1782710
Depends on: 1782722
Depends on: 1782731
Depends on: 1783309
Depends on: 1781293
Depends on: 1783957
Depends on: 1784565
Depends on: 1789693
Depends on: 1795296
Depends on: 1792758
Depends on: 1798769
You need to log in before you can comment on or make changes to this bug.