1653409 - Use IOSurface rather than Client Storage to upload software-decoded video on macOS

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, task, P3)

Description

[Markus Stange [:mstange]]

In order to do bug 1653048 for software-decoded video, we will need to have the video data in an IOSurface. Currently we have the data in a shmem, and the data gets uploaded to a GL texture via client storage.

Not all videos will be able to use the optimized path from bug 1653048. For example, videos that are subject to certain effects such as masks, 3d transforms or filters will not be able to be promoted to "compositor surfaces". For those videos we will still need a GL texture.

We can get a GL texture for an IOSurface. That means we can use the IOSurface upload path for all videos, and we can switch to it even before we do bug 1653048.

Some more relevant info from bug 1653048:

For software decoded video and 2D canvas we currently only have a shmem. As a first step, we can do a CPU-side copy into an IOSurface in the compositor. As a second step, we should move the IOSurface closer to where the data is being generated (e.g. software video decoding) so that we can skip that copy.

We'll also need to keep in mind the buffer lifetimes. Giving an IOSurface to the window server means that the IOSurface content must not be touched until the window server is done with it. In the past, WR was doing a copy during the composite, so the IOSurface could be unlocked right after that composite. But now it will need to stay locked for longer.

Comment 1

[Matt Woodrow (:mattwoodrow)]

Attachment: Bug 1653409 - Upload software decoded video directly to IOSurface on OSX. r?mstange

Depends on D84638

Comment 2

[Matt Woodrow (:mattwoodrow)]

Attachment: Bug 1653409 - Set ImageBitmap's picture rect using the picture rect of layers::Image, not the size. r?mstange

Depends on D85355

Comment 3

[Matt Woodrow (:mattwoodrow)]

Attachment: Bug 1653409 - Specify correct color space and YCbCr matrix on IOSurfaces, so that CoreAnimation converts them correctly. r?mstange

This overrides the colorspace of hardware decoded videos to specify the monitor's color space, to prevent CoreAnimation from doing conversion.
This probably isn't the right long term answer, but it matches our current rendering and readback code paths.

A note for the future: IOSurfaceSetValue/IOSurfaceCopyValue silently fail if the IOSurface is 'in use'.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c2626b736f34
Upload software decoded video directly to IOSurface on OSX. r=mstange
https://hg.mozilla.org/integration/autoland/rev/7b57934c0765
Specify correct color space and YCbCr matrix on IOSurfaces, so that CoreAnimation converts them correctly. r=mstange
https://hg.mozilla.org/integration/autoland/rev/4558b3cfd6fe
Set ImageBitmap's picture rect using the picture rect of layers::Image, not the size. r=mstange

Comment 6

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/c2626b736f34
https://hg.mozilla.org/mozilla-central/rev/7b57934c0765
https://hg.mozilla.org/mozilla-central/rev/4558b3cfd6fe