Closed Bug 1653048 Opened 1 year ago Closed 1 year ago

Use CoreAnimation for scaling and YUV conversion of video

Categories

(Core :: Graphics: WebRender, enhancement, P3)

All
macOS
enhancement

Tracking

()

RESOLVED FIXED
81 Branch
Tracking Status
firefox81 --- fixed

People

(Reporter: mstange, Assigned: mattwoodrow)

References

(Blocks 2 open bugs, Regressed 1 open bug)

Details

(Keywords: power, Whiteboard: wr-planning)

Attachments

(1 file)

This is the macOS version of bug 1623530.

At the moment, videos get on the screen by having a dedicated CALayer with its own IOSurface in screen space. Then WebRender draws the video into that IOSurface, doing scaling and color conversion in the process. And then CoreAnimation puts the layer on the screen without further scaling.

We would like to skip the WebRender draw (it's an extra copy) and do the scaling and color conversion using CoreAnimation.

In order to do this, we'll need to extend the OS compositor API with a method that lets WebRender say "Add a surface containing this external image, scaled to the following rect" (bug 1623638), and we'll need to have an IOSurface handle for that external image.
For hardware decoded video and for WebGL, we should already have an IOSurface, unless we accidentally read back the contents into a shmem at some point in the pipeline.
For software decoded video and 2D canvas we currently only have a shmem. As a first step, we can do a CPU-side copy into an IOSurface in the compositor. As a second step, we should move the IOSurface closer to where the data is being generated (e.g. software video decoding) so that we can skip that copy.

We'll also need to keep in mind the buffer lifetimes. Giving an IOSurface to the window server means that the IOSurface content must not be touched until the window server is done with it. In the past, WR was doing a copy during the composite, so the IOSurface could be unlocked right after that composite. But now it will need to stay locked for longer.

Whiteboard: wr-planning
Blocks: wr-mac
Severity: -- → S4
Priority: -- → P3
Severity: S4 → N/A
Depends on: 1653409

(In reply to Markus Stange [:mstange] from comment #0)

For software decoded video and 2D canvas we currently only have a shmem. [..]

I filed bug 1653409 on this part.

Keywords: power
Blocks: 1653417
Assignee: nobody → matt.woodrow
Status: NEW → ASSIGNED
Pushed by mwoodrow@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/06d589672105
Add support for external images in the wr::Compositor API, and implement it for MacOS IOSurfaces. r=mstange,gw
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → 81 Branch
Regressions: 1657428
No longer regressions: 1657428
No longer regressions: 1657555
Regressions: 1658058
Regressions: 1658856
See Also: → 1711461
See Also: → 1623530
You need to log in before you can comment on or make changes to this bug.