Closed Bug 1653417 Opened 4 years ago Closed 3 years ago

[meta] Use AVSampleBufferDisplayLayer to display video frames on macOS

Categories

(Core :: Graphics, enhancement, P3)

All
macOS
enhancement

Tracking

()

RESOLVED FIXED
94 Branch
Tracking Status
relnote-firefox --- 94+
firefox94 --- fixed

People

(Reporter: mstange, Assigned: bradwerth)

References

(Blocks 6 open bugs)

Details

(Keywords: feature-testing-meta, power, Whiteboard: [mac:power])

Attachments

(7 files)

Once we have dedicated CALayers with IOSurfaces for videos (bug 1653048), we should investigate switching to AVSampleBufferDisplayLayer, which reportedly has lower power usage.

More specifically, AVSampleBufferDisplayLayer is eligible for "detachment" (some kind of low power display mode).
Furthermore, -[AVSampleBufferDisplayLayer enqueueSampleBuffer:] does not respect CATransactions, so we will need to detect when our layer tree has other changes than just a video frame update, and maybe recreate the layer so that the video frame updates atomatically together with the other changes.

Chrome uses AVSampleBufferDisplayLayer and we can benefit from the research they did about it. I've attached some notes which I copied from various chromium issues and commit messages.

Severity: -- → S3
Priority: -- → P3

I'll try to complete this.

Assignee: nobody → bwerth
Whiteboard: [mac:power]
Attachment #9207049 - Attachment description: Bug 1653417 Part 1: Add scaffolding for NativeLayerCA to make specialized video layers. → WIP: Bug 1653417 Part 1: Add scaffolding for NativeLayerCA to make specialized video layers.
Attachment #9207049 - Attachment description: WIP: Bug 1653417 Part 1: Add scaffolding for NativeLayerCA to make specialized video layers. → Bug 1653417 Part 1: Add scaffolding for NativeLayerCA to make specialized video layers.
Attachment #9224799 - Attachment description: WIP: Bug 1653417 Part 2: Make NativeLayerCA use AVSampleBufferDisplayLayers for video. → Bug 1653417 Part 2: Make NativeLayerCA use AVSampleBufferDisplayLayers for video.

The proposed patches appear to make power consumption somewhat worse. I measured 5 minutes of fullscreen streaming of https://www.adultswim.com/streams/rick-and-morty and got these average wattage values:

Local Opt Firefox baseline: 4.98W
Local Opt Firefox patched: 5.76W
Safari: 4.78W
Chrome: 4.14W

I'll keep working on this to see if there's some additional attributes that we need to provide to get benefit from this.

The size of the NativeLayerCA::Representation struct seems to be a factor in
the power consumption of native layer updates. Converting this series of bools
into a bitfield is an easy way to reduce the structure size.

Attachment #9207049 - Attachment description: Bug 1653417 Part 1: Add scaffolding for NativeLayerCA to make specialized video layers. → Bug 1653417 Part 2: Add scaffolding for NativeLayerCA to make specialized video layers.
Attachment #9224799 - Attachment description: Bug 1653417 Part 2: Make NativeLayerCA use AVSampleBufferDisplayLayers for video. → Bug 1653417 Part 3: Make NativeLayerCA use AVSampleBufferDisplayLayers for video.
Attachment #9238827 - Attachment description: Bug 1653417 Part 3: Make nsCocoaWindow notify its native layer when toggling fullscreen. → Bug 1653417 Part 4: Make nsCocoaWindow notify its native layer when toggling fullscreen.
Attachment #9238828 - Attachment description: Bug 1653417 Part 4: Make NativeLayerRootCA attempt to isolate video layers. → Bug 1653417 Part 5: Make NativeLayerRootCA attempt to isolate video layers.

Parts of this patch series have performance or visual correctness problems.
Defining a pref allows users to opt in to this feature to get performance
benefits for fullscreen video, at the expense of some performance for
non-fullscreen video.

Attachment #9240161 - Attachment description: Bug 1653417 Part 1: Convert NativeLayerCA::Representation flags to use a bitfield. → Bug 1653417 Part 2: Convert NativeLayerCA::Representation flags to use a bitfield.
Attachment #9207049 - Attachment description: Bug 1653417 Part 2: Add scaffolding for NativeLayerCA to make specialized video layers. → Bug 1653417 Part 3: Add scaffolding for NativeLayerCA to make specialized video layers.
Attachment #9224799 - Attachment description: Bug 1653417 Part 3: Make NativeLayerCA use AVSampleBufferDisplayLayers for video. → Bug 1653417 Part 4: Make NativeLayerCA use AVSampleBufferDisplayLayers for video.
Attachment #9238827 - Attachment description: Bug 1653417 Part 4: Make nsCocoaWindow notify its native layer when toggling fullscreen. → Bug 1653417 Part 5: Make nsCocoaWindow notify its native layer when toggling fullscreen.
Attachment #9238828 - Attachment description: Bug 1653417 Part 5: Make NativeLayerRootCA attempt to isolate video layers. → Bug 1653417 Part 6: Make NativeLayerRootCA attempt to isolate video layers.
Attachment #9240170 - Attachment description: Bug 1653417 Part 1: Define a pref. → Bug 1653417 Part 1: Define a pref for specialized video layers on macOS.
Attachment #9238827 - Attachment description: Bug 1653417 Part 5: Make nsCocoaWindow notify its native layer when toggling fullscreen. → Bug 1653417 Part 2: Make nsCocoaWindow notify its native layer when toggling fullscreen.
Attachment #9240161 - Attachment description: Bug 1653417 Part 2: Convert NativeLayerCA::Representation flags to use a bitfield. → Bug 1653417 Part 3: Convert NativeLayerCA::Representation flags to use a bitfield.
Attachment #9207049 - Attachment description: Bug 1653417 Part 3: Add scaffolding for NativeLayerCA to make specialized video layers. → Bug 1653417 Part 4: Add scaffolding for NativeLayerCA to specialize video layers.
Attachment #9224799 - Attachment description: Bug 1653417 Part 4: Make NativeLayerCA use AVSampleBufferDisplayLayers for video. → Bug 1653417 Part 5: Make NativeLayerCA use AVSampleBufferDisplayLayers for video.

With these patches applied, fullscreen Youtube video on macOS plays while consuming only 80% of the power it otherwise would. There are still visual problems -- controls do not appear in fullscreen -- so the feature remains protected behind the macOS-only gfx.layers.specialized-video pref.

(In reply to Brad Werth [:bradwerth] from comment #4)

The proposed patches appear to make power consumption somewhat worse. I measured 5 minutes of fullscreen streaming of https://www.adultswim.com/streams/rick-and-morty and got these average wattage values:

Local Opt Firefox baseline: 4.98W
Local Opt Firefox patched: 5.76W
Safari: 4.78W
Chrome: 4.14W

I'll keep working on this to see if there's some additional attributes that we need to provide to get benefit from this.

How was this measured?

(In reply to Jean-Yves Avenard [:jya] from comment #10)

(In reply to Brad Werth [:bradwerth] from comment #4)

The proposed patches appear to make power consumption somewhat worse. I measured 5 minutes of fullscreen streaming of https://www.adultswim.com/streams/rick-and-morty and got these average wattage values:
How was this measured?

Those results were measured on an Intel-chip Macbook Pro with the Intel-provided tool Intel Power Gadget, run for about 5 minutes. I've since switched to a different method that uses the same hardware sensors. I'm measuring with the terminal command powermetrics, specifically the command:

sudo powermetrics | grep "package power"

which prints out a line with the total package power every 5 seconds. I start that command, get the browser ready to go, then switch to the browser (and fullscreen mode if that's what I'm testing). I measure for about a minute, switch back to the Terminal and kill the process. I throw out the first and last sample and average the rest. I use a spreadsheet to average the copy-and-pasted text from the Terminal.

Blocks: 1731135
Blocks: 1731136
Blocks: 1731650
Pushed by bwerth@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2254f5bced20
Part 1: Define a pref for specialized video layers on macOS. r=gfx-reviewers,jrmuizel
Pushed by bwerth@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3d9d61f1a621
Part 2: Make nsCocoaWindow notify its native layer when toggling fullscreen. r=gfx-reviewers,mstange
https://hg.mozilla.org/integration/autoland/rev/6bb1dd91e24b
Part 3: Convert NativeLayerCA::Representation flags to use a bitfield. r=gfx-reviewers,mstange
https://hg.mozilla.org/integration/autoland/rev/c03f05a6d2f1
Part 4: Add scaffolding for NativeLayerCA to specialize video layers. r=gfx-reviewers,mstange
https://hg.mozilla.org/integration/autoland/rev/ba98a3bf58a1
Part 5: Make NativeLayerCA use AVSampleBufferDisplayLayers for video. r=gfx-reviewers,mstange
https://hg.mozilla.org/integration/autoland/rev/06a1fcbd9648
Part 6: Make NativeLayerRootCA attempt to isolate video layers. r=gfx-reviewers,jrmuizel
Blocks: 1731691
Blocks: 1731815
Regressions: 1731859
Regressions: 1732694
Summary: Use AVSampleBufferDisplayLayer to display video frames on macOS → [meta] Use AVSampleBufferDisplayLayer to display video frames on macOS
See Also: → 1711461
Regressions: 1731871

This is something I assume we'll want to call out in the Fx94 beta relnotes. Can you please nominate?

Flags: needinfo?(bwerth)

Release Note Request (optional, but appreciated)
[Why is this notable]: This feature reduces power consumption on fullscreen YouTube and Twitch on macOS. Fullscreen videos on other services may also consume less power.
[Affects Firefox for Android]: No.
[Suggested wording]: Firefox macOS now uses Apple's low power mode for fullscreen video on Youtube and Twitch. This meaningfully extends battery life in long viewing sessions.
[Links (documentation, blog post, etc)]: https://developer.apple.com/documentation/webkit/delivering_video_content_for_safari

relnote-firefox: --- → ?
Flags: needinfo?(bwerth)
Regressions: 1731922

Added to the Fx94 beta relnotes.

Blocks: 1735420
Regressions: 1734854
Blocks: 1737682
Depends on: 1737998
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: