[meta] Use AVSampleBufferDisplayLayer to display video frames on macOS
Categories
(Core :: Graphics, enhancement, P3)
Tracking
()
People
(Reporter: mstange, Assigned: bradwerth)
References
(Blocks 6 open bugs)
Details
(Keywords: feature-testing-meta, power, Whiteboard: [mac:power])
Attachments
(7 files)
8.57 KB,
text/plain
|
Details | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review |
Once we have dedicated CALayers with IOSurfaces for videos (bug 1653048), we should investigate switching to AVSampleBufferDisplayLayer, which reportedly has lower power usage.
More specifically, AVSampleBufferDisplayLayer is eligible for "detachment" (some kind of low power display mode).
Furthermore, -[AVSampleBufferDisplayLayer enqueueSampleBuffer:]
does not respect CATransactions, so we will need to detect when our layer tree has other changes than just a video frame update, and maybe recreate the layer so that the video frame updates atomatically together with the other changes.
Chrome uses AVSampleBufferDisplayLayer
and we can benefit from the research they did about it. I've attached some notes which I copied from various chromium issues and commit messages.
Updated•4 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Assignee | ||
Comment 2•3 years ago
|
||
Updated•3 years ago
|
Assignee | ||
Comment 3•3 years ago
|
||
Depends on D107292
Updated•3 years ago
|
Updated•3 years ago
|
Assignee | ||
Comment 4•3 years ago
•
|
||
The proposed patches appear to make power consumption somewhat worse. I measured 5 minutes of fullscreen streaming of https://www.adultswim.com/streams/rick-and-morty and got these average wattage values:
Local Opt Firefox baseline: 4.98W
Local Opt Firefox patched: 5.76W
Safari: 4.78W
Chrome: 4.14W
I'll keep working on this to see if there's some additional attributes that we need to provide to get benefit from this.
Updated•3 years ago
|
Assignee | ||
Comment 5•3 years ago
|
||
Depends on D116643
Assignee | ||
Comment 6•3 years ago
|
||
Depends on D124149
Assignee | ||
Comment 7•3 years ago
|
||
The size of the NativeLayerCA::Representation struct seems to be a factor in
the power consumption of native layer updates. Converting this series of bools
into a bitfield is an easy way to reduce the structure size.
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Assignee | ||
Comment 8•3 years ago
|
||
Parts of this patch series have performance or visual correctness problems.
Defining a pref allows users to opt in to this feature to get performance
benefits for fullscreen video, at the expense of some performance for
non-fullscreen video.
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Assignee | ||
Comment 9•3 years ago
|
||
With these patches applied, fullscreen Youtube video on macOS plays while consuming only 80% of the power it otherwise would. There are still visual problems -- controls do not appear in fullscreen -- so the feature remains protected behind the macOS-only gfx.layers.specialized-video
pref.
Comment 10•3 years ago
|
||
(In reply to Brad Werth [:bradwerth] from comment #4)
The proposed patches appear to make power consumption somewhat worse. I measured 5 minutes of fullscreen streaming of https://www.adultswim.com/streams/rick-and-morty and got these average wattage values:
Local Opt Firefox baseline: 4.98W Local Opt Firefox patched: 5.76W Safari: 4.78W Chrome: 4.14W
I'll keep working on this to see if there's some additional attributes that we need to provide to get benefit from this.
How was this measured?
Assignee | ||
Comment 11•3 years ago
|
||
(In reply to Jean-Yves Avenard [:jya] from comment #10)
(In reply to Brad Werth [:bradwerth] from comment #4)
The proposed patches appear to make power consumption somewhat worse. I measured 5 minutes of fullscreen streaming of https://www.adultswim.com/streams/rick-and-morty and got these average wattage values:
How was this measured?
Those results were measured on an Intel-chip Macbook Pro with the Intel-provided tool Intel Power Gadget, run for about 5 minutes. I've since switched to a different method that uses the same hardware sensors. I'm measuring with the terminal command powermetrics
, specifically the command:
sudo powermetrics | grep "package power"
which prints out a line with the total package power every 5 seconds. I start that command, get the browser ready to go, then switch to the browser (and fullscreen mode if that's what I'm testing). I measure for about a minute, switch back to the Terminal and kill the process. I throw out the first and last sample and average the rest. I use a spreadsheet to average the copy-and-pasted text from the Terminal.
Comment 12•3 years ago
|
||
Pushed by bwerth@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/2254f5bced20 Part 1: Define a pref for specialized video layers on macOS. r=gfx-reviewers,jrmuizel
Comment 13•3 years ago
|
||
Pushed by bwerth@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/3d9d61f1a621 Part 2: Make nsCocoaWindow notify its native layer when toggling fullscreen. r=gfx-reviewers,mstange https://hg.mozilla.org/integration/autoland/rev/6bb1dd91e24b Part 3: Convert NativeLayerCA::Representation flags to use a bitfield. r=gfx-reviewers,mstange https://hg.mozilla.org/integration/autoland/rev/c03f05a6d2f1 Part 4: Add scaffolding for NativeLayerCA to specialize video layers. r=gfx-reviewers,mstange https://hg.mozilla.org/integration/autoland/rev/ba98a3bf58a1 Part 5: Make NativeLayerCA use AVSampleBufferDisplayLayers for video. r=gfx-reviewers,mstange https://hg.mozilla.org/integration/autoland/rev/06a1fcbd9648 Part 6: Make NativeLayerRootCA attempt to isolate video layers. r=gfx-reviewers,jrmuizel
Comment 14•3 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/2254f5bced20
https://hg.mozilla.org/mozilla-central/rev/3d9d61f1a621
https://hg.mozilla.org/mozilla-central/rev/6bb1dd91e24b
https://hg.mozilla.org/mozilla-central/rev/c03f05a6d2f1
https://hg.mozilla.org/mozilla-central/rev/ba98a3bf58a1
https://hg.mozilla.org/mozilla-central/rev/06a1fcbd9648
Updated•3 years ago
|
Updated•3 years ago
|
Comment 15•3 years ago
|
||
This is something I assume we'll want to call out in the Fx94 beta relnotes. Can you please nominate?
Assignee | ||
Comment 16•3 years ago
|
||
Release Note Request (optional, but appreciated)
[Why is this notable]: This feature reduces power consumption on fullscreen YouTube and Twitch on macOS. Fullscreen videos on other services may also consume less power.
[Affects Firefox for Android]: No.
[Suggested wording]: Firefox macOS now uses Apple's low power mode for fullscreen video on Youtube and Twitch. This meaningfully extends battery life in long viewing sessions.
[Links (documentation, blog post, etc)]: https://developer.apple.com/documentation/webkit/delivering_video_content_for_safari
Description
•