1569686 - Update Media Capabilities API decodingInfo response to return powerEfficient only when Firefox can do hardware decode

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[Asif Youssuff]

We often see complaints that Firefox lags when playing video compared to other browsers, and can't yet benchmark AV1 to ensure that it is "smooth" (see 1530996).

It'd be nice to increase power efficiency by returning "powerEfficient" more often (right now on my hardware it doesn't appear on most resolutions or codecs) and accurately, by ensuring that it returns only for codecs and resolutions with working hardware decode.

I think it makes sense to NOT support this when resistFingerprinting is on, but it should be an overall win for video players that care to query on this information.

This is the page I used to see whether Firefox accurately reports this per my hardware: https://beaufortfrancois.github.io/sandbox/media-capabilities/decoding-info.html

Comment 1

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Agreed that we want to accurately report this information to ensure a smooth user experience. Do you have examples of cases where reporting is currently causing issues? It would be useful to have some scenarios where we can start debugging from.

Comment 2

[Asif Youssuff]

I am not entirely sure which players actually use this information, but I am led to believe that YouTube may, based on this comment on reddit: https://www.reddit.com/r/firefox/comments/cizjii/100_cpu_usage_while_playing_some_videos/evbdxkv/

Even if they do not, this is something that Mozilla could encourage players to check to ensure that playback uses an efficient decode method as a prioritized method.

Bug 1400787 shows an example of Firefox performing poorly compared to other browsers where this could help as well.

A bad situation occurred on that same reddit thread, where YouTube served the user AV1 video on a baytrail n3540 -- it is unclear to me whether YouTube decided to send this because of the Media Capabilities API of whether there is some other heuristic in place.

Comment 3

[Asif Youssuff]

FWIW, at least on my hardware and OS, based on my proposal here, none of the codecs and sizes would be seen as powerEfficient, since Firefox doesn't offer hardware decode of video on Linux https://beaufortfrancois.github.io/sandbox/media-capabilities/decoding-info.html

Instead, I see that 640x480, 640x360, 320x240 across a number of codecs appear as powerEfficient.

Unfortunately, the situation is exactly the same in Windows, where DXVA checker shows the following for my hardware:

Intel(R) HD Graphics 630
MPEG2_VLD: DXVA2/D3D11, SD / HD / FHD
MPEG2_IDCT: DXVA2/D3D11, SD / HD / FHD
MPEG2_A: DXVA1, SD / HD / FHD
MPEG2_C: DXVA1, SD / HD / FHD
VC1_VLD_2_Intel: DXVA2/D3D11, SD / HD / FHD / 4K
VC1_VLD_Intel: DXVA2/D3D11, SD / HD / FHD / 4K
VC1_VLD2010: DXVA2/D3D11, SD / HD / FHD / 4K
H264_VLD_NoFGT: DXVA2/D3D11, SD / HD / FHD / 4K
H264_VLD_Stereo_Progressive_NoFGT: DXVA2/D3D11, SD / HD / FHD / 4K
H264_VLD_Stereo_NoFGT: DXVA2/D3D11, SD / HD / FHD / 4K
H264_VLD_Multiview_NoFGT: DXVA2/D3D11, SD / HD / FHD / 4K
VP8_VLD: DXVA2/D3D11, SD / HD / FHD / 4K
HEVC_VLD_Main: DXVA2/D3D11, SD / HD / FHD / 4K / 8K
HEVC_VLD_Main10: DXVA2/D3D11, SD / HD / FHD / 4K / 8K
VP9_VLD_Profile0: DXVA2/D3D11, SD / HD / FHD / 4K / 8K
VP9_VLD_10bit_Profile2: DXVA2/D3D11, SD / HD / FHD / 4K / 8K
VP9_VLD_Intel: DXVA2/D3D11, SD / HD / FHD / 4K / 8K
WMV9_IDCT: DXVA1/2/D3D11, SD / HD / FHD / 4K
VC1_IDCT: DXVA1/2/D3D11, SD / HD / FHD / 4K

(cut some lines that looked uninteresting)

Chrome does better here, as it reports most codecs as being smooth and power efficient, as I would expect based on https://en.wikichip.org/wiki/intel/hd_graphics_630#Hardware_Accelerated_Video for my i7-7700HQ.

I understand that DXVA checker may not be entirely accurate either because the driver needs to support it, but it certainly seems like Chrome is doing something different (and more accurate) than Firefox at this time.

Comment 4

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Alex, do we have any bugs currently tracking power efficiency for MediaCaps?

Comment 5

[Alex Chronopoulos [:achronop]]

No, we don't have any. We can use this one. The way that we report power-efficient does not change in Bug 1530996.

Comment 6

[George Sofianos]

My Firefox nightly is currently affected by this, my GPU (AMD 5700XT) supports HW decoding for VP9 (through VAAPI and on Windows), but powerEfficient always returns false on both Operating Systems.

I'm not sure if Youtube supports the MediaCapabilities API, but videos like https://www.youtube.com/watch?v=kFz9afj8lu0 will serve AV1 instead of VP9 (For 1080p/8k) if the Youtube setting is left to Auto.

Comment 7

[George Sofianos]

I checked Youtube player source code and it's definitely using the MediaCapabilities API. I've taken a screenshot of the object it sends for a random video. Link -> https://i.imgur.com/wYfAYkf.jpg

I'm also pasting the codecs sent in case the image will not be available in the future

[
"mimeType": "video/webm; codecs="vp09.00.51.08.01.01.01.01.00"",
"mimeType": "video/webm; codecs="vp09.00.51.08.01.01.01.01.00"",
"mimeType": "video/mp4; codecs="av01.0.04M.08"",
"mimeType": "video/mp4; codecs="av01.0.01M.08"",
"mimeType": "video/mp4; codecs="av01.0.00M.08"",
"mimeType": "video/mp4; codecs="av01.0.00M.08""
]

Comment 8

[Zaggy1024]

Attachment: Bug 1569686 - Fixed MediaCapabilities API not returning powerEfficient=true all VP8/VP9 codecs. r=alwu

Depends on D138884

Comment 9

[Zaggy1024]

Attachment: Video support test.html

Adding a test page here to show the various codec strings I've been testing.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a7ab49506ec3
Fixed MediaCapabilities API not returning powerEfficient=true all VP8/VP9 codecs. r=alwu

Comment 11

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/a7ab49506ec3