1646329 - [meta] Use VA-API decoder with WebRTC

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement)

Description

[Martin Stránský [:stransky] (ni? me)]

Let's use VA-API decoder with WebRTC, from my experience it's VP8 streams which can be processed by ffmpeg.

Comment 1

[Paul Menzel]

Just a note, that Chromium, last time I looked, has VP8 GPU decoding disabled.

Comment 2

[Darkspirit]

(In reply to Paul Menzel from comment #1)

Just a note, that Chromium, last time I looked, has VP8 GPU decoding disabled.

On ChromeOS? (Chrome uses VAAPI only on ChromeOS.)

Comment 3

[Robert Mader [:rmader]]

from my experience it's VP8 streams which can be processed by ffmpeg.

Jitsi also supports x264 and vp9 (not to mention upcoming AV1 support at some point) - the default is still vp8 though (see e.g. 1). Just as a note that we won't be stuck with vp8 forever and therefore it's great to have VAAPI support :)

1: https://community.jitsi.org/t/flagship-vp9-support-notes-and-discussion/33921/53

Comment 4

[Shmerl]

Thanks for planning it.

For the reference, see comments form libwebrtc project here: https://bugs.chromium.org/p/webrtc/issues/detail?id=11296

Chromium supposedly already supports, but outside the library.

Comment 5

[Shmerl]

What about encoder though? That might have even bigger value for WebRTC.

Comment 6

[russianneuromancer]

Agree, low-end tablets, laptops and Compute Sticks based on Intel Atom (BayTrail and CherryTrail generations) would greatly benefit from hardware-based encoding.

Comment 7

[Martin Stránský [:stransky] (ni? me)]

The WebRTC decoding should work now in latest nightly, you just need to set media.ffmpeg.low-latency.enabled to true at about:config.

Comment 8

[Martin Stránský [:stransky] (ni? me)]

(In reply to Shmerl from comment #5)

What about encoder though? That might have even bigger value for WebRTC.

I don't think so. On a regular call you typically need to decode many streams for various participant but encode only one from you.
I'd love to see any testing/numbers if the vaapi decoding here helps or not.

VA-API based encoding will need WebRTC hacking to get images from camera exported as dmabuf surfaces and encoded directly in GPU memory, WebRTC uses kNative surface type for that.

Comment 9

[Shmerl]

I don't think so. On a regular call you typically need to decode many streams for various participant but encode only one from you.

My assumption was simple. Encoding is always more intense, so even if it's a simple 1:1 call, encoding will load the CPU significantly. For a laptop, it drains the battery. Some testing and comparison of course would be good, but I'd assume it can also depend on the used codec and actual encoder implementation.

Comment 10

[Paul Menzel]

(In reply to Shmerl from comment #9)

I don't think so. On a regular call you typically need to decode many streams for various participant but encode only one from you.

My assumption was simple. Encoding is always more intense, so even if it's a simple 1:1 call, encoding will load the CPU significantly. For a laptop, it drains the battery. Some testing and comparison of course would be good, but I'd assume it can also depend on the used codec and actual encoder implementation.

That could be, but this issue is about decoding. Please create a separate issue.

Comment 11

[Martin Stránský [:stransky] (ni? me)]

This can be used now with both Wayland and X11 EGL backends.

Comment 12

[Martin Stránský [:stransky] (ni? me)]

btw. it also needs enabled media.navigator.mediadatadecoder_vpx_enabled in release.
Also that may work from FF81 as WebRTC/VP8 is sometimes encoded to YV12 which is supported by Bug 1655747.

Comment 13

[chewie lin [sl424]]

i read that zoom uses parts of webrtc protocol, does that mean it could use vaapi decoder?

Comment 14

[Esokrarkose]

Are there any blockers for this issue? I often notice the laptop running hot when using conference services like jitsi, zoom etc. Also is there a report for the encoding part already? Turning the camera on in Jitsi doubles my cpu usage, so it seems the encoding is indeed expensive.

Comment 15

[Martin Stránský [:stransky] (ni? me)]

(In reply to Esokrarkose from comment #14)

Are there any blockers for this issue? I often notice the laptop running hot when using conference services like jitsi, zoom etc. Also is there a report for the encoding part already? Turning the camera on in Jitsi doubles my cpu usage, so it seems the encoding is indeed expensive.

It's on by default when vaapi is enabled. What you see is missing encoding support - Bug 1658900.

Comment 16

[Vincent Chernin]

This issue now depends on Bug #1709009. Solving Bug #1665329 was necessary for this bug to be fixed, but Bug #1665329’s change was reversed due to Bug #1680313. This should be reoponded due being reverted by as noted in Bug #1709009.

In simpler terms for WebRTC video to be decoded on Linux media.navigator.mediadatadecoder_vpx_enabled needs to be set true (again) for release. Bug #1709009 tracks this fix.

Comment 17

[Robert Mader [:rmader]]

Note: as VAAPI needs the EGL backend, in order to enable this by default, this additionally depends on bug 1695933 and/or bug 1543600. Otherwise the env variables MOZ_X11_EGL=1 or MOZ_ENABLE_WAYLAND=1 are needed.

Comment 18

[Mathew Hodson]

Martin do you want to reopen this and add bug 1709009, bug 1695933, and bug 1543600 as dependencies?

Comment 19

[Francois Guerraz]

If it's about making it enabled by default, I don't think it's realistic without solving the sandboxing issues (bug 1619585 and bug 1683808) also bug 1659054 is still an issue with WebRTC (not general VAAPI accel).

Comment 20

[Martin Stránský [:stransky] (ni? me)]

Yes, let's reopen it.

Comment 21

[Martin Stránský [:stransky] (ni? me)]

Fixed and used now.