Open Bug 1646329 Opened 2 years ago Updated 1 day ago

[meta] Use VA-API decoder with WebRTC

Categories

(Core :: Audio/Video: Playback, enhancement)

enhancement

Tracking

()

REOPENED

People

(Reporter: stransky, Assigned: stransky)

References

(Depends on 2 open bugs, Blocks 1 open bug)

Details

(Keywords: meta)

Let's use VA-API decoder with WebRTC, from my experience it's VP8 streams which can be processed by ffmpeg.

Just a note, that Chromium, last time I looked, has VP8 GPU decoding disabled.

(In reply to Paul Menzel from comment #1)

Just a note, that Chromium, last time I looked, has VP8 GPU decoding disabled.

On ChromeOS? (Chrome uses VAAPI only on ChromeOS.)

from my experience it's VP8 streams which can be processed by ffmpeg.

Jitsi also supports x264 and vp9 (not to mention upcoming AV1 support at some point) - the default is still vp8 though (see e.g. 1). Just as a note that we won't be stuck with vp8 forever and therefore it's great to have VAAPI support :)

1: https://community.jitsi.org/t/flagship-vp9-support-notes-and-discussion/33921/53

Thanks for planning it.

For the reference, see comments form libwebrtc project here: https://bugs.chromium.org/p/webrtc/issues/detail?id=11296

Chromium supposedly already supports, but outside the library.

What about encoder though? That might have even bigger value for WebRTC.

Agree, low-end tablets, laptops and Compute Sticks based on Intel Atom (BayTrail and CherryTrail generations) would greatly benefit from hardware-based encoding.

See Also: → consolidation
Depends on: 1655747

The WebRTC decoding should work now in latest nightly, you just need to set media.ffmpeg.low-latency.enabled to true at about:config.

Depends on: 1657845

(In reply to Shmerl from comment #5)

What about encoder though? That might have even bigger value for WebRTC.

I don't think so. On a regular call you typically need to decode many streams for various participant but encode only one from you.
I'd love to see any testing/numbers if the vaapi decoding here helps or not.

VA-API based encoding will need WebRTC hacking to get images from camera exported as dmabuf surfaces and encoded directly in GPU memory, WebRTC uses kNative surface type for that.

Assignee: nobody → stransky

I don't think so. On a regular call you typically need to decode many streams for various participant but encode only one from you.

My assumption was simple. Encoding is always more intense, so even if it's a simple 1:1 call, encoding will load the CPU significantly. For a laptop, it drains the battery. Some testing and comparison of course would be good, but I'd assume it can also depend on the used codec and actual encoder implementation.

(In reply to Shmerl from comment #9)

I don't think so. On a regular call you typically need to decode many streams for various participant but encode only one from you.

My assumption was simple. Encoding is always more intense, so even if it's a simple 1:1 call, encoding will load the CPU significantly. For a laptop, it drains the battery. Some testing and comparison of course would be good, but I'd assume it can also depend on the used codec and actual encoder implementation.

That could be, but this issue is about decoding. Please create a separate issue.

This can be used now with both Wayland and X11 EGL backends.

Summary: [Wayland] Use VA-API decoder with WebRTC → Use VA-API decoder with WebRTC
Depends on: 1659054

btw. it also needs enabled media.navigator.mediadatadecoder_vpx_enabled in release.
Also that may work from FF81 as WebRTC/VP8 is sometimes encoded to YV12 which is supported by Bug 1655747.

i read that zoom uses parts of webrtc protocol, does that mean it could use vaapi decoder?

Depends on: 1665324
Depends on: 1665329
See Also: → 1656252

Are there any blockers for this issue? I often notice the laptop running hot when using conference services like jitsi, zoom etc. Also is there a report for the encoding part already? Turning the camera on in Jitsi doubles my cpu usage, so it seems the encoding is indeed expensive.

(In reply to Esokrarkose from comment #14)

Are there any blockers for this issue? I often notice the laptop running hot when using conference services like jitsi, zoom etc. Also is there a report for the encoding part already? Turning the camera on in Jitsi doubles my cpu usage, so it seems the encoding is indeed expensive.

It's on by default when vaapi is enabled. What you see is missing encoding support - Bug 1658900.

No longer depends on: 1659054
Status: NEW → RESOLVED
Closed: 10 months ago
Resolution: --- → FIXED

This issue now depends on Bug #1709009. Solving Bug #1665329 was necessary for this bug to be fixed, but Bug #1665329’s change was reversed due to Bug #1680313. This should be reoponded due being reverted by as noted in Bug #1709009.

In simpler terms for WebRTC video to be decoded on Linux media.navigator.mediadatadecoder_vpx_enabled needs to be set true (again) for release. Bug #1709009 tracks this fix.

Note: as VAAPI needs the EGL backend, in order to enable this by default, this additionally depends on bug 1695933 and/or bug 1543600. Otherwise the env variables MOZ_X11_EGL=1 or MOZ_ENABLE_WAYLAND=1 are needed.

Martin do you want to reopen this and add bug 1709009, bug 1695933, and bug 1543600 as dependencies?

Flags: needinfo?(stransky)

If it's about making it enabled by default, I don't think it's realistic without solving the sandboxing issues (bug 1619585 and bug 1683808) also bug 1659054 is still an issue with WebRTC (not general VAAPI accel).

Yes, let's reopen it.

Status: RESOLVED → REOPENED
Flags: needinfo?(stransky)
Resolution: FIXED → ---
Depends on: 1709009, 1695933
Depends on: 1619585, 1683808
Depends on: 1743159
Depends on: 1659054
No longer depends on: 1683808
Summary: Use VA-API decoder with WebRTC → [meta] Use VA-API decoder with WebRTC
You need to log in before you can comment on or make changes to this bug.