Open Bug 1658900 Opened 4 years ago Updated 3 months ago

Use VA-API encoder with WebRTC


(Core :: WebRTC: Audio/Video, enhancement, P4)





(Reporter: stransky, Unassigned)


(Depends on 1 open bug, Blocks 1 open bug)


Let's use VA-API encoder with WebRTC. That involves to implement kNative WebRTC dmabuf surfaces and encode frames directly at GPU memory by ffmpeg encoder similar to Apple/Android.

Bug title says decoder and comment says encoder ._.

Summary: Use VA-API decoder with WebRTC → Use VA-API encoder with WebRTC
Severity: -- → S4
Priority: -- → P4
Depends on: 1724900

For the record, I'm currently working on bug 1724900 (using Pipewire and xdg-portals for camera access) and by default Pipewire appears to prefer Dmabuf as buffer sharing mechanism there as well (in YUV2 format).
So once we have that, we can send both screen sharing and camera stream over to VAAPI and encode it there. And we should be able to use the camera input in Webrender directly.

But only if widget.use-xdg-desktop-portal set to non-default true, I suppose? Firefox use no portals by default.

Getting dmabuf from camera it not difficult, it can be done here:

the tricky part is to route the dmabuf buffer through webrtc.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #5)

I think the path is to implement VideoFrameBuffer kNative type based on dmabuf:

Filed as Bug 1729167.

For the debugging we can create dmabuf surfaces and upload captured frames there (as we do that for SW decoded frames).

With the latest work on FFmpegVideoEncoder it may be fairly easy to implement HW encoder as the hard work is already done.
With media.webrtc.platformencoder set to true WebRTC routes encoding to FFmpegVideoEncoder module.

We may need to implement VA-API encoding there, simple example is at:
(we may need something better configured but it's enough as a starting point).

We may consider direct dmabuf frame upload and do all YUV operations on GPU (scale & etc) and then supply frame for encoding.

You need to log in before you can comment on or make changes to this bug.