1917191 - Skip desktop capture frames that have not gotten dirty since the previous frame

Status: ASSIGNED

(Core :: WebRTC: Audio/Video, task)

Description

[Andreas Pehrson [:pehrsons]]

DesktopFrame instances (frames captured by DesktopCapturers) expose updated_region containing info on what rects in the frame had changed since the previous frame.

Currently we ignore this and process every frame regardless of whether it is a duplicate of a frame we have already seen, or not. These frames are very large so processing them in software is expensive. This bug is to ignore all duplicates, to skip a costly image format conversion, several copies, a probable downscaling pass, and possibly rendering and encoding depending on use case.

This should improve performance for desktop capture in a lot of cases, though doesn't help for the worst-case (where say a video or an animation is showing in the capture source).

Comment 1

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1917191 - Break frame conversion logic out of DesktopCaptureImpl. r?#webrtc-reviewers

This will allow implementing other converter modules.

Comment 2

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1917191 - Add a pref to skip conversion of duplicate frames from desktop capturers. r?#webrtc-reviewers

Comment 3

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1917191 - In TabCapturerWebrtc mark all frames dirty so they don't get ignored. r?#webrtc-reviewers

Comment 4

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1917191 - Enable the pref to skip duplicate frames from desktop capturers. r?#webrtc-reviewers

Comment 5

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1917191 - Don't require a minimum framerate from getDisplayMedia. r?jib

User Agents may optimize handling of idle sources by not surfacing
duplicate frames.

Comment 6

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1917191 - Replace DesktopCaptureImpl::mCallbacks by a VideoBroadcaster. r?#webrtc-reviewers

Comment 7

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1917191 - Tell Pipewire to use damage regions. r?#webrtc-reviewers!, r?grulja!