1942245 - OpenH264 screencapture encoding for webrtc accumulates latency

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[Andreas Pehrson [:pehrsons]]

I have been testing this on https://orphis.github.io/webrtc-sandbox/simulcast-playground.html?resolution=screen&captureFramerate=60&codec=H264&contentHint=&encodings.0.rid=&encodings.0.active=true&encodings.0.priority=low&encodings.0.maxBitrate=&encodings.0.scaleResolutionDownBy=1&encodings.0.maxFramerate=&encodings.0.scalabilityMode=&encodings.0.codec=&degradationPreference=&splitLayer=merged&device=default

Once started, click to view "Encoder Stats" and keep an eye on the "Avg Enc Time" which builds up.

Comment 1

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1942245 - Simplify and Modernize WebrtcGmpVideoCodec code. r?aosmond

Comment 2

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1942245 - Remove GmpInitDoneRunnable from WebrtcGmpVideoCodec. r?aosmond

Comment 3

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1942245 - Remove DataMutex from mInputImageMap. r?aosmond

Its use is single threaded.

Comment 4

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1942245 - Limit the number of images in flight in webrtc gmp encoder. r?aosmond

Frames from screen capture are often large and with OpenH264 in screen content
mode, encoding them takes several times longer than encoding non-screen content.

This can accumulate a growing backlog of frames to be encoded. Libwebrtc doesn't
adapt fast to a high encoder latency, so limit the number of frames in flight
instead.

This patch also adds reporting of frames that have been dropped, either by
OpenH264 or by the WebrtcGmpVideoEncoder.

Comment 5

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1942245 - Always allow frame drops in libwebrtc encoders. r?#webrtc-reviewers

If anything frame drops should not be prohibited when encoding screen capture,
since we don't want to downscale screen capture frames to keep text legible.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/528daad05ba8
Simplify and Modernize WebrtcGmpVideoCodec code. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/e673cfb778ea
Remove GmpInitDoneRunnable from WebrtcGmpVideoCodec. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/bb3495397f18
Remove DataMutex from mInputImageMap. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/bdb23a14a202
Limit the number of images in flight in webrtc gmp encoder. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/7f70d3a869d1
Always allow frame drops in libwebrtc encoders. r=webrtc-reviewers,ng

Comment 7

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/528daad05ba8
https://hg.mozilla.org/mozilla-central/rev/e673cfb778ea
https://hg.mozilla.org/mozilla-central/rev/bb3495397f18
https://hg.mozilla.org/mozilla-central/rev/bdb23a14a202
https://hg.mozilla.org/mozilla-central/rev/7f70d3a869d1