1999970 - Notify GMP codec impl of skipped encodings

Status: RESOLVED FIXED

(Core :: Audio/Video: GMP, task)

Description

[Andreas Pehrson [:pehrsons]]

See bug 1992567 comment 51 for details. When running the WebRTC GMP OpenH264 encoder at 4k@60 screensharing, it cannot keep up with realtime. It still seems to allocate a buffer to hold the encoded frame, which on skip is immediately dropped -- and to meet bitrate targets, OpenH264 may indeed drop frames. Turning off frame dropping through the API doesn't seem to help.

To avoid what is more or less an unbounded leak, I try to set a hard limit of maximum 1 frame allowed in flight (to the GMP process). When OpenH264 drops a frame however, the WebRTC GMP encoder doesn't learn about this and encoding stalls.

A higher limit (5) seems to work in practice such that we don't have the unbounded leak, but there are no guarantees as we'd again stall the encoding if OpenH264 were to drop 5 frames in a row.

Making the WebRTC GMP encoder synchronous again as per bug 1999747 would also not work, as it also results in an effective limit of 1 frame in flight. Adding a callback to the GMP API for signaling a dropped frame to the client, and implementing this in OpenH264, seems like the best way forward.

Comment 1

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1999970 - Retrofit a Skipped callback on top of GMP encoder API using input frame destroy calls. r?aosmond!,bwc

This allows the GMP client (in this case the WebRTC GMP encoder) to know whether
the encoder plugin decided to drop a frame. This lets us avoid flushing the
encoder with more encode jobs than it can handle.

Comment 2

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1999970 - Handle dropped-frame callbacks in the GMP PEM. r?#media-playback-reviewers

The OpenH264 encoder may silently drop a frame due to rate control.
If not handled in the GMP PEM, it may risk stalling waiting for an encoded frame
that never arrives.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/9a220092ae14
https://hg.mozilla.org/integration/autoland/rev/9e5dffd97e3a
Retrofit a Skipped callback on top of GMP encoder API using input frame destroy calls. r=aosmond,media-playback-reviewers,webrtc-reviewers,bwc
https://github.com/mozilla-firefox/firefox/commit/20ccceccb14f
https://hg.mozilla.org/integration/autoland/rev/39d124ba4e96
Handle dropped-frame callbacks in the GMP PEM. r=media-playback-reviewers,padenot

Comment 4

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/9e5dffd97e3a
https://hg.mozilla.org/mozilla-central/rev/39d124ba4e96

Comment 5

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1999970 - Retrofit a Skipped callback on top of GMP encoder API using input frame destroy calls.

This allows the GMP client (in this case the WebRTC GMP encoder) to know whether
the encoder plugin decided to drop a frame. This lets us avoid flushing the
encoder with more encode jobs than it can handle.

Original Revision: https://phabricator.services.mozilla.com/D272613

Comment 6

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1999970 - Handle dropped-frame callbacks in the GMP PEM.

The OpenH264 encoder may silently drop a frame due to rate control.
If not handled in the GMP PEM, it may risk stalling waiting for an encoded frame
that never arrives.

Original Revision: https://phabricator.services.mozilla.com/D274755

Comment 7

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr140/rev/f1785c071117
https://hg.mozilla.org/releases/mozilla-esr140/rev/465363e6a20e