1999970 - Notify GMP codec impl of skipped encodings

Status: ASSIGNED

(Core :: Audio/Video: GMP, task)

Description

[Andreas Pehrson [:pehrsons]]

See bug 1992567 comment 51 for details. When running the WebRTC GMP OpenH264 encoder at 4k@60 screensharing, it cannot keep up with realtime. It still seems to allocate a buffer to hold the encoded frame, which on skip is immediately dropped -- and to meet bitrate targets, OpenH264 may indeed drop frames. Turning off frame dropping through the API doesn't seem to help.

To avoid what is more or less an unbounded leak, I try to set a hard limit of maximum 1 frame allowed in flight (to the GMP process). When OpenH264 drops a frame however, the WebRTC GMP encoder doesn't learn about this and encoding stalls.

A higher limit (5) seems to work in practice such that we don't have the unbounded leak, but there are no guarantees as we'd again stall the encoding if OpenH264 were to drop 5 frames in a row.

Making the WebRTC GMP encoder synchronous again as per bug 1999747 would also not work, as it also results in an effective limit of 1 frame in flight. Adding a callback to the GMP API for signaling a dropped frame to the client, and implementing this in OpenH264, seems like the best way forward.

Comment 1

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1999970 - Retrofit a Skipped callback on top of GMP encoder API using input frame destroy calls. r?aosmond!,bwc

This allows the GMP client (in this case the WebRTC GMP encoder) to know whether
the encoder plugin decided to drop a frame. This lets us avoid flushing the
encoder with more encode jobs than it can handle.