1917583 - Teach GMP to do SVC when using OpenH264

Status: RESOLVED FIXED

(Core :: Audio/Video: Web Codecs, defect, P3)

Description

[Paul Adenot (:padenot)]

OpenH264 has SVC capabilities, but GMP lacks a way to use it. In temporal SVC, packets tagged by their layer index when output by the encoder. If n temporal layers are configured on the encoder, a decoder will be able to decode the stream if all layers with lower than a certain index are decoded (in order). For example, if n is 3 (typical), then the encoded images are going to be encoded like so:

0 (key frame), 1, 2, 1, 0, 1, 2, 1, 0, 1, 2, 0 (forced keyframe, e.g. because too much packet loss, sequence restarts:), 1, 2, 1, 0, 1, 2, 1, 0, ...

A decoder can decode the stream correctly if it only receives the frames tagged 0 and 1, or just the frames tagged 0. The stream will be at a lower frame rate, but otherwise decoding will work.

Web Codecs needs to implement L1T2 and L1T3, respectively, 2 and 3 temporal layers.

The necessary things we need to add to the GMP interface are:

The temporal layer identifier on the encoded packet:

• https://searchfox.org/mozilla-central/source/dom/media/gmp/gmp-api/gmp-video-frame-encoded.h#89, something to hold the temporal layer id (a small integer that is always lower than the number of temporal layers, e.g. 0, 1 or 2 if doing L1T3).

The SVC configuration when creating the encoder:

• https://searchfox.org/mozilla-central/source/dom/media/gmp/gmp-api/gmp-video-codec.h#247, use it here: https://searchfox.org/mozilla-central/source/dom/media/platforms/agnostic/gmp/GMPVideoEncoder.cpp#151

It goes here: https://github.com/cisco/openh264/blob/3668daf135dea7d18737b01ce32774a7a2ffba32/codec/api/wels/codec_app_def.h#L550, when initing the encoder.

Now, when actually encoding, we get the layer id on the struct that's the last param:

• https://github.com/cisco/openh264/blob/master/codec/api/wels/codec_api.h#L309
• https://github.com/cisco/openh264/blob/3668daf135dea7d18737b01ce32774a7a2ffba32/codec/api/wels/codec_app_def.h#L644, we need to send that back to gecko (by setting on the encoded packet).

Comment 1

[Andrew Osmond [:aosmond] (he/him)]

Attachment: mozilla/gmp-api GitHub Pull Request

Comment 2

[Andrew Osmond [:aosmond] (he/him)]

Attachment: cisco/openh264 GitHub Pull Request

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1917583 - Make OpenH264 GMP video encoder support SVC.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ea6f6e7afb1f
Make OpenH264 GMP video encoder support SVC. r=media-playback-reviewers,chunmin

Comment 5

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/ea6f6e7afb1f