OpenH264 has SVC capabilities, but GMP lacks a way to use it. In temporal SVC, packets tagged by their layer index when output by the encoder. If `n` temporal layers are configured on the encoder, a decoder will be able to decode the stream if all layers with lower than a certain index are decoded (in order). For example, if `n` is 3 (typical), then the encoded images are going to be encoded like so: 0 (key frame), 1, 2, 1, 0, 1, 2, 1, 0, 1, 2, 0 (forced keyframe, e.g. because too much packet loss, sequence restarts:), 1, 2, 1, 0, 1, 2, 1, 0, ... A decoder can decode the stream correctly if it only receives the frames tagged 0 and 1, or just the frames tagged 0. The stream will be at a lower frame rate, but otherwise decoding will work. Web Codecs needs to implement `L1T2` and `L1T3`, respectively, 2 and 3 temporal layers. The necessary things we need to add to the GMP interface are: The temporal layer identifier on the encoded packet: - https://searchfox.org/mozilla-central/source/dom/media/gmp/gmp-api/gmp-video-frame-encoded.h#89, something to hold the temporal layer id (a small integer that is always lower than the number of temporal layers, e.g. 0, 1 or 2 if doing `L1T3`). The SVC configuration when creating the encoder: -https://searchfox.org/mozilla-central/source/dom/media/gmp/gmp-api/gmp-video-codec.h#247, use it here: https://searchfox.org/mozilla-central/source/dom/media/platforms/agnostic/gmp/GMPVideoEncoder.cpp#151 It goes here: https://github.com/cisco/openh264/blob/3668daf135dea7d18737b01ce32774a7a2ffba32/codec/api/wels/codec_app_def.h#L550, when initing the encoder. Now, when actually encoding, we get the layer id on the struct that's the last param: https://github.com/cisco/openh264/blob/master/codec/api/wels/codec_api.h#L309, https://github.com/cisco/openh264/blob/3668daf135dea7d18737b01ce32774a7a2ffba32/codec/api/wels/codec_app_def.h#L644, we need to send that back to gecko (by setting on the encoded packet).
Bug 1917583 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
OpenH264 has SVC capabilities, but GMP lacks a way to use it. In temporal SVC, packets tagged by their layer index when output by the encoder. If `n` temporal layers are configured on the encoder, a decoder will be able to decode the stream if all layers with lower than a certain index are decoded (in order). For example, if `n` is 3 (typical), then the encoded images are going to be encoded like so: 0 (key frame), 1, 2, 1, 0, 1, 2, 1, 0, 1, 2, 0 (forced keyframe, e.g. because too much packet loss, sequence restarts:), 1, 2, 1, 0, 1, 2, 1, 0, ... A decoder can decode the stream correctly if it only receives the frames tagged 0 and 1, or just the frames tagged 0. The stream will be at a lower frame rate, but otherwise decoding will work. Web Codecs needs to implement `L1T2` and `L1T3`, respectively, 2 and 3 temporal layers. The necessary things we need to add to the GMP interface are: The temporal layer identifier on the encoded packet: - https://searchfox.org/mozilla-central/source/dom/media/gmp/gmp-api/gmp-video-frame-encoded.h#89, something to hold the temporal layer id (a small integer that is always lower than the number of temporal layers, e.g. 0, 1 or 2 if doing `L1T3`). The SVC configuration when creating the encoder: - https://searchfox.org/mozilla-central/source/dom/media/gmp/gmp-api/gmp-video-codec.h#247, use it here: https://searchfox.org/mozilla-central/source/dom/media/platforms/agnostic/gmp/GMPVideoEncoder.cpp#151 It goes here: https://github.com/cisco/openh264/blob/3668daf135dea7d18737b01ce32774a7a2ffba32/codec/api/wels/codec_app_def.h#L550, when initing the encoder. Now, when actually encoding, we get the layer id on the struct that's the last param: - https://github.com/cisco/openh264/blob/master/codec/api/wels/codec_api.h#L309 - https://github.com/cisco/openh264/blob/3668daf135dea7d18737b01ce32774a7a2ffba32/codec/api/wels/codec_app_def.h#L644, we need to send that back to gecko (by setting on the encoded packet).