1210175 - simulcast support for H264

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, enhancement, P3)

Description

[Byron Campen [:bwc]]

Right now, we only get a single H264 encoder per webrtc.org channel. We either need:

1) The ability to have multiple encoders, each associated with a single simulcast stream in the webrtc.org code.

or

2) Support for encoding multiple resolutions/framerates/bitrates/etc in the H264 encoder, along with the plumbing necessary to associate these encodings with the corresponding simulcast streams in the webrtc.org code.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Simulcast is a P1, but h264 in simulcast could follow the initial release if need be.

Comment 2

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.

Comment 3

[cwehrung]

FYI Chrome will get this soon: https://bugs.chromium.org/p/webrtc/issues/detail?id=5840

Comment 4

[Francesco Durighetto]

Hi guys,
Is there any plan to implement h264 simulcast in FF too?
seems that chrome is almost ready to ship it:
https://webrtc-review.googlesource.com/c/src/+/64100

Comment 5

[mirh]

Depends on bug 1607238?

Comment 6

[Byron Campen [:bwc]]

^

Comment 7

[Thatoo]

and thus Jitsi is only working with Google product...
Too bad...

Comment 8

[Dan Minor [:dminor]]

Agreed.

Comment 10

[Richard Newman [:rnewman]]

More accurate dependency: Bug 1654112.

Comment 11

[Peter Gervai]

"We have implemented to reject all Firefox from the conference, only Chrome and Safari allowed, and since then we had no problems."

What can I say? Sic transit gloria mundi.

Comment 12

[Alfredo]

Any update on this?
Thank you

Comment 13

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - Remove references to inexistent simulcast pref. r?#webrtc-reviewers

Comment 14

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - Make WebRTC GMP encoder drop frames when bitrate 0 is configured. r?#webrtc-reviewers

Bitrate 0 could be configured when used with the SimulcastEncoderAdapter.

Comment 15

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - Don't set up temporal layers when using simulcast with H264. r?#webrtc-reviewers

The GMP API of OpenH264 doesn't support configuring temporal layers.

Comment 16

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - Implement simulcast for H264 behind a pref and enable it by default. r?#webrtc-reviewers

Comment 17

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - Remove test_pc_simulcastOffer_screenshare max-fs modification. r?#webrtc-reviewers

It is unnecessary since bug 1932381 landed.

Comment 18

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - Run test_pc_simulcastOffer_screenshare.html on all codecs where we support simulcast. r?#webrtc-reviewers

Comment 19

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - Run test_pc_simulcastOffer.html on all codecs where we support simulcast and enable it on Android. r?#webrtc-reviewers

Comment 20

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - Make fakeopenh264 handle odd width and height. r?#webrtc-reviewers

Comment 21

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - Add an option to make VideoFrameEmitter fill the entire canvas. r?#webrtc-reviewers

fakeopenh264 works by encoding some metadata and when decoding generating an
image from that metadata. It figures out what color to use for image generation
by averaging the color of all pixels in the encoder and putting that in the
payload.

To make VideoFrameEmitter work with fakeopenh264 we must therefore use a single
color across the entire canvas.

Comment 22

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - Cherry-pick upstream libwebrtc commit 4a6a7465d0 r?#webrtc-reviewers

Upstream commit: https://webrtc.googlesource.com/src/+/4a6a7465d03f0b581122f11385aca87294eebd5d
In ParseNonParameterSetNalu check BitstreamReader::Ok before returning early

   ~BitstreamReader() DCHECKs that the last read has been verified, so all
   paths where we may leave the slice_reader instance's scope early must be
   guarded by an Ok().

   Bug: None
   Change-Id: Ic67f87c04d1f042392c1dd6a066fdccf26e19003
   Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/369540
   Reviewed-by: Sergey Silkin <[e-mail]>
   Commit-Queue: Danil Chapovalov <[e-mail]>
   Reviewed-by: Tomas Gunnarsson <[e-mail]>
   Reviewed-by: Danil Chapovalov <[e-mail]>
   Cr-Commit-Position: refs/heads/main@{#43463}

Comment 23

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Like bug 1692873 this needs a release note for developers. Something like: "WebRTC simulcast is now supported with H264 video."

Comment 24

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1210175 - In webrtc GMP encoder set bitrate based on the allocation of all temporal layers. r?#webrtc-reviewers

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ec1f62c299a
Remove references to inexistent simulcast pref. r=webrtc-reviewers,bwc
https://hg.mozilla.org/integration/autoland/rev/0f26930f651d
Make WebRTC GMP encoder drop frames when bitrate 0 is configured. r=webrtc-reviewers,ng
https://hg.mozilla.org/integration/autoland/rev/5c59074b51bb
Implement simulcast for H264 behind a pref and enable it by default. r=webrtc-reviewers,bwc
https://hg.mozilla.org/integration/autoland/rev/2873625fe17f
Remove test_pc_simulcastOffer_screenshare max-fs modification. r=webrtc-reviewers,bwc
https://hg.mozilla.org/integration/autoland/rev/8d3ace737d57
Run test_pc_simulcastOffer_screenshare.html on all codecs where we support simulcast. r=webrtc-reviewers,bwc
https://hg.mozilla.org/integration/autoland/rev/d3b7681ac176
Run test_pc_simulcastOffer.html on all codecs where we support simulcast and enable it on Android. r=webrtc-reviewers,bwc
https://hg.mozilla.org/integration/autoland/rev/804e73d3b870
Make fakeopenh264 handle odd width and height. r=webrtc-reviewers,media-playback-reviewers,bwc,aosmond
https://hg.mozilla.org/integration/autoland/rev/60b401685488
Add an option to make VideoFrameEmitter fill the entire canvas. r=webrtc-reviewers,bwc
https://hg.mozilla.org/integration/autoland/rev/1c00fa1024ca
Cherry-pick upstream libwebrtc commit 4a6a7465d0 r=webrtc-reviewers,mjf
https://hg.mozilla.org/integration/autoland/rev/923c6123afcb
In webrtc GMP encoder set bitrate based on the allocation of all temporal layers. r=webrtc-reviewers,dbaker

Comment 26

[amarc]

https://hg.mozilla.org/mozilla-central/rev/0ec1f62c299a
https://hg.mozilla.org/mozilla-central/rev/0f26930f651d
https://hg.mozilla.org/mozilla-central/rev/5c59074b51bb
https://hg.mozilla.org/mozilla-central/rev/2873625fe17f
https://hg.mozilla.org/mozilla-central/rev/8d3ace737d57
https://hg.mozilla.org/mozilla-central/rev/d3b7681ac176
https://hg.mozilla.org/mozilla-central/rev/804e73d3b870
https://hg.mozilla.org/mozilla-central/rev/60b401685488
https://hg.mozilla.org/mozilla-central/rev/1c00fa1024ca
https://hg.mozilla.org/mozilla-central/rev/923c6123afcb

Comment 27

[Donal Meehan [:dmeehan]]

:pehrsons could you consider nominating this for a release note? (Process info)

Comment 28

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Release Note Request (optional, but appreciated)
[Why is this notable]: Simulcast is needed by any video conferencing site that want to do large video calls, where peers are connected via a media server rather than directly to each other (peer to peer). H264 is the only other codec to be supported than VP8, and H264 simulcast is supported with hardware codecs on Android.
[Affects Firefox for Android]: Yes, in particular H264 is the only video codec that we have both simulcast and hardware support for.
[Suggested wording]: Firefox now supports sending multiple simultaneous versions of the same source over WebRTC, so called simulcast, with the H264 video codec. H264 is the second video codec after VP8 to be supported for sending simulcast.
[Links (documentation, blog post, etc)]: n/a

Comment 29

[Donal Meehan [:dmeehan]]

Thanks, added to the Fx136 nightly release notes, please allow 30 minutes for the site to update.
Keeping the relnote-firefox flag as ? to keep it on the radar for inclusion in the final Fx136 release notes.

Comment 30

[Hamish Willee]

The MDN WebRTC codecs guide has this note for H.264

Firefox for Android 68 and later do not support AVC (H.264) anymore.
This is due to a change in Google Play store requirements that prevent Firefox from downloading and installing the OpenH264 codec needed to handle H.264 in WebRTC connections. See <a href="https://support.mozilla.org/en-US/kb/firefox-android-openh264">this article on SUMO</a> for details.

Obviously no longer true, since this says we support it in hardware.
Can you tell me what version we started supporting it on Android again - is it this 74 as per https://bugzilla.mozilla.org/show_bug.cgi?id=1592140 ?

FYI I'm looking at the docs in https://github.com/mdn/content/pull/38079/files
It is quite likely that the page is out of date, so welcome any suggestions.

Comment 31

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

(In reply to Hamish Willee from comment #30)

The MDN WebRTC codecs guide has this note for H.264

Firefox for Android 68 and later do not support AVC (H.264) anymore.
This is due to a change in Google Play store requirements that prevent Firefox from downloading and installing the OpenH264 codec needed to handle H.264 in WebRTC connections. See <a href="https://support.mozilla.org/en-US/kb/firefox-android-openh264">this article on SUMO</a> for details.

Obviously no longer true, since this says we support it in hardware.
Can you tell me what version we started supporting it on Android again - is it this 74 as per https://bugzilla.mozilla.org/show_bug.cgi?id=1592140 ?

Thanks for catching this! Indeed it should be working since bug 1592140.

FYI I'm looking at the docs in https://github.com/mdn/content/pull/38079/files
It is quite likely that the page is out of date, so welcome any suggestions.

I don't have much feedback to the docs on your patch, but perhaps we should mention the dependency descriptor RTP header extension, added in bug 1945261 to accompany mainly AV1 simulcast support. I'll explain it briefly here:
Older codecs like VP8 and VP9 included in their RTP payload a "payload header" with information on frame dependencies (keyframes have no dependencies, interframes typically depend on the previous frame, which depend on the previous, etc. to create a chain to the most recent keyframe). This is generally fine and SFUs can forward packets and switch simulcast layers without a receiver noticing. AV1 however doesn't have this information in the payload. Also, in scenarios involving end-to-end encryption (E2EE) the payload is encrypted and inaccessible to SFUs, for all codecs. The AV1 spec therefore defined a new RTP header extension called dependency descriptor. The RTP header (as opposed to the RTP payload header which is part of the payload) is not encrypted under E2EE. It being defined as a proper RTP header extension makes it suitable also for other codecs, and SFUs can then be made codec agnostic. Bug 1945261 added support for DD when doing simulcast with VP8, VP9 (which we don't do simulcast for by default yet, bug 1633876) and AV1. Bug 1947115 (OpenH264 on desktop) and bug 1947116 (Android H264 hw) adds support for it with our other codecs.

Comment 32

[Hamish Willee]

FF135 MDN docs for this can be tracked in https://github.com/mdn/content/pull/38079

@Andreas - thank you so much for the response. I have captured this in https://github.com/mdn/content/pull/38079 - you're welcome to review.

There wasn't a perfect place for the "DD" information so I have included it under the AV1 docs in the WebRTC codecs guide (because that is where it is in the spec). The section does note that this is generically useful and why.