Closed Bug 1210175 Opened 10 years ago Closed 6 months ago

simulcast support for H264

Categories

(Core :: WebRTC: Audio/Video, enhancement, P3)

enhancement

Tracking

()

RESOLVED FIXED
136 Branch
Tracking Status
firefox44 --- wontfix
relnote-firefox --- 136+
firefox136 --- fixed

People

(Reporter: bwc, Assigned: pehrsons)

References

(Blocks 1 open bug, Regressed 1 open bug)

Details

(Keywords: dev-doc-complete, webcompat:platform-bug, Whiteboard: [jitsi-meet])

Attachments

(10 files, 1 obsolete file)

48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
Right now, we only get a single H264 encoder per webrtc.org channel. We either need: 1) The ability to have multiple encoders, each associated with a single simulcast stream in the webrtc.org code. or 2) Support for encoding multiple resolutions/framerates/bitrates/etc in the H264 encoder, along with the plumbing necessary to associate these encodings with the corresponding simulcast streams in the webrtc.org code.
Simulcast is a P1, but h264 in simulcast could follow the initial release if need be.
Rank: 23
Priority: -- → P2
Mass change P2->P3 to align with new Mozilla triage process.
Priority: P2 → P3
Hi guys, Is there any plan to implement h264 simulcast in FF too? seems that chrome is almost ready to ship it: https://webrtc-review.googlesource.com/c/src/+/64100

Depends on bug 1607238?

^

Flags: needinfo?(dminor)

Agreed.

Type: defect → enhancement
Rank: 23
Depends on: 1607238
Flags: needinfo?(dminor)
Whiteboard: [jitsi-meet]

More accurate dependency: Bug 1654112.

Depends on: 1654112
No longer depends on: 1607238
No longer blocks: 1716090
Severity: normal → S3

"We have implemented to reject all Firefox from the conference, only Chrome and Safari allowed, and since then we had no problems."

What can I say? Sic transit gloria mundi.

Any update on this?
Thank you

Assignee: nobody → apehrson
Status: NEW → ASSIGNED

Bitrate 0 could be configured when used with the SimulcastEncoderAdapter.

The GMP API of OpenH264 doesn't support configuring temporal layers.

fakeopenh264 works by encoding some metadata and when decoding generating an
image from that metadata. It figures out what color to use for image generation
by averaging the color of all pixels in the encoder and putting that in the
payload.

To make VideoFrameEmitter work with fakeopenh264 we must therefore use a single
color across the entire canvas.

Upstream commit: https://webrtc.googlesource.com/src/+/4a6a7465d03f0b581122f11385aca87294eebd5d
In ParseNonParameterSetNalu check BitstreamReader::Ok before returning early

   ~BitstreamReader() DCHECKs that the last read has been verified, so all
   paths where we may leave the slice_reader instance's scope early must be
   guarded by an Ok().

   Bug: None
   Change-Id: Ic67f87c04d1f042392c1dd6a066fdccf26e19003
   Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/369540
   Reviewed-by: Sergey Silkin <ssilkin@webrtc.org>
   Commit-Queue: Danil Chapovalov <danilchap@webrtc.org>
   Reviewed-by: Tomas Gunnarsson <tommi@webrtc.org>
   Reviewed-by: Danil Chapovalov <danilchap@webrtc.org>
   Cr-Commit-Position: refs/heads/main@{#43463}
Attachment #9441918 - Attachment description: Bug 1210175 - Restrict temporal layers when using simulcast to VP8. r?#webrtc-reviewers → Bug 1210175 - Don't set up temporal layers when using simulcast with H264. r?#webrtc-reviewers
No longer blocks: 1937908
Depends on: 1937908
Attachment #9441918 - Attachment is obsolete: true

Like bug 1692873 this needs a release note for developers. Something like: "WebRTC simulcast is now supported with H264 video."

Keywords: dev-doc-needed
Blocks: 1944497
Pushed by pehrsons@gmail.com: https://hg.mozilla.org/integration/autoland/rev/0ec1f62c299a Remove references to inexistent simulcast pref. r=webrtc-reviewers,bwc https://hg.mozilla.org/integration/autoland/rev/0f26930f651d Make WebRTC GMP encoder drop frames when bitrate 0 is configured. r=webrtc-reviewers,ng https://hg.mozilla.org/integration/autoland/rev/5c59074b51bb Implement simulcast for H264 behind a pref and enable it by default. r=webrtc-reviewers,bwc https://hg.mozilla.org/integration/autoland/rev/2873625fe17f Remove test_pc_simulcastOffer_screenshare max-fs modification. r=webrtc-reviewers,bwc https://hg.mozilla.org/integration/autoland/rev/8d3ace737d57 Run test_pc_simulcastOffer_screenshare.html on all codecs where we support simulcast. r=webrtc-reviewers,bwc https://hg.mozilla.org/integration/autoland/rev/d3b7681ac176 Run test_pc_simulcastOffer.html on all codecs where we support simulcast and enable it on Android. r=webrtc-reviewers,bwc https://hg.mozilla.org/integration/autoland/rev/804e73d3b870 Make fakeopenh264 handle odd width and height. r=webrtc-reviewers,media-playback-reviewers,bwc,aosmond https://hg.mozilla.org/integration/autoland/rev/60b401685488 Add an option to make VideoFrameEmitter fill the entire canvas. r=webrtc-reviewers,bwc https://hg.mozilla.org/integration/autoland/rev/1c00fa1024ca Cherry-pick upstream libwebrtc commit 4a6a7465d0 r=webrtc-reviewers,mjf https://hg.mozilla.org/integration/autoland/rev/923c6123afcb In webrtc GMP encoder set bitrate based on the allocation of all temporal layers. r=webrtc-reviewers,dbaker
Regressions: 1944573
Regressions: 1944574

:pehrsons could you consider nominating this for a release note? (Process info)

Flags: needinfo?(apehrson)

Release Note Request (optional, but appreciated)
[Why is this notable]: Simulcast is needed by any video conferencing site that want to do large video calls, where peers are connected via a media server rather than directly to each other (peer to peer). H264 is the only other codec to be supported than VP8, and H264 simulcast is supported with hardware codecs on Android.
[Affects Firefox for Android]: Yes, in particular H264 is the only video codec that we have both simulcast and hardware support for.
[Suggested wording]: Firefox now supports sending multiple simultaneous versions of the same source over WebRTC, so called simulcast, with the H264 video codec. H264 is the second video codec after VP8 to be supported for sending simulcast.
[Links (documentation, blog post, etc)]: n/a

relnote-firefox: --- → ?
Flags: needinfo?(apehrson)

Thanks, added to the Fx136 nightly release notes, please allow 30 minutes for the site to update.
Keeping the relnote-firefox flag as ? to keep it on the radar for inclusion in the final Fx136 release notes.

No longer regressions: 1944573
Blocks: 1932065
See Also: → 1947115
See Also: → 1947116

The MDN WebRTC codecs guide has this note for H.264

Firefox for Android 68 and later do not support AVC (H.264) anymore.
This is due to a change in Google Play store requirements that prevent Firefox from downloading and installing the OpenH264 codec needed to handle H.264 in WebRTC connections. See <a href="https://support.mozilla.org/en-US/kb/firefox-android-openh264">this article on SUMO</a> for details.

Obviously no longer true, since this says we support it in hardware.
Can you tell me what version we started supporting it on Android again - is it this 74 as per https://bugzilla.mozilla.org/show_bug.cgi?id=1592140 ?

FYI I'm looking at the docs in https://github.com/mdn/content/pull/38079/files
It is quite likely that the page is out of date, so welcome any suggestions.

Flags: needinfo?(apehrson)

(In reply to Hamish Willee from comment #30)

The MDN WebRTC codecs guide has this note for H.264

Firefox for Android 68 and later do not support AVC (H.264) anymore.
This is due to a change in Google Play store requirements that prevent Firefox from downloading and installing the OpenH264 codec needed to handle H.264 in WebRTC connections. See <a href="https://support.mozilla.org/en-US/kb/firefox-android-openh264">this article on SUMO</a> for details.

Obviously no longer true, since this says we support it in hardware.
Can you tell me what version we started supporting it on Android again - is it this 74 as per https://bugzilla.mozilla.org/show_bug.cgi?id=1592140 ?

Thanks for catching this! Indeed it should be working since bug 1592140.

FYI I'm looking at the docs in https://github.com/mdn/content/pull/38079/files
It is quite likely that the page is out of date, so welcome any suggestions.

I don't have much feedback to the docs on your patch, but perhaps we should mention the dependency descriptor RTP header extension, added in bug 1945261 to accompany mainly AV1 simulcast support. I'll explain it briefly here:
Older codecs like VP8 and VP9 included in their RTP payload a "payload header" with information on frame dependencies (keyframes have no dependencies, interframes typically depend on the previous frame, which depend on the previous, etc. to create a chain to the most recent keyframe). This is generally fine and SFUs can forward packets and switch simulcast layers without a receiver noticing. AV1 however doesn't have this information in the payload. Also, in scenarios involving end-to-end encryption (E2EE) the payload is encrypted and inaccessible to SFUs, for all codecs. The AV1 spec therefore defined a new RTP header extension called dependency descriptor. The RTP header (as opposed to the RTP payload header which is part of the payload) is not encrypted under E2EE. It being defined as a proper RTP header extension makes it suitable also for other codecs, and SFUs can then be made codec agnostic. Bug 1945261 added support for DD when doing simulcast with VP8, VP9 (which we don't do simulcast for by default yet, bug 1633876) and AV1. Bug 1947115 (OpenH264 on desktop) and bug 1947116 (Android H264 hw) adds support for it with our other codecs.

Flags: needinfo?(apehrson)

FF135 MDN docs for this can be tracked in https://github.com/mdn/content/pull/38079

@Andreas - thank you so much for the response. I have captured this in https://github.com/mdn/content/pull/38079 - you're welcome to review.

There wasn't a perfect place for the "DD" information so I have included it under the AV1 docs in the WebRTC codecs guide (because that is where it is in the spec). The section does note that this is generically useful and why.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: