1633876 - WebRTC does not send simulcast when VP9 is negotiated

Status: NEW

(Core :: WebRTC: Audio/Video, defect, P3)

Description

[Jonathan Lennox]

Attachment: SDP offer/answer from about:webrtc

When I negotiate a WebRTC VP9 stream with simulcast, Firefox (Nightly) only sends a single VP9 video stream, not the requested three streams.

The resulting video stream has temporal scalability, but no spatial scalability.

In the same scenario with Chrome, Chrome sends a single spatially-scalable stream, with three spatial layers. My suspicion is that in a libwebrtc import, the "single stream" part was carried over but the "SVC" part was not.

(This is with FF Nightly so it is not Bug 1628851.)

I tested this with a jitsi-meet installation configured to negotiate VP9 rather than VP8. I can give access if desired.

Comment 1

[Alex Chronopoulos [:achronop]]

Byron, can you please help me triage this issue?

Comment 2

[Byron Campen [:bwc]]

I don't see VP9 in that attachment at all. Is that the right attachment?

Comment 3

[Byron Campen [:bwc]]

Yeah, this does in fact seem to be the case. Disabling VP8 in this test case breaks the test:

https://searchfox.org/mozilla-central/source/dom/media/tests/mochitest/test_peerConnection_simulcastAnswer.html

Nothing about our signaling code treats VP8 and VP9 differently in any significant way, so this is probably a limitation/bug in webrtc.org, or perhaps the VP9 encoder. Any ideas here Dan?

Comment 4

[Jonathan Lennox]

Attachment: Offer/answer from VP9 failed simulcast

Comment 5

[Jonathan Lennox]

Yes, sorry, the original attachment was the working VP8 simulcast - here's the VP9. Notice that there's very little difference between them.

The webrtc.org code has a whole bunch of very opaque field trial flags for VP9 SVC, so maybe those are being set incorrectly in the Firefox import.

Comment 6

[Dan Minor [:dminor]]

I suspect this was caused by Bug 1525393. We wanted to update libvpx for security reasons, but our version of libwebrtc was to old to work with it. I ended up disabling some vp9 simulcast code to get things building (see https://phabricator.services.mozilla.com/D63925, which was partial cherrypick of an upstream commit). The upstream libwebrtc changes required to work with the newer libvpx were too substantial to backport.

It's definitely possible I missed something when notching out that code, but I'm afraid we might be stuck like this until we update libwebrtc again.

Comment 7

[Byron Campen [:bwc]]

I guess we need to have separate simulcast tests for VP8 and VP9?

Comment 8

[Dan Minor [:dminor]]

I double-checked, and https://searchfox.org/mozilla-central/source/dom/media/tests/mochitest/test_peerConnection_simulcastAnswer.html fails with VP8 disabled with a build from prior to the libvpx update in Bug 1525393 landing, so I guess there is a pre-existing issue. It is still possible that the changes in Bug 1525393 would prevent us from fixing this.

Comment 9

[Nils Ohlmeier [:drno]]

I vaguely recall that years ago VP8 SVC got enabled in Firefox, because people requested it. At the time VP9 wasn't in use at all. So I would not be surprised if still somewhere in the libwebrtc or libvpx SVC only gets turned on for VP8, but not for VP9. Maybe Randell still remembers where SVC gets turned on/off?

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

I'm not sure where... Nor do I remember doing that. Perhaps it just came along in an update? I don't see any bugs on it, or any checkin messages in the logs in media/webrtc.

Andreas?

Comment 11

[Andreas Pehrson [:pehrsons]]

I see bug 1343143 for temporal scalability with VP8. But there are no tests so I wouldn't be surprised if it's broken.

Then there's bug 1341285 that added the prefs media.peerconnection.video.svc.spatial and media.peerconnection.video.svc.temporal for controlling VP9 SVC parameters. But I think it was mostly to have something to plumb into libwebrtc.

The biggest hurdle to implementing this is probably tests. The implementation is mainly configuration/signaling bits. We do have tests for just simulcast for vp8 so getting something going for simulcast for other codecs is probably not that hard. SVC OTOH I'm not so sure about.

Comment 12

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

STRs:

1. Open https://jsfiddle.net/jib1/Lq5zm39h/show
2. Select VP9
3. Hit the gUM! button

Expected result (e.g. skip step 2 to use VP8 which WFM):

• 3 videos: small, medium and large

Actual result:

• One large video only.

This test uses a mid<->rid switching trick to separate and receive individual simulcast streams in a browser. The difference here between VP8 and VP9 seems a clear indication only the main layer of the simulcast output is received.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(The fiddle in comment 12 doesn't work in Chrome unfortunately, since Chrome doesn't like the way we modify SDP to avoid bug 1396922. I'll try to update the fiddle to use setCodecPreferences if available)

Comment 14

[Ckristian Zoli]

Is there any ETA for this to be addressed?

We, Vonage, are releasing VP9 SVC, and we would like to be able to support FF as well.

Comment 15

[Andreas Pehrson [:pehrsons]]

I am currently looking at simulcast. We'll probably do something for VP9, though it's not an obvious thing to ship given Chrome's behavior to interpret it as SVC.

SVC through scalabilityMode is also on the horizon, as you can see in bug 1571470.