1492500 - [WebRTC] VP9 video freezes when RTP flexible mode is used

Status: RESOLVED WORKSFORME

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[andysem]

Attachment: rtp_video_packets.log.xz

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0
Build ID: 20180913170256

Steps to reproduce:

If Firefox 62 receives an RTP video stream with VP9 spatial SVC, packetized using flexible mode (F=1, see https://tools.ietf.org/html/draft-ietf-payload-vp9-06#section-4.2), the browser only displays frozen keyframes, without any delta frames in between. When the stream contains no SVC (i.e. only one spatial layer), some delta frames are displayed, but then video freezes anyway until the next keyframe is received (i.e. it looks like periodic or occasional video freezes).

The same kind of streams, created with the same software, are displayed correctly in Chrome 69. If the video stream is packetized in non-flexible mode (F=0, and P_DIFF and N fields removed from VP9 payload headers), Firefox 62 displays the stream correctly. (Though, this doesn't work as a workaround because, due to e.g. https://bugs.chromium.org/p/webrtc/issues/detail?id=9679 Chrome is not able to display such stream.)

Other features of the stream and observations:
- The SVC stream contains 4 spatial layers: 160x90, 320x180, 640x360 and 1280x720. I tried with just two spatial layers and the result was the same.
- Enabling temporal SVC for a single-spatial-layer video seems to help somewhat as the video doesn't freeze but occasional frame drops can still be seen. Enabling temporal SVC for a spatial SVC stream (4 spatial layers) has no effect, only keyframes are displayed.
- VP9 payload headers in RTP packets contain the following fields: I=1, L=1, F=1, Picture ID, TID, U, SID, D, P_DIFF & N (for inter-frame predicted frames), SS (in the first packet of a keyframe). When present, SS contains N_S=3, Y=1, G=0, and width and height of each spatial layer. Picture Group parameters are not present. I tried to work around the problem by removing some of the fields, like Picture ID or SS, and it didn't help.

I'm attaching a log that contains an example problematic stream of VP9 RTP packets. Each line in the log is an unencrypted RTP packet, starting with RTP header (https://tools.ietf.org/html/rfc3550#section-5.1), which was sent to Firefox. I'm also attaching SDPs that were used to setup the stream (the offer was generated by Firefox, the answer - by our software).

Comment 1

[andysem]

Attachment: sdp.log

Comment 2

[Dan Minor [:dminor]]

We use code from webrtc.org to interface with codecs, but we're currently on branch 57 which is pretty old, from Feb 2017. I'm working on an update to branch 64, which is slightly newer, from Jan 2018, which might improve things here. That is tracked by Bug 1376873.

I've also recently updated the version of libvpx to v1.7.0 in Bug 1433158. Based on your description I doubt this would make any difference, but it would be great if you could retest with Firefox Nightly or Beta and see if the issue continues to reproduce.

The webrtc.org update is still a work in progress, but if you are willing, you could grab a recent build from here [1], for instance, here is a Linux x64 build [2] and try that as well.

Thank you for your report!

[1] https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=199952821&revision=b2e0bfe4cd7b990c564ce23edfb3f0b614aec307
[2] https://queue.taskcluster.net/v1/task/VXqcgEcsRbubtI55E-3vCg/runs/0/artifacts/public/build/target.tar.bz2

Comment 3

[andysem]

I've tested the build you linked and it behaves differently.

- When the encoded VP9 stream has no spatial SVC then it plays properly, with no freezing. I can't be 100% certain that the problem is fixed as I suspect it may not reproduce reliably, but in my limited testing the video didn't freeze once.
- When the encoded VP9 stream has 4 spatial layers, but only the base layer is sent initially then the base layer is displayed properly with no freezing. But as soon as the stream switches to an upper spatial layer (i.e. from layer 0 to 1 or from 1 to 2, etc.) the decoded video becomes corrupted and starts freezing. Layer switching only happens on keyframes, so I'm pretty sure the stream is decodable. Switching to non-flexible mode in this case fixes video freezing but not corruption. Firefox 62 in this test (spatial SVC layer changes in non-flexible mode) works without problem - no video freezing or corruption.
- If the application sends 4 spatial layers to the browser from the start, then Firefox does not display decoded video at all and only continuously requests a keyframe (which the application sends but it doesn't help). Switching to non-flexible mode does not help.

So some cases got better while other regressed.

Comment 4

[andysem]

See also this bug: https://bugs.chromium.org/p/webrtc/issues/detail?id=9789

The workaround described there does help the case with temporal SVC, in which case long-term references are not created by libvpx. But it doesn't help any other cases (i.e. no-SVC, spatial SVC and spatial+temporal SVC streams still contain long-term references and cause video freezes in Firefox 62).

Comment 5

[jaya.allamsetty]

We are seeing this issue in Jitsi as well. Whenever Firefox tries to render a VP9 stream coming from Chrome, it displays only frozen keyframes.

Comment 6

[btom1990]

Might be fixed with https://bugzilla.mozilla.org/show_bug.cgi?id=1654112

Comment 7

[btom1990]

@ja(In reply to jaya.allamsetty from comment #5)

We are seeing this issue in Jitsi as well. Whenever Firefox tries to render a VP9 stream coming from Chrome, it displays only frozen keyframes.

Could you test with the current Firefox Beta?

Comment 8

[jaya.allamsetty]

(In reply to btom1990 from comment #7)

@ja(In reply to jaya.allamsetty from comment #5)

We are seeing this issue in Jitsi as well. Whenever Firefox tries to render a VP9 stream coming from Chrome, it displays only frozen keyframes.

Could you test with the current Firefox Beta?

I gave it a quick try, the issue doesn't seem to be fixed yet. I still see freezes on Firefox when its decoding a VP9 stream. The call is between Chrome (doing 3 spatial and temporal layers) and Firefox and media is being routed by the Jitsi video bridge. On the Chrome outbound-rip stats, I see the PLI count going up steadily.

Comment 9

[Alfredo]

It could be related to the non standard SDP format used by chromium https://webrtc.org/getting-started/unified-plan-transition-guide?

Comment 10

[btom1990]

Chrome 97 removes Plan B: https://developer.chrome.com/blog/deps-rems-97/#remove-sdp-plan-b

Which version did you test with?

Comment 11

[Alfredo]

(In reply to btom1990 from comment #10)

Chrome 97 removes Plan B: https://developer.chrome.com/blog/deps-rems-97/#remove-sdp-plan-b

Which version did you test with?

Unfortunately, the deprecation of plan B is postponed to chromium 102 https://chromestatus.com/features/5823036655665152

Comment 12

[btom1990]

Jitsi seems to use Unified Plan by default since a few releases:

https://community.jitsi.org/t/switch-to-unified-plan-on-chrome/98322/21

Comment 13

[jaya.allamsetty]

Yes, I am a Jitsi dev and we switched to Unified plan on Chrome mid 2021. Even otherwise, the session itself wouldn't get established if it was a plan-b deprecation issue. The scenario that I described above was for a successful call with VP9 as the preferred codec, media sent and being received by Firefox but not rendered correctly on Firefox. The send bitrate was also low on Firefox. Unfortunately, Firefox's webrtc-internals page has very limited information which makes it hard to identify the issue. On the Chrome side, I could tell that there were lot of PLIs being received for the outbound-rtp stream which possibly means that Firefox is not able to decode video frames and is requesting key frames continuously.

Comment 14

[Juliusz Chroboczek]

I can no longer reproduce this bug using Firefox 106. (Unlike #1633876, which is apparently still there.)

Comment 15

[Nils Ohlmeier [:drno]]

I also tested this today with Fx 110 and I no longer see the freezes.

A remaining issue is that Firefox sends with 100-200kb/s instead of the 1-1.5Mb/s for VP8. As a result the video quality with VP9 is worse compared to VP8 under the same conditions. But I'll open a new bug for that.

Comment 16

[Nils Ohlmeier [:drno]]

As we have now multiple reports confirming that this problem no longer exists I'll close it.
Andysem if you disagree please comment in here so we can re-open the issue for you.