Closed Bug 1917800 Opened 1 year ago Closed 9 months ago

SetCodecPreference on Ingress freezes streams

Categories

(Core :: WebRTC: Audio/Video, defect)

Product:
Core
Component:
WebRTC: Audio/Video
Version:
Firefox 128
Type:
defect

Tracking

()

Status:
RESOLVED INCOMPLETE

People

(Reporter: k0nserv, Unassigned)

Details

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:130.0) Gecko/20100101 Firefox/130.0

Steps to reproduce:

We noticed a regression in Firefox in our WebRTC video product(https://goteam.video) in release 128. This release enables setCodecPreferences which we use as follows:

        const capabilities = RTCRtpSender.getCapabilities('video');

        const desiredCodecs = capabilities?.codecs.filter(({ mimeType, sdpFmtpLine }) => {
            // Restrict codecs based on what we support upstream and to work around this bug
            // https://bugs.chromium.org/p/chromium/issues/detail?id=1346307
            const mimeTypeLowered = mimeType.toLowerCase();
            const isVP9 = mimeTypeLowered === 'video/vp9';
            const isAV1 = mimeTypeLowered === 'video/av1';
            const isPacketizationMode0H264 =
                mimeTypeLowered === 'video/h264' && sdpFmtpLine?.includes('packetization-mode=0');

            return !(isVP9 || isAV1 || isPacketizationMode0H264);
        });

        if (desiredCodecs) {
            transceiver.setCodecPreferences(desiredCodecs);
        }

This is applied on both transceivers created by remote offers(from onTrack) and those created in the client itself.

It turns out it is the use of setCodecPreferences for transceivers created in response to a remote offer that causes the problem. The temporary fix I'm shipping, uses user agent sniffing to disable the use of setCodecPreferences for transceivers created in response to remote offers.

Actual results:

Reproduction steps

  1. Change user agent in Firefox 128 to a Chrome user agent.
  2. Open https://goteam.video/<room_id>?nomic in Firefox, use any alphanumeric value for room_id
  3. Open https://goteam.video/<room_id>?nomic in Chrome
  4. In Chrome turn on the camera by pressing "c" or clicking the camera button
  5. Observe the Camera stream being received in Firefox
  6. In Firefox turn on the camera by pressing "c" or clicking the camera button
  7. Observe the incoming stream from Chrome freezing

Details

The SDP the server offers to Firefox when the camera is added looks like this

      "m=video 9 UDP/TLS/RTP/SAVPF 96 97",
      "c=IN IP4 0.0.0.0",
      "a=ice-ufrag:VUAV",
      "a=ice-pwd:zojvDUPnSJOIG6h0rBanNl",
      "a=ice-options:trickle",
      "a=fingerprint:sha-256 24:6D:C8:D0:87:57:8E:E8:CA:7D:0C:57:39:D8:FC:DB:C6:40:88:48:B3:85:8E:7F:37:0C:63:28:DA:89:EB:EB",
      "a=setup:passive",
      "a=mid:Nkb",
      "a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time",
      "a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01",
      "a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid",
      "a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id",
      "a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id",
      "a=extmap:13 urn:3gpp:video-orientation",
      "a=sendonly",
      "a=msid:hAtykzT72H5bGLVKpVEz.camera AqDG5TFfgS7SZruXL.cam",
      "a=rtcp-mux",
      "a=rtpmap:96 VP8/90000",
      "a=rtcp-fb:96 transport-cc",
      "a=rtcp-fb:96 goog-remb",
      "a=rtcp-fb:96 ccm fir",
      "a=rtcp-fb:96 nack",
      "a=rtcp-fb:96 nack pli",
      "a=rtpmap:97 rtx/90000",
      "a=fmtp:97 apt=96",
      "a=ssrc:3958593893 cname:AqDG5TFfgS7SZruXL.cam",
      "a=ssrc:3958593893 msid:hAtykzT72H5bGLVKpVEz.camera AqDG5TFfgS7SZruXL.cam",
      "a=ssrc:941523860 cname:AqDG5TFfgS7SZruXL.cam",
      "a=ssrc:941523860 msid:hAtykzT72H5bGLVKpVEz.camera AqDG5TFfgS7SZruXL.cam",
      "a=ssrc-group:FID 3958593893 941523860",

and Firefox's response

      "m=video 9 UDP/TLS/RTP/SAVPF 96 97",
      "c=IN IP4 0.0.0.0",
      "a=recvonly",
      "a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time",
      "a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01",
      "a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid",
      "a=fmtp:96 max-fs=12288;max-fr=60",
      "a=fmtp:97 apt=96",
      "a=ice-pwd:6e269f17eef6168dc84e230a6c7f482d",
      "a=ice-ufrag:dcbe7b63",
      "a=mid:Nkb",
      "a=rtcp-fb:96 nack",
      "a=rtcp-fb:96 nack pli",
      "a=rtcp-fb:96 ccm fir",
      "a=rtcp-fb:96 goog-remb",
      "a=rtcp-fb:96 transport-cc",
      "a=rtcp-mux",
      "a=rtpmap:96 VP8/90000",
      "a=rtpmap:97 rtx/90000",
      "a=setup:active",
      "a=ssrc:1737335907 cname:{2024a02d-0b50-4d5d-8e10-110b2bb3dbdc}",

This is fine. 96 is agreed as the payload number for VP8 and 97 is agreed as the RTX.

Here is the offer when a camera stream is added in Firefox

      "m=video 9 UDP/TLS/RTP/SAVPF 120 124 126 127 97 98 123 122 119",
      "c=IN IP4 0.0.0.0",
      "a=recvonly",
      "a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid",
      "a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time",
      "a=extmap:5 urn:ietf:params:rtp-hdrext:toffset",
      "a=extmap:6/recvonly http://www.webrtc.org/experiments/rtp-hdrext/playout-delay",
      "a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01",
      "a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1",
      "a=fmtp:98 apt=97",
      "a=fmtp:120 max-fs=12288;max-fr=60",
      "a=fmtp:124 apt=120",
      "a=fmtp:127 apt=126",
      "a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1",
      "a=fmtp:119 apt=122",
      "a=ice-pwd:dfc23c2330ae4aaf49565afdf46f259e",
      "a=ice-ufrag:b0378bf0",
      "a=mid:co3",
      "a=rtcp-fb:96 nack",
      "a=rtcp-fb:96 nack pli",
      "a=rtcp-fb:96 ccm fir",
      "a=rtcp-fb:96 goog-remb",
      "a=rtcp-fb:96 transport-cc",
      "a=rtcp-fb:121 nack",
      "a=rtcp-fb:121 nack pli",
      "a=rtcp-fb:121 ccm fir",
      "a=rtcp-fb:121 goog-remb",
      "a=rtcp-fb:121 transport-cc",
      "a=rtcp-fb:126 nack",
      "a=rtcp-fb:126 nack pli",
      "a=rtcp-fb:126 ccm fir",
      "a=rtcp-fb:126 goog-remb",
      "a=rtcp-fb:126 transport-cc",
      "a=rtcp-fb:98 nack",
      "a=rtcp-fb:98 nack pli",
      "a=rtcp-fb:98 ccm fir",
      "a=rtcp-fb:98 goog-remb",
      "a=rtcp-fb:98 transport-cc",
      "a=rtcp-fb:123 nack",
      "a=rtcp-fb:123 nack pli",
      "a=rtcp-fb:123 ccm fir",
      "a=rtcp-fb:123 goog-remb",
      "a=rtcp-fb:123 transport-cc",
      "a=rtcp-fb:122 nack",
      "a=rtcp-fb:122 nack pli",
      "a=rtcp-fb:122 ccm fir",
      "a=rtcp-fb:122 goog-remb",
      "a=rtcp-fb:122 transport-cc",
      "a=rtcp-fb:120 nack",
      "a=rtcp-fb:120 nack pli",
      "a=rtcp-fb:120 ccm fir",
      "a=rtcp-fb:97 nack",
      "a=rtcp-fb:97 nack pli",
      "a=rtcp-fb:97 ccm fir",
      "a=rtcp-mux",
      "a=rtcp-rsize",
      "a=rtpmap:120 VP8/90000",
      "a=rtpmap:124 rtx/90000",
      "a=rtpmap:126 H264/90000",
      "a=rtpmap:127 rtx/90000",
      "a=rtpmap:97 H264/90000",
      "a=rtpmap:98 rtx/90000",
      "a=rtpmap:123 ulpfec/90000",
      "a=rtpmap:122 red/90000",
      "a=rtpmap:119 rtx/90000",
      "a=setup:actpass",
      "a=ssrc:3065917274 cname:{92273a1a-f292-435b-bbe3-b6f59408d325}",

This seems wrong for several reasons: The agreed payload numbers 96 and 97 are missing entirely from the m= line. However, there are still rtcp-fb lines that mention them, in addition 97 has changed from RTX/90000 to H264/90000.

The answer from our server keeps 96 and 97 with the same mappings, but it should probably reject the incoming offer.

Expected results:

Firefox should have offered the new stream with VP8 mapped to 96 and RTX/90000 mapped to 97 as it accepted in the incoming offer.

Flags: needinfo?(docfaraday)

Point of clarification. In the offer from Firefox SDP I only included the existing stream from Chrome mid: co3. I should of course have included the m= line for the newly added track too mid: 1. Here it is

      "m=video 0 UDP/TLS/RTP/SAVPF 120 124 126 127 97 98 123 122 119",
      "c=IN IP4 0.0.0.0",
      "a=bundle-only",
      "a=sendonly",
      "a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid",
      "a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time",
      "a=extmap:5 urn:ietf:params:rtp-hdrext:toffset",
      "a=extmap:6/recvonly http://www.webrtc.org/experiments/rtp-hdrext/playout-delay",
      "a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01",
      "a=extmap:10/sendonly urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id",
      "a=extmap:11/sendonly urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id",
      "a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1",
      "a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1",
      "a=fmtp:120 max-fs=12288;max-fr=60",
      "a=fmtp:124 apt=120",
      "a=fmtp:127 apt=126",
      "a=fmtp:98 apt=97",
      "a=fmtp:119 apt=122",
      "a=ice-pwd:dfc23c2330ae4aaf49565afdf46f259e",
      "a=ice-ufrag:b0378bf0",
      "a=mid:1",
      "a=msid:{c293814a-eed5-4734-83ca-bd8c32960ca9} {6b609dc4-40cb-4d33-8ac4-69b18ccc85ed}",
      "a=rid:l send",
      "a=rid:m send",
      "a=rid:h send",
      "a=rtcp-fb:120 nack",
      "a=rtcp-fb:120 nack pli",
      "a=rtcp-fb:120 ccm fir",
      "a=rtcp-fb:120 goog-remb",
      "a=rtcp-fb:120 transport-cc",
      "a=rtcp-fb:121 nack",
      "a=rtcp-fb:121 nack pli",
      "a=rtcp-fb:121 ccm fir",
      "a=rtcp-fb:121 goog-remb",
      "a=rtcp-fb:121 transport-cc",
      "a=rtcp-fb:126 nack",
      "a=rtcp-fb:126 nack pli",
      "a=rtcp-fb:126 ccm fir",
      "a=rtcp-fb:126 goog-remb",
      "a=rtcp-fb:126 transport-cc",
      "a=rtcp-fb:97 nack",
      "a=rtcp-fb:97 nack pli",
      "a=rtcp-fb:97 ccm fir",
      "a=rtcp-fb:97 goog-remb",
      "a=rtcp-fb:97 transport-cc",
      "a=rtcp-fb:123 nack",
      "a=rtcp-fb:123 nack pli",
      "a=rtcp-fb:123 ccm fir",
      "a=rtcp-fb:123 goog-remb",
      "a=rtcp-fb:123 transport-cc",
      "a=rtcp-fb:122 nack",
      "a=rtcp-fb:122 nack pli",
      "a=rtcp-fb:122 ccm fir",
      "a=rtcp-fb:122 goog-remb",
      "a=rtcp-fb:122 transport-cc",
      "a=rtcp-mux",
      "a=rtcp-rsize",
      "a=rtpmap:120 VP8/90000",
      "a=rtpmap:124 rtx/90000",
      "a=rtpmap:126 H264/90000",
      "a=rtpmap:127 rtx/90000",
      "a=rtpmap:97 H264/90000",
      "a=rtpmap:98 rtx/90000",
      "a=rtpmap:123 ulpfec/90000",
      "a=rtpmap:122 red/90000",
      "a=rtpmap:119 rtx/90000",
      "a=setup:actpass",
      "a=simulcast:send l;m;h",
      "a=ssrc:1592217545 cname:{92273a1a-f292-435b-bbe3-b6f59408d325}",
      "a=ssrc:1436240394 cname:{92273a1a-f292-435b-bbe3-b6f59408d325}",
      "a=ssrc:4135477427 cname:{92273a1a-f292-435b-bbe3-b6f59408d325}",
      "a=ssrc:1744076442 cname:{92273a1a-f292-435b-bbe3-b6f59408d325}",
      "a=ssrc:3648878484 cname:{92273a1a-f292-435b-bbe3-b6f59408d325}",
      "a=ssrc:2199984778 cname:{92273a1a-f292-435b-bbe3-b6f59408d325}",
      "a=ssrc-group:FID 1592217545 1436240394",
      "a=ssrc-group:FID 4135477427 1744076442",
      "a=ssrc-group:FID 3648878484 2199984778",

It has much the same problem of remapping the payload number 97 and omitting 96 entirely

In fact, even after removing setCodecPreferences on the incoming transceiver Firefox does the wrong thing it seems.

The existing track mid: co3 uses 96 as VP8/90000 and 97 as RTX/90000, but for the new track mid: 1 it does the same thing where it remaps 97 to h264/90000 and gets rid of payload number 96.

My understanding is that because these are all part of the same bundle group the payload numbers across all video tracks must use the same mappings

Let's loop in Dan as well, he has worked on setCodecPreferences recently.

Flags: needinfo?(dbaker)

I was consulting the specs and I want to retract my point about the payload number mappings. I forgot that it differs based on direction and whether offering or answering.

Still something is a bit odd when doing setCodecPreferences on an incoming receiver(payload 96 being referenced in rtcp-fb when not present as a format). NB: We entirely removed this because it's apparently no longer needed to work around the bug in Chromium, this means the reproduction case is no longer relevant.

Sorry about the confusion

So we're definitely seeing bug 1909443 here, is there anything wrong besides the errant rtcp-fb lines?

Flags: needinfo?(docfaraday) → needinfo?(light.moon5419)

So we're definitely seeing bug 1909443 here, is there anything wrong besides the errant rtcp-fb lines?

Yeah I think so, but it's not clear from that bug report if any incoming streams freeze as we were seeing.

Flags: needinfo?(light.moon5419)

I'm not able to reproduce this with the given steps. I was seeing setCodecPreferences being called in my debugger and video playing on both Firefox and Chrome.

  • Could you provide your about:support info
  • If possible a packet capture when this is happening along with the information from about:webrtc could help.

In addition would you be willing to get a profile log when this is occurring (may be a good idea to test using a Nightly build).

  • Close all tabs
  • Navigate to about:logging
  • In the drop down for "Logging preset:" select WebRTC
  • Click the button above Set Log Modules
  • Click on Start Logging
  • Reproduce issue in a separate tab
  • Go back to the about:logging tab and click Stop Logging
  • A new profiler tab should launch
  • In the top right click Upload Local Profile
    • Select Include hidden threads
    • Click Upload
    • Paste url for uploaded profile here

If the media is no longer playing on Firefox it's possible we are expecting new payload type numbers not being honored from signaling. The packet capture along with signaling information or a profile should help point to what is happening and what/where the fix is.

Flags: needinfo?(light.moon5419)

Hey Dan,

we decided to entirely remove the call to setCodecPreferences on transceivers created in response to remote offers(turns out we no longer need it for the Chromium bug), this is why you cannot reproduce(sorry about that). This was what I ultimately established as the root cause for the freezing.

I have copies of the SDP exchanges both pre and post the fix that I can share. Also, from looking at about:webrtc there were incoming packets for the SSRC in questions but decodedFrames was stuck. If you think it'd be very useful I can go back to the version before and carry out the steps outlined or if it's enough I can share the SDPs.

Flags: needinfo?(light.moon5419)

I'd really like to see the payload types for the packets being sent and received and the SDP in order to establish what is happening. I do intend to fix bug 1909443 but am concerned this issue is unrelated. My hope is that we no longer signaled to receive a payload type like 96 but were still sent it causing those packets to be discarded.

Flags: needinfo?(dbaker) → needinfo?(light.moon5419)
Keywords: stalled

A needinfo is requested from the reporter, however, the reporter is inactive on Bugzilla. Given that the bug is still UNCONFIRMED, closing the bug as incomplete.

For more information, please visit BugBot documentation.

Status: UNCONFIRMED → RESOLVED
Closed: 9 months ago
Flags: needinfo?(light.moon5419)
Resolution: --- → INCOMPLETE

Since the bug is closed, the stalled keyword is now meaningless.
For more information, please visit BugBot documentation.

Keywords: stalled
You need to log in before you can comment on or make changes to this bug.