<a class="header-button" href="https://bugzilla.mozilla.org/home" title="Go to home page"> Bugzilla

Reporter

Comment 2

•

5 years ago

Component may be WebRTC: Audio/video, but I'm not sure firefox internals ;)

Comment 3

•

5 years ago

Thank you for your bug report!

Your SDP looks fine to me, but I'm not an expert.

I've tested Firefox Nightly against meet.jit.si which uses simulcast and I'm seeing three streams as expected. It would be very helpful if you could capture some logs. This is easiest on Linux, if you set MOZ_LOG="signaling:5" in the environment and then run firefox, redirecting stderr to a log file. You may have to disable the sandbox for this work, in about:config, set security.sandbox.content.level to 0.

On jitsi, I'm seeing lines like the following, showing the three simulcast streams being set up:

[Child 235315: Unnamed thread 0x7fd591959940]: I/signaling [|WebrtcVideoSessionConduit] VideoStreamFactory.cpp:196: CreateEncoderStreams Input frame 1280x720, RID 3 scaling to 320x180
[Child 235315: Unnamed thread 0x7fd591959940]: I/signaling [|WebrtcVideoSessionConduit] VideoStreamFactory.cpp:196: CreateEncoderStreams Input frame 1280x720, RID 2 scaling to 640x360
[Child 235315: Unnamed thread 0x7fd591959940]: I/signaling [|WebrtcVideoSessionConduit] VideoStreamFactory.cpp:196: CreateEncoderStreams Input frame 1280x720, RID 1 scaling to 1280x720

I'd be curious if you are seeing something similar. Attaching the logs here would help us diagnose the problem.

Thanks!

Flags: needinfo?(shino.shun+bugzilla.mozilla)

Reporter

Comment 4

•

5 years ago

Thank you for your reply and advice for digging further.

Some lines around "scaling" logs:

[Child 68374: Main Thread]: I/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:324: PeerConnectionImpl: PeerConnectionImpl constructor for 
[Child 68374: Main Thread]: D/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:232: Created PeerConnection: 0x7ff64ab30800
[Child 68374: Main Thread]: D/signaling [main|MediaTransportHandler] MediaTransportHandler.cpp:197: MediaTransportHandlerSTS done
[Child 68374: Main Thread]: I/signaling [main|sdp_config] sdp_config.c:86: SDP: Initialized config pointer: 0x7ff64f7d53c0
[Child 68374: Main Thread]: E/signaling [main|SDP Parse] sdp_main.c:1363: SDP Parse Error  Warning: Unrecognized attribute (rtcp-rsize) , line 16
[Child 68374: Main Thread]: D/signaling [main|sdp_attr] sdp_attr.c:1381: sdp_parse_attr_fmtp Unknown fmtp type (x-google-start-bitrate) - ignoring
[Child 68374: Main Thread]: E/signaling [main|SDP Parse] sdp_main.c:1363: SDP Parse Error  Warning: Invalid named events specified for fmtp attribute., line 29
[Child 68374: Main Thread]: V/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:458: Create
[Child 68374: Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1375: Init this=0x7ff64ab14800
[Child 68374: Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1382: Init Initialization Done
[Child 68374: Main Thread]: V/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:469: Create Successfully created VideoConduit 
[Child 68374: Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1447: SetReceiverTransport 
[Child 68374: Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1401: AttachRenderer
[Child 68374: Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1435: SetTransmitterTransport 
[Child 68374: Main Thread]: I/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:1468: SetRemoteDescription: pc = 54c8d9da36bdc654, asking JS to create transceiver
[Child 68374: Main Thread]: D/signaling [main|MediaTransportHandler] MediaTransportHandler.cpp:407: operator() starting
[Child 68374: Socket Thread]: D/signaling [Socket Thread|MediaTransportHandler] MediaTransportHandler.cpp:495: operator() done
[Child 68374: Main Thread]: I/signaling [main|PeerConnectionMedia] PeerConnectionMedia.cpp:73: OnStunAddrsAvailable: receiving (6) stun addrs
[Child 68374: Main Thread]: D/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:1264: CreateAnswer()
[Child 68374: Main Thread]: I/signaling [main|sdp_config] sdp_config.c:86: SDP: Initialized config pointer: 0x7ff65b684660
[Child 68374: Main Thread]: D/signaling [main|PeerConnectionMedia] PeerConnectionMedia.cpp:218: ACTIVATING TRANSPORT! - PC 54c8d9da36bdc654: level=0 components=1
[Child 68374: Socket Thread]: D/signaling [Socket Thread|MediaTransportHandler] MediaTransportHandler.cpp:547: PC:1599708463779418 (id=6442450947 url=http://127.0.0.1:3333/simulcast_sendonly?audio=false): Creating ICE media stream=transport_0 components=1
[Child 68374: Main Thread]: V/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:593: ConfigureCodecMode 
[Child 68374: Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:841: ConfigureSendMediaCodec for VP8
[Child 68374: Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:856: ConfigureSendMediaCodec for VideoConduit:0x7ff64ab14800 stream count:3
[Child 68374: Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:2177: StartTransmittingLocked Attemping to start... 
[Child 68374: Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1898: OnSinkWantsChanged (send SSRC 2148881470 (0x8015543e)) - wants pixels = 2147483647
[Child 68374: Main Thread]: D/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:1052: InitializeDataChannel
[Child 68374: Main Thread]: D/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:1066: InitializeDataChannel: We did not negotiate datachannel
[Child 68374: Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1898: OnSinkWantsChanged (send SSRC 2148881470 (0x8015543e)) - wants pixels = 2147483647
[Child 68374: Main Thread]: D/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2522: IceGatheringStateChange 1
[Child 68374: Socket Thread]: D/signaling [Socket Thread|MediaTransportHandler] MediaTransportHandler.cpp:599: PC:1599708463779418 (id=6442450947 url=http://127.0.0.1:3333/simulcast_sendonly?audio=false): Activating ICE media stream=transport_0 components=1
[Child 68374: Socket Thread]: D/signaling [Socket Thread|PeerConnectionMedia] PeerConnectionMedia.cpp:701: OnCandidateFound_s: transport_0
[Child 68374: Main Thread]: D/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2553: UpdateDefaultCandidate
[Child 68374: Main Thread]: D/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2438: Passing local candidate to content: candidate:0 1 UDP 92086271 203.0.113.1 11490 typ relay raddr 203.0.113.1 rport 11490
[Child 68374: Main Thread]: D/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2455: IceConnectionStateChange: 1
[Child 68374: Main Thread]: D/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2455: IceConnectionStateChange: 2
[Child 68374: WebRTCPD #1]: V/signaling [WebRTCPD #1|WebrtcVideoSessionConduit] VideoConduit.cpp:1923: WebrtcVideoConduit 0x7ff64ab14800 SendVideoFrame (send SSRC 2148881470 (0x8015543e))
[Child 68374: WebRTCPD #1]: V/signaling [WebRTCPD #1|WebrtcVideoSessionConduit] VideoConduit.cpp:1929: SendVideoFrame: call SelectSendResolution with 1280x720
[Child 68374: Unnamed thread 0x7ff64f80d940]: I/signaling [|WebrtcVideoSessionConduit] VideoStreamFactory.cpp:196: CreateEncoderStreams Input frame 320x176, RID high scaling to 320x176
[Child 68374: Unnamed thread 0x7ff64f80d940]: I/signaling [|WebrtcVideoSessionConduit] VideoStreamFactory.cpp:196: CreateEncoderStreams Input frame 320x176, RID middle scaling to 320x176
[Child 68374: Unnamed thread 0x7ff64f80d940]: I/signaling [|WebrtcVideoSessionConduit] VideoStreamFactory.cpp:196: CreateEncoderStreams Input frame 320x176, RID low scaling to 320x176
[Child 68374: WebRTCPD #1]: V/signaling [WebRTCPD #1|WebrtcVideoSessionConduit] VideoConduit.cpp:1923: WebrtcVideoConduit 0x7ff64ab14800 SendVideoFrame (send SSRC 2148881470 (0x8015543e))

There seems two error lines, I will dig them on my side.

Flags: needinfo?(shino.shun+bugzilla.mozilla)

Reporter

Comment 5

•

5 years ago

I've tested Firefox Nightly against meet.jit.si which uses simulcast and I'm seeing three streams as expected

Could you please give me some information about this??

Is it server-offered or client-offered?
If server-offered, what is the orders of WebRTC API calls?
- The order now I'm trying is: setRemoteDescription -> addTrack -> transceiver.sender.setParameters -> createAnswer
- background of this question is that Chrome had a bug that addTrack should be called after setRemoteDescription.

Thanks!!

Comment 6

•

5 years ago

So what is strange to me is that your input stream is already 320x176 prior to scaling, which is probably why it is not being scaled even smaller. In my example, the input stream was 1280x720 and then was scaled. In the line just above it is [Child 68374: WebRTCPD #1]: V/signaling [WebRTCPD #1|WebrtcVideoSessionConduit] VideoConduit.cpp:1929: SendVideoFrame: call SelectSendResolution with 1280x720 so I'm not sure what is going on there. I'll have a closer look at the code and see if I can determine what is happening.

I'm sorry, I can't really say much about how meet.jit.si is configuring simulcast, that's outside my area of expertise. I just wanted to rule out simulcast being completely broken, which has happened in the past due to the difficulty of testing it inside Firefox.

Updated

•

5 years ago

Severity: -- → S3

Status: UNCONFIRMED → NEW

Ever confirmed: true

Priority: -- → P3

Comment 7

•

5 years ago

I'm stumped. Andreas, do you have any ideas on this?

Flags: needinfo?(apehrson)

BugBot [:suhaib / :marco/ :calixte]

Assignee

Comment 8

•

5 years ago

I had a hunch that the order of the encodings is important when calling setParameters. I think that's it.

We have assumptions that the first encoding is the full-resolution one. Because legacy. Our implementation of setParameters originates from before the spec had settled on it. Then we added support for scaleResolutionDownBy and improved the support for maxBitrate, but we didn't bring the API to spec. Bug 1401592 tracks that work.

When you are calling setParameters with the order low, medium, high you're essentially hitting undefined behavior. When I tested this locally by changing the order in our unittests I encountered the following:

Setting simulcast on the answer: Failed this assert. This is a debug build thing, in non-debug I expect to reproduce the reported symptoms.
Setting simulcast on the offer: No streams sent at all -- unclear what's up there.

I think the easiest way past this is to make sure that setParameters is called with the highest-resolution stream first. I don't think the rest have to be high-to-low.

A mitigation on our end could be to sort out the highest resolution stream and put it first before handing it to our simulcast scaling logic.

Long term we'll fix bug 1401592 but that could take a while.

Component: WebRTC: Signaling → WebRTC: Audio/Video

Depends on: 1401592

Flags: needinfo?(apehrson)

Summary: VP8 Simulcast streams does not respect RTCRtpEncodingParameters.scaleResolutionDownBy → Simulcast streams must be configured with highest resolution stream first

Comment 9

•

5 years ago

The component has been changed since the backlog priority was decided, so we're resetting it.
For more information, please visit auto_nag documentation.

Priority: P3 → --

Assignee

Updated

•

5 years ago

Priority: -- → P3

Version: 80 Branch → unspecified

sjw

Updated

•

5 years ago

Whiteboard: [jitsi-meet]

Comment 10

•

5 years ago

Hi, why did you set the [jitsi-meet] whiteboard here? From my testing, jitsi is not affected by this bug. Has something changed?

Flags: needinfo?(sjw+bugzilla)

sjw

Comment 11

•

5 years ago

This has been reported in https://github.com/jitsi/jitsi-meet/issues/4758

Flags: needinfo?(sjw+bugzilla)

Assignee

Comment 12

•

5 years ago

Testing beta.meet.jit.si I see in the sdp a=simulcast:send 1;2;3, and in the log:

[Child 1577738: WebRTCPD #1]: V/signaling [WebRTCPD #1|WebrtcVideoSessionConduit] VideoConduit.cpp:1928: SendVideoFrame: call SelectSendResolution with 1280x720
[Child 1577738: Unnamed thread 0x7f88831d7ee0]: I/signaling [|WebrtcVideoSessionConduit] VideoStreamFactory.cpp:196: CreateEncoderStreams Input frame 1280x720, RID 3 scaling to 320x180
[Child 1577738: Unnamed thread 0x7f88831d7ee0]: I/signaling [|WebrtcVideoSessionConduit] VideoStreamFactory.cpp:196: CreateEncoderStreams Input frame 1280x720, RID 2 scaling to 640x360
[Child 1577738: Unnamed thread 0x7f88831d7ee0]: I/signaling [|WebrtcVideoSessionConduit] VideoStreamFactory.cpp:196: CreateEncoderStreams Input frame 1280x720, RID 1 scaling to 1280x720

In conclusion:

jitsi has the stream with the highest resolution first, and therefore
this is not a jitsi issue

Feel free to re-add if you have evidence on the contrary.

Whiteboard: [jitsi-meet]

Óvári

Comment 13

•

5 years ago

Screen sharing doesn't work bug 1663286 if the presenter and all the viewer are in audio-only mode. Screen sharing worked in Firefox in version 79 and before. Does simulcast stop broadcasting the screen share?

A presentation only shows in the presenters Firefox; however, the screen share does not show on any of the viewers browsers or Jitsi Apps.

Hence, should the whiteboard include jitsi-meet?

Thank you

Nils Ohlmeier [:drno]

Comment 14

•

5 years ago

(In reply to Óvári from comment #13)

Screen sharing doesn't work bug 1663286 if the presenter and all the viewer are in audio-only mode. Screen sharing worked in Firefox in version 79 and before. Does simulcast stop broadcasting the screen share?

I suggest you ask that question in the bug report you referenced your self. Let's keep this bug focused on the problem initially reported, which is not related to Jitsi.

Reporter

Comment 15

•

5 years ago

(In reply to Andreas Pehrson [:pehrsons] from comment #8)

I had a hunch that the order of the encodings is important when calling setParameters. I think that's it.

Thanks a lot for hunting!

We have assumptions that the first encoding is the full-resolution one.

Understood.

I think the easiest way past this is to make sure that setParameters is called with the highest-resolution stream first. I don't think the rest have to be high-to-low.

This will be a nice workaround at the moment by application.
A concern about it is the order of stream truncation because there is a spec:

https://github.com/w3c/webrtc-pc/issues/2537
https://github.com/w3c/webrtc-pc/pull/2554
According to this, with high/middle/low ordering, "low" will be truncated first in low-bandwidth case and "high" remains to the last.
My expectation is "low" remains as far as possible.

Long term we'll fix bug 1401592 but that could take a while.

Hope the fix will land soon!!

Thanks again for the great effort to develop/maintain the great browser!

Assignee

Comment 16

•

5 years ago

(In reply to shino.shun+bugzilla.mozilla from comment #15)

This will be a nice workaround at the moment by application.
A concern about it is the order of stream truncation because there is a spec:

https://github.com/w3c/webrtc-pc/issues/2537

https://github.com/w3c/webrtc-pc/pull/2554
According to this, with high/middle/low ordering, "low" will be truncated first in low-bandwidth case and "high" remains to the last.
My expectation is "low" remains as far as possible.

I see. I will try to do a quick fix such that the order is not important, and bug 1401592 will take this to spec properly further down the line.

Assignee: nobody → apehrson

Status: NEW → ASSIGNED

Priority: P3 → P2

rugk

Comment 17

•

5 years ago

This here is required for Jitsi Meet to work properly.

As such, as you had some whiteboard for [jitsi-meet], please assign that to this.
More information see: https://github.com/jitsi/jitsi-meet/issues/4758#issuecomment-692602039

Assignee

Comment 18

•

5 years ago

(In reply to rugk from comment #17)

This here is required for Jitsi Meet to work properly.

As such, as you had some whiteboard for [jitsi-meet], please assign that to this.
More information see: https://github.com/jitsi/jitsi-meet/issues/4758#issuecomment-692602039

No. See comments 10-14. Thank you and once again let's keep this bug focused on what has been reported here.

Assignee

Comment 19

•

5 years ago

Attached file Bug 1663368 - Add mochitests for simulcast with custom layer-resolution-order. r?dminor — Details

Assignee

Comment 20

•

5 years ago

Attached file Bug 1663368 - Don't assume a certain stream order in Mozilla code. r?dminor — Details

Assignee

Comment 21

•

5 years ago

Attached file Bug 1663368 - Cherry-pick and retrofit webrtc.org bug 10069 (REVERT this before applying next webrtc.org update). r?dminor — Details

See https://crbug.com/webrtc/10069 for details and full diffs.

Assignee

Comment 22

•

5 years ago

Attached file Bug 1663368 - Update VideoConduit to use the new APIs. r?dminor — Details

https://hg.mozilla.org/mozilla-central/rev/955a6b2a9b8e
https://hg.mozilla.org/mozilla-central/rev/7433e1757d9b
https://hg.mozilla.org/mozilla-central/rev/2776093cc8c8
https://hg.mozilla.org/mozilla-central/rev/0d95f7108862
https://hg.mozilla.org/mozilla-central/rev/9f9774a08103

Assignee

Comment 23

•

5 years ago

Attached file Bug 1663368 - Update gn-configs and re-generate moz.build files. r?dminor — Details

Pulsebot

Comment 24

•

5 years ago

Pushed by pehrsons@gmail.com: https://hg.mozilla.org/integration/autoland/rev/955a6b2a9b8e Add mochitests for simulcast with custom layer-resolution-order. r=dminor https://hg.mozilla.org/integration/autoland/rev/7433e1757d9b Don't assume a certain stream order in Mozilla code. r=dminor https://hg.mozilla.org/integration/autoland/rev/2776093cc8c8 Cherry-pick and retrofit webrtc.org bug 10069 (REVERT this before applying next webrtc.org update). r=dminor https://hg.mozilla.org/integration/autoland/rev/0d95f7108862 Update gn-configs and re-generate moz.build files. r=dminor https://hg.mozilla.org/integration/autoland/rev/9f9774a08103 Update VideoConduit to use the new APIs. r=dminor

Atila Butkovits

Comment 25

•

5 years ago

bugherder

Status: ASSIGNED → RESOLVED

Closed: 5 years ago

status-firefox83: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → 83 Branch

Reporter

Comment 26

•

5 years ago

Thanks a lot for the fix!!

Please let me ask two questions:

Does the target milestone 83 mean this fix will ship in the version 83?
Can I test the fix with Firefox Nightly? If so, how can I find the fix commits are included in which nightly?

[1] https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/firefox-83.0a1.en-US.win64.txt
[2] https://hg.mozilla.org/mozilla-central/pushloghtml

Comment 27

•

5 years ago

Yes, this will ship in Firefox 83.

If you visit the ftp site for Nightly you can take a look at the latest firefox-83.0a1.en-US.win64.txt file [1] which will include the commit used to build Nightly. You can then check the pushlog [2] and see if the commits you care about have been pushed prior to the commit used to build Nightly. I checked, and Andreas' patches are part of the current Nightly. There's probably an easier way to determine this that I'm not aware of.

Please let us know if this fixes things for you.

Assignee

Comment 28

•

5 years ago

hg.m.o will actually include links to the first release with a patch. See the last patch that landed here for instance: https://hg.mozilla.org/mozilla-central/rev/9f9774a08103

Through that you can click your way to the files of the first nightly release with the patches for the platform you have, and download target.zip or an installer from there.

Reporter

Comment 29

•

5 years ago

I tried nightly and confirmed that each three streams had expected resolution.
Thank you both for fixing the issue and giving information, Andreas and Dan!!