864114 - Establishing a handshake with a local pc with audio only and remote pc with video only fails to play the remote video stream

Status: RESOLVED FIXED

(Core :: WebRTC: Networking, defect, P4)

Description

[Jason Smith [:jsmith]]

Attachment: Test Case

STR:

1. Load the attached test case
2. Accept permissions for your camera and mic

Expected:

Audio should playback in one media element, video in the other.

Actual:

Audio will playback in one of the media elements, but the video stream will fail to play.

Comment 1

[Eric Rescorla (:ekr)]

Why is this tagged as "TURN". Does it succeed without TURN?

Comment 2

[Jason Smith [:jsmith]]

(In reply to Eric Rescorla (:ekr) from comment #1)
> Why is this tagged as "TURN". Does it succeed without TURN?

Nope, it doesn't succeed without TURN either. So apparently this is now a general PC bug. A bad one at that.

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Since the first side (offer) is audio-only, not even willing to receive video, it can't be made to have a video media line in the answer (answers cannot add m= lines, only offers).  So to have the answerer add video would require accepting it as audio-only, then re-offering from the original answerer with video added to the original offerer.

This (among many other reasons) is because the original offerer isn't prepared to receive video - he has no port opened with ICE, for example.  In generic SDP, the IP address to receive video might even be different than the one to receive audio.

To make this work, the offerer must offer to receive video.

Here are the SDP's from your test (print out with localoffer.sdp, not localoffer).  Note also that the answer uses a=recvonly - it's willing to receive audio, but cannot send it with the video-only mediastream attached.

[21:48:04.189] "local pc createOffer successful with: v=0
o=Mozilla-SIPUA-23.0a1 8237 0 IN IP4 0.0.0.0
s=SIP Call
t=0 0
a=ice-ufrag:c2cb0f3b
a=ice-pwd:44e7a00c5f315bd851155e2299ab82a3
a=fingerprint:sha-256 2E:F0:AE:50:AC:58:B2:CC:6B:39:11:AD:81:1F:4E:C5:E9:79:C1:EE:0B:52:7F:A7:16:79:13:90:B8:A8:1A:F0
m=audio 55337 RTP/SAVPF 109 0 8 101
c=IN IP4 98.111.140.34
a=rtpmap:109 opus/48000/2
a=ptime:20
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv
a=candidate:0 1 UDP 2113667327 192.168.1.17 55337 typ host
a=candidate:1 1 UDP 1694302207 98.111.140.34 55337 typ srflx raddr 192.168.1.17 rport 55337
a=candidate:2 1 UDP 2112422143 192.168.122.1 49310 typ host
a=candidate:3 1 UDP 1693057023 98.111.140.34 49310 typ srflx raddr 192.168.122.1 rport 49310
a=candidate:0 2 UDP 2113667326 192.168.1.17 56396 typ host
a=candidate:1 2 UDP 1694302206 98.111.140.34 56396 typ srflx raddr 192.168.1.17 rport 56396
a=candidate:2 2 UDP 2112422142 192.168.122.1 56136 typ host
a=candidate:3 2 UDP 1693057022 98.111.140.34 56136 typ srflx raddr 192.168.122.1 rport 56136

[21:48:04.285] "remote pc createAnswer successful with: v=0
o=Mozilla-SIPUA-23.0a1 26837 0 IN IP4 0.0.0.0
s=SIP Call
t=0 0
a=ice-ufrag:7577ba19
a=ice-pwd:71b00f05e6fdab7b258f8f117c6246ba
a=fingerprint:sha-256 82:D3:AA:0F:EB:C0:F5:F2:E3:61:F6:34:40:46:4D:2E:03:BA:94:6B:1E:23:F5:64:EC:36:06:AA:29:92:CE:26
m=audio 55738 RTP/SAVPF 109 101
c=IN IP4 98.111.140.34
a=rtpmap:109 opus/48000/2
a=ptime:20
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=recvonly
a=candidate:0 1 UDP 2113667327 192.168.1.17 55738 typ host
a=candidate:1 1 UDP 1694302207 98.111.140.34 55738 typ srflx raddr 192.168.1.17 rport 55738
a=candidate:2 1 UDP 2112422143 192.168.122.1 40720 typ host
a=candidate:3 1 UDP 1693057023 98.111.140.34 40720 typ srflx raddr 192.168.122.1 rport 40720
a=candidate:0 2 UDP 2113667326 192.168.1.17 44928 typ host
a=candidate:1 2 UDP 1694302206 98.111.140.34 44928 typ srflx raddr 192.168.1.17 rport 44928
a=candidate:2 2 UDP 2112422142 192.168.122.1 54025 typ host
a=candidate:3 2 UDP 1693057022 98.111.140.34 54025 typ srflx raddr 192.168.122.1 rport 54025

Comment 4

[Jason Smith [:jsmith]]

I don't agree this is invalid - that's in violation of the W3C spec for WebRTC.

http://dev.w3.org/2011/webrtc/editor/webrtc.html#constraints

The spec indicates that the default media constraints for OfferToReceiveVideo and OfferToReceiveAudio defaults to a *non-mandatory" true. This means that the expectation here is that in the audio in one peer case, video in the other peer case, the expectation is *optional*, not *required.* So this is a valid callback.

If I'm misinterpreting this anyways, we should be erroring out gracefully here if this was expected behavior. But I'm reading the spec differently here.

Comment 5

[Jason Smith [:jsmith]]

Running the same test case in Google Chrome, they have differing behavior here than us, so I'm not getting the broken stream in Chrome. The handshake ends up being successful, but onaddstream fails to fire on both the local and remote peer.

Comment 6

[Eric Rescorla (:ekr)]

(In reply to Jason Smith [:jsmith] from comment #4)
> I don't agree this is invalid - that's in violation of the W3C spec for
> WebRTC.
> 
> http://dev.w3.org/2011/webrtc/editor/webrtc.html#constraints
> 
> The spec indicates that the default media constraints for
> OfferToReceiveVideo and OfferToReceiveAudio defaults to a *non-mandatory"
> true. This means that the expectation here is that in the audio in one peer
> case, video in the other peer case, the expectation is *optional*, not
> *required.* So this is a valid callback.

Those are the default constraints if you offer to send media.

"This is an enum type constraint that can take the values "true" and "false". The default is a non mandatory "true" for an RTCPeerConnection object that has a video stream at the point in time when the constraints are being evaluated and is non mandatory "false" otherwise."

I don't understand what you are looking for in terms of callbacks. What seems to be happening here is that one side wanted to send video and the other side didn't want to receive it, so it's not being sent. That's not an error, it's normal. Arguably, the API should allow you to interrogate the negotiation results, but it doesn't seem to yet.

Comment 7

[Jason Smith [:jsmith]]

(In reply to Eric Rescorla (:ekr) from comment #6)
> (In reply to Jason Smith [:jsmith] from comment #4)
> > I don't agree this is invalid - that's in violation of the W3C spec for
> > WebRTC.
> > 
> > http://dev.w3.org/2011/webrtc/editor/webrtc.html#constraints
> > 
> > The spec indicates that the default media constraints for
> > OfferToReceiveVideo and OfferToReceiveAudio defaults to a *non-mandatory"
> > true. This means that the expectation here is that in the audio in one peer
> > case, video in the other peer case, the expectation is *optional*, not
> > *required.* So this is a valid callback.
> 
> Those are the default constraints if you offer to send media.
> 
> "This is an enum type constraint that can take the values "true" and
> "false". The default is a non mandatory "true" for an RTCPeerConnection
> object that has a video stream at the point in time when the constraints are
> being evaluated and is non mandatory "false" otherwise."

Ah, I see the misinterpretation then. Video has to be provided to get the optional case. But in the audio case, that's not true - so we establish a requirement in that case on one direction.

> 
> I don't understand what you are looking for in terms of callbacks. What
> seems to be happening here is that one side wanted to send video and the
> other side didn't want to receive it, so it's not being sent. That's not an
> error, it's normal. Arguably, the API should allow you to interrogate the
> negotiation results, but it doesn't seem to yet.

I'd argue we shouldn't ever end up in a case where you can get a broken stream from a remote peer. If the behavior we have right now is expected, then I think we should be following Chrome's behavior here - we don't fire onaddstream if we know the stream we are giving to a particular peer is "broken."

My guess is that Chrome made the design decision that we should never given a broken stream to a developer because it will create confusion and broken behavior. Although this might be a good question to ask someone from the Chrome team about...

Comment 8

[Eric Rescorla (:ekr)]

(In reply to Jason Smith [:jsmith] from comment #7)
> (In reply to Eric Rescorla (:ekr) from comment #6)
> > (In reply to Jason Smith [:jsmith] from comment #4)
> > > I don't agree this is invalid - that's in violation of the W3C spec for
> > > WebRTC.
> > > 
> > > http://dev.w3.org/2011/webrtc/editor/webrtc.html#constraints
> > > 
> > > The spec indicates that the default media constraints for
> > > OfferToReceiveVideo and OfferToReceiveAudio defaults to a *non-mandatory"
> > > true. This means that the expectation here is that in the audio in one peer
> > > case, video in the other peer case, the expectation is *optional*, not
> > > *required.* So this is a valid callback.
> > 
> > Those are the default constraints if you offer to send media.
> > 
> > "This is an enum type constraint that can take the values "true" and
> > "false". The default is a non mandatory "true" for an RTCPeerConnection
> > object that has a video stream at the point in time when the constraints are
> > being evaluated and is non mandatory "false" otherwise."
> 
> Ah, I see the misinterpretation then. Video has to be provided to get the
> optional case. But in the audio case, that's not true - so we establish a
> requirement in that case on one direction.
> 

No,there is an offerer/answerer asymmetry for the reasons that Jesup indicated.


> > I don't understand what you are looking for in terms of callbacks. What
> > seems to be happening here is that one side wanted to send video and the
> > other side didn't want to receive it, so it's not being sent. That's not an
> > error, it's normal. Arguably, the API should allow you to interrogate the
> > negotiation results, but it doesn't seem to yet.
> 
> I'd argue we shouldn't ever end up in a case where you can get a broken
> stream from a remote peer. If the behavior we have right now is expected,
> then I think we should be following Chrome's behavior here - we don't fire
> onaddstream if we know the stream we are giving to a particular peer is
> "broken."
>

Yes, onaddstream should never fire.


> My guess is that Chrome made the design decision that we should never given
> a broken stream to a developer because it will create confusion and broken
> behavior. Although this might be a good question to ask someone from the
> Chrome team about...

Comment 9

[Jason Smith [:jsmith]]

Sounds like the bug here is then that we're firing onaddstream when we shouldn't be in the case of the test case given here. Do we agree then that's the bug here?

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

It'd be interesting to know if this still happens.  Not high priority (nor is this bug).

Comment 11

[juan becerra [:juanb]]

The behavior described in comment #0 still happens in the latest Firefox Dev Edition and nightly. On Chrome I cannot reproduce the problem, because I am never prompted to share my camera/mic.

Comment 12

[Byron Campen [:bwc]]

This test case doesn't use offerToReceiveAudio, which it definitely needs to do according to the latest spec. Looking at the behavior now, the SDP contains the correct stuff (each end has one audio m-section). Further, we won't be firing onaddstream for the offerer here, since we have mochitests that verify that this doesn't happen if the remote end doesn't negotiate any m-sections with send.