Closed Bug 1347578 Opened 8 years ago Closed 4 years ago

remotestream.onremovetrack not fired when a remote track is removed during renegotiation

Categories

(Core :: WebRTC: Signaling, defect, P3)

Product:
Core
Component:
WebRTC: Signaling
Version:
54 Branch
Type:
defect

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: ibc, Unassigned)

Details

When a re-offer is received and such a re-offer removes a track (by setting the m= port to 0 and setting a=inactive) the MediaStream holding such a remote track does not get its 'removetrack' event fired. This means that such a remote MediaStream still contains the now "ended" track. If later a new remote track (a video track) is added (remotestream.onaddtrack is fired) the MediaStream will contain both, the old "ended" track and the new active one. Unfortunatelly this causes the video not being rendered in the HTMLVideoElement.
This is the first remote SDP offer, which contains a sending video track (a=mid:dwksbhsb): sdp:v=0 o=mediasoup 0527309242215156 1 IN IP4 0.0.0.0 s=- t=0 0 a=ice-lite a=fingerprint:sha-256 D4:5C:2D:27:E7:70:2A:CB:FD:C5:F7:7E:45:76:B0:A7:C3:15:86:80:2E:07:3E:EF:82:7E:18:82:50:D5:FB:08 a=msid-semantic: WMS * a=group:BUNDLE recv-audio-track-1 recv-video-track-1 dwksbhsb y860fc5i m=audio 7 RTP/SAVPF 100 c=IN IP4 127.0.0.1 a=rtpmap:100 opus/48000/2 a=fmtp:100 maxplaybackrate=48000;stereo=1;useinbandfec=1 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=setup:actpass a=mid:recv-audio-track-1 a=recvonly a=ice-ufrag:t1p0mtsij1z54t2v a=ice-pwd:6mmpj65j4yky1mf4zbvu4buw9fhrefg5 a=candidate:udpcandidate 1 udp 1078862079 192.168.1.33 44909 typ host a=end-of-candidates a=rtcp-mux a=rtcp-rsize m=video 7 RTP/SAVPF 123 c=IN IP4 127.0.0.1 a=rtpmap:123 VP8/90000 a=fmtp:123 max-fr=60;max-fs=12288 a=rtcp-fb:123 nack a=rtcp-fb:123 nack pli a=rtcp-fb:123 ccm fir a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=setup:actpass a=mid:recv-video-track-1 a=recvonly a=ice-ufrag:t1p0mtsij1z54t2v a=ice-pwd:6mmpj65j4yky1mf4zbvu4buw9fhrefg5 a=candidate:udpcandidate 1 udp 1078862079 192.168.1.33 44909 typ host a=end-of-candidates a=rtcp-mux a=rtcp-rsize m=video 7 RTP/SAVPF 123 c=IN IP4 127.0.0.1 a=rtpmap:123 VP8/90000 a=rtcp-fb:123 ccm fir a=rtcp-fb:123 nack a=rtcp-fb:123 nack pli a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=setup:actpass a=mid:dwksbhsb a=msid:xbxruqoyHYWRbi6titHZEETmudIfH3pLt8Jk 1f3195e7-3c1b-40ec-8da3-6103c52b20b8 a=sendonly a=ice-ufrag:t1p0mtsij1z54t2v a=ice-pwd:6mmpj65j4yky1mf4zbvu4buw9fhrefg5 a=candidate:udpcandidate 1 udp 1078862079 192.168.1.33 44909 typ host a=end-of-candidates a=ssrc:3340627738 cname:HSBPQEZ5uwcw52b6 a=rtcp-mux a=rtcp-rsize m=audio 7 RTP/SAVPF 100 c=IN IP4 127.0.0.1 a=rtpmap:100 opus/48000/2 a=fmtp:100 minptime=10;useinbandfec=1 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=setup:actpass a=mid:y860fc5i a=msid:xbxruqoyHYWRbi6titHZEETmudIfH3pLt8Jk 8531fb79-288f-4e7d-b6ce-2c3308ba38e8 a=sendonly a=ice-ufrag:t1p0mtsij1z54t2v a=ice-pwd:6mmpj65j4yky1mf4zbvu4buw9fhrefg5 a=candidate:udpcandidate 1 udp 1078862079 192.168.1.33 44909 typ host a=end-of-candidates a=ssrc:1599705591 cname:HSBPQEZ5uwcw52b6 a=rtcp-mux a=rtcp-rsize It properly triggers the pc.addtrack event for the a=mid:dwksbhsb video track. Later, a re-offer removing such a video track is received: sdp:v=0 o=mediasoup 0527309242215156 1 IN IP4 0.0.0.0 s=- t=0 0 a=ice-lite a=fingerprint:sha-256 D4:5C:2D:27:E7:70:2A:CB:FD:C5:F7:7E:45:76:B0:A7:C3:15:86:80:2E:07:3E:EF:82:7E:18:82:50:D5:FB:08 a=msid-semantic: WMS * a=group:BUNDLE recv-audio-track-1 recv-video-track-1 dwksbhsb y860fc5i m=audio 7 RTP/SAVPF 100 c=IN IP4 127.0.0.1 a=rtpmap:100 opus/48000/2 a=fmtp:100 maxplaybackrate=48000;stereo=1;useinbandfec=1 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=setup:actpass a=mid:recv-audio-track-1 a=recvonly a=ice-ufrag:t1p0mtsij1z54t2v a=ice-pwd:6mmpj65j4yky1mf4zbvu4buw9fhrefg5 a=candidate:udpcandidate 1 udp 1078862079 192.168.1.33 44909 typ host a=end-of-candidates a=rtcp-mux a=rtcp-rsize m=video 7 RTP/SAVPF 123 c=IN IP4 127.0.0.1 a=rtpmap:123 VP8/90000 a=fmtp:123 max-fr=60;max-fs=12288 a=rtcp-fb:123 nack a=rtcp-fb:123 nack pli a=rtcp-fb:123 ccm fir a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=setup:actpass a=mid:recv-video-track-1 a=recvonly a=ice-ufrag:t1p0mtsij1z54t2v a=ice-pwd:6mmpj65j4yky1mf4zbvu4buw9fhrefg5 a=candidate:udpcandidate 1 udp 1078862079 192.168.1.33 44909 typ host a=end-of-candidates a=rtcp-mux a=rtcp-rsize m=video 0 RTP/SAVPF 123 c=IN IP4 127.0.0.1 a=rtpmap:123 VP8/90000 a=rtcp-fb:123 ccm fir a=rtcp-fb:123 nack a=rtcp-fb:123 nack pli a=mid:dwksbhsb a=inactive m=audio 7 RTP/SAVPF 100 c=IN IP4 127.0.0.1 a=rtpmap:100 opus/48000/2 a=fmtp:100 minptime=10;useinbandfec=1 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=setup:actpass a=mid:y860fc5i a=msid:xbxruqoyHYWRbi6titHZEETmudIfH3pLt8Jk 8531fb79-288f-4e7d-b6ce-2c3308ba38e8 a=sendonly a=ice-ufrag:t1p0mtsij1z54t2v a=ice-pwd:6mmpj65j4yky1mf4zbvu4buw9fhrefg5 a=candidate:udpcandidate 1 udp 1078862079 192.168.1.33 44909 typ host a=end-of-candidates a=ssrc:1599705591 cname:HSBPQEZ5uwcw52b6 a=rtcp-mux a=rtcp-rsize Here the a=mid:dwksbhsb video section is clearly "closed" (port 0 and a=inactive). In fact, such a remote MediaStreamTrack has now state "ended". However the 'removetrack' event was not triggered by its remote MediaStream so it remains when calling remotestream.getVideoTracks() (now with "ended" status).
Interesting. After the renegotiation above, pc.getReceivers() just retrieves the audio RtpReceiver, cool. However, pc.getRemoteStreams()[0].getTracks() still retrieves both the active audio track and the ended video track.
Not sure what is going on there. Can you try running this scenario on a debug build? Maybe it will hit an assertion.
Flags: needinfo?(ibc)
We don't implement the "removetrack" event. You could use the "ended" event on the track instead. Could this be because the MediaStream you are looking at becomes inactive? A media element will end when that happens. This landed in 52, see bug 1208316.
The MediaStream does not (should not) become inactive since its audio track is always there and live. I'll check the track "ended" event. > Can you try running this scenario on a debug build? I'm sorry but I'n not used to build Firefox and, unfortunately, I don't have the time to do it.
No, track "ended" event is not fired neither. Note that, as explained above, the remote stream still contains the ended track (which in fact now has readyStatus:ended), but no event was fired. Also, if ended I understand that it should no longer be into the remote stream (but it does).
The (remote) track "ended" event DOES fire once the PeerConnection fires the (deprecated) "removestream" event. As a side note, it took over 15 seconds (or even more) before the remotestream.active boolean became false (I mean after the PC "removestream" event)...
> As a side note, it took over 15 seconds (or even more) before the remotestream.active boolean became false (I mean after the PC "removestream" event)... In a second attempt it was instant, so forget about that.
(In reply to Iñaki Baz Castillo from comment #5) > The MediaStream does not (should not) become inactive since its audio track > is always there and live. I'll check the track "ended" event. > > > Can you try running this scenario on a debug build? > > I'm sorry but I'n not used to build Firefox and, unfortunately, I don't have > the time to do it. Debug builds are available in automation. What platform/OS would you want to test on; I can give you a link
OSX Sierra 10.12.3 Thanks!
Try this.... https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-macosx64-debug/1489765160/firefox-55.0a1.en-US.mac.dmg Future reference: At https://treeherder.mozilla.org/#/jobs?repo=mozilla-central find OS X 10.7 debug (it builds for 10.7 and 10.10), click on the (first) green 'B', then in the lower-left corner click on the build: link. Then download the .dmg and install.
Thanks a lot Randell! Will check it on next week and report here if I see something related to the issue.
Well, just tried the scenario above right now with NightlyDebug 55.01. Nothing happens in the debug output when the remote MediaStreamTrack is ended (I mean, the remote track "ended" event is not fired, however it has readyStatys:ended). No assertion error at all in the debug output.
Flags: needinfo?(ibc)
Well, completely unrelated (I think) but I've got some crashes with NightlyDebug and a 100% reproducible assertion error: - Open https://demo.mediasoup.org with NightlyDebug 55.XX. - Once connected, remove local video track by clicking into the "webcam" icon in the local video. (this removes the local video track from the PeerConnection, gets a reoffer from the server and answers according). - Then re-add video. (this calls getUserMedia({video:true}), attaches the video track to the PeerConnection, gets a reoffer from the server and answer according). This happens in the debug output: [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 [Child 60010] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/m-cen-m64-d-000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 3064 etc etc
Rank: 25
Priority: -- → P2
Mass change P2->P3 to align with new Mozilla triage process.
Priority: P2 → P3

Overtaken by events. Remote tracks are no longer "ended" but are "muted" and removed from their streams.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.