1338521 - Video fails to appear (or freezes?) after refreshing page during a Jitsi call

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P1)

Description

[Iulia Cristescu, QA [:JuliaC] (away, please needinfo? [e-mail])]

[Affected versions]:
- latest Nightly 54.0a1 (2017-02-09)

[Affected platforms]:
- Windows 7 x64
- Mac OS X 10.11.6
- Mac OS X 10.12.1 Beta

[Steps to reproduce]:
1. Launch Firefox 
2. Go to https://meet.jit.si/, create a room, provide the camera and microphone permissions and join it
3. Join the same room from another station
4. Unplug the headset/camera
5. Connect back the headset/camera
6. Refresh/restart the page from the host side in order to reestablish the call
  - Inspect the host side video stream
7. Refresh the page from the guest side
  - Inspect the guest side video stream

[Expected result]:
- The call is successfully reestablished in both host and guest sides
- No audio/video issues are encountered

[Actual result]:
- [step 6] No video is streamed for the host side
- [step 7] No video is streamed for the guest side

[Regression range]:
- I will investigate this as soon as possible

Comment 1

[Andreas Pehrson [:pehrsons] (back in Aug 2021)]

This should definitely work after a page reload.

We also need to narrow this down, but my guess is either capture code or MSG.

Iulia, after this happens - can you see the local video on either side or is that broken too?

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

Also, can you make a run using non-e10s, and with MOZ_LOG=cubeb:5,MediaStreamGraph:4,GetUserMedia:5,MediaManager:5 and upload the logs?  thanks

Comment 3

[Ryan VanderMeulen [:RyanVM]]

We think this might have been fixed by some other work. Can we please try re-testing this? If it does still reproduce, the logging from comment 2 would still be helpful to get :)

Comment 4

[Iulia Cristescu, QA [:JuliaC] (away, please needinfo? [e-mail])]

Sorry for the delay. I didn't manage to get the necessary time for properly investigating this till today. 
I tested the issue  on 54.0a2 (2017-03-15) through a call between Windows 7 x64 and macOS 10.12.3, with e10s off and with MOZ_LOG=cubeb:5,MediaStreamGraph:4,GetUserMedia:5,MediaManager:5. Here are the logs https://drive.google.com/file/d/0B0nYKG6PRiCcWlRQb29wd1Rpekk/view?usp=sharing. 
The issue is still reproducible, but steps 4 and 5 from comment 0 are not useful anymore. 
The same behavior is met on 55.0a1 (2017-03-15), e10s on/off. 
Please let me know if further investigation is needed.

Comment 5

[Paul Adenot (:padenot)]

Looks like the cubeb side of things have been fixed, per comment 4 ("steps 4 and 5 from comment 0 are not useful anymore"). Let me know if I can be of any help here.

Comment 6

[Nathan Froyd [:froydnj]]

jesup, can you take a look at the logs from comment 4, or find somebody else who can?  Thanks!

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Nothing sticks out in the logs, though I could be missing something. They're 48MB!

★ ~/Downloads/Logs 2 $ grep llocated *
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Audio device 1 allocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Video device 0 allocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Audio device 1 deallocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Video device 0 deallocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Audio device 1 allocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Video device 1 allocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Audio device 1 deallocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Video device 1 deallocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Audio device 1 allocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Video device 2 allocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Audio device 1 deallocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Video device 2 deallocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Audio device -1 allocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Video device 3 allocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Audio device -1 deallocated
Window7 log.txt:[Unnamed thread 0000000025B53C00]: D/MediaManager Video device 3 deallocated
macOS log.txt:[Unnamed thread 0x13595f680]: D/MediaManager Audio device -1 allocated
macOS log.txt:[Unnamed thread 0x13595f680]: D/MediaManager Video device 0 allocated
macOS log.txt:[Unnamed thread 0x13595f680]: D/MediaManager Audio device -1 deallocated
macOS log.txt:[Unnamed thread 0x13595f680]: D/MediaManager Video device 0 deallocated
macOS log.txt:[Unnamed thread 0x13595f680]: D/MediaManager Audio device -1 allocated
macOS log.txt:[Unnamed thread 0x13595f680]: D/MediaManager Video device 1 allocated
macOS log.txt:[Unnamed thread 0x13595f680]: D/MediaManager Audio device -1 deallocated
macOS log.txt:[Unnamed thread 0x13595f680]: D/MediaManager Video device 1 deallocated
★ ~/Downloads/Logs 2 $ 

Not sure why -1 is an ok audio device, but otherwise, it does look like we're switching to a new video device, here (two cameras, or pull/replug?). Padenot, do you see anything?

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I'm able to reproduce it in a call between 54 on my mac and 54 on my win10 box, and also in a call between two tabs in 54 on the same mac. I used different cameras on each end (I have two on the mac) to avoid confusing myself about what I'm looking at.

I see remote video freezing on one end at a time, while local camera appears to be working ok (the jitsi self-view is deceiving: On the end suspected of sending frozen video, it appears black until I click on "Fellow Jitster" next to it, and then suddenly local video appears in the self-view without problem. That looks like a site bug.

It seems intermittent. I was able to freeze either end, just refreshing the page, even with an initial call (not restarting browser). I was not able to freeze both ends at the same time, though I may have given up too soon.

So this would point to a peer connection problem rather than a gum problem (or possibly even a jitsi JS problem). Drno, any tips on what logs would be helpful?

Comment 9

[Nils Ohlmeier [:drno]]

I'm not entirely convinced what a/the PeerConnection issue here would be. I assume you actually see ICE connecting on about:webrtc. As we don't have any easily accessible stats about RTCP complaining about the received video I would start with the new RTP logger (https://blog.mozilla.org/webrtc/debugging-encrypted-rtp-is-more-fun-than-it-used-to-be/) and check in Wireshark if the Firefox which renders only black sends out NACK's and/or PLI's.

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Iulia Cristescu, QA [:IuliaC] from comment #0)
> [Actual result]:
> - [step 6] No video is streamed for the host side
> - [step 7] No video is streamed for the guest side

Hi Iulia, when this happens does remote video appear frozen or does remote video not appear at all?
Also, do both self-views still work when this happens?

And lastly, any luck with the regression range?

Comment 11

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

The link to the other bug is just fyi. I think jitsi actually avoids it by asking for camera and mic separately.

Comment 12

[Iulia Cristescu, QA [:JuliaC] (away, please needinfo? [e-mail])]

(In reply to Jan-Ivar Bruaroey [:jib] from comment #10)
> (In reply to Iulia Cristescu, QA [:IuliaC] from comment #0)
> > [Actual result]:
> > - [step 6] No video is streamed for the host side
> > - [step 7] No video is streamed for the guest side
> 
> Hi Iulia, when this happens does remote video appear frozen or does remote
> video not appear at all?
> Also, do both self-views still work when this happens?
> 
> And lastly, any luck with the regression range?

Hello, Jan! 
Tested again using 54.0a2 (2017-04-07). When the issue occurs, the remote video does not appear at all and the self-views still work. 
Sorry, but I didn't have the necessary time for tracking the regression range yet.

Comment 13

[Peter Chang[:pchang]]

Julia, are you available to check the regression range?

Randell, are you able to reproduce this issue?

Comment 14

[Iulia Cristescu, QA [:JuliaC] (away, please needinfo? [e-mail])]

(In reply to Peter Chang[:pchang] from comment #13)
> Julia, are you available to check the regression range?
> 
> Randell, are you able to reproduce this issue?

Hello, Peter! This is what I managed to find out related to this matter: 
- it seems that Jitsi is not supported until 44.0a2 (2015-10-29) (the room is created, but the device permissions doorhanger is not displayed and the call is not initiated)
- starting with 44.0a2 (2015-10-29), Jitsi is supported and the issue is already reproducible
Tested this in calls between Windows 10 x64 and Mac OS X 10.11.6.

Comment 15

[Nils Ohlmeier [:drno]]

Attachment: Bug 1338521: don't over-write remote SSRC with random value

Review commit: https://reviewboard.mozilla.org/r/134908/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/134908/

Comment 16

[Nils Ohlmeier [:drno]]

Randell, I think the patch I put up clearly fixes a problem where we overwrite the remote SSRC with a random SSRC. But I'm not sure if that is all. Here are parts of a log from a bad call (the log is huge):

2017-04-28 19:41:12.270390 UTC - [Main Thread]: E/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:711: 0x7f42e27de000 SetRemoteSSRC: 0x96189d8b
2017-04-28 19:41:12.270393 UTC - [Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1989: StopReceiving Engine Already Receiving . Attemping to Stop 
2017-04-28 19:41:12.270476 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyRealtimeTrackData() listener=0x7f42b14e9a80, offset=0, duration=128
2017-04-28 19:41:12.270480 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyRealtimeTrackData() listener=0x7f42b8e49080, offset=0, duration=128
2017-04-28 19:41:12.270497 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline Discarding packets because transport not ready
2017-04-28 19:41:12.270525 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyQueuedChanges()
2017-04-28 19:41:12.270532 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyQueuedChanges()
2017-04-28 19:41:12.270533 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline Discarding packets because transport not ready
2017-04-28 19:41:12.270553 UTC - [AudioProxy #1]: D/signaling [AudioProxy #1|WebrtcAudioSessionConduit] AudioConduit.cpp:607: SendAudioFrame 
2017-04-28 19:41:12.270568 UTC - [Main Thread]: E/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:412: 0x7f42e27de000 DeleteRecvStream: Deleting stream 0x7f43141d8da0
2017-04-28 19:41:12.271510 UTC - [Main Thread]: E/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:457: 0x7f42e27de000 CreateRecvStream: Created stream 0x7f42b8f67e00
2017-04-28 19:41:12.271529 UTC - [Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:2004: StartReceiving Attemping to start... 
2017-04-28 19:41:12.271576 UTC - [Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1119: ConfigureRecvMediaCodecs 
2017-04-28 19:41:12.271606 UTC - [Main Thread]: D/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:1231: InitializeDataChannel

...

2017-04-28 19:42:54.994292 UTC - [Socket Thread]: V/mtransport Flow[77a24699a449b988:0,rtp(none)]; Layer[ice]: PacketReceived(0-1493408470876874 (id=2147483655 url=https://meet.jit.si/nilstest) aLevel=0,1,1107)
2017-04-28 19:42:54.994297 UTC - [Socket Thread]: V/mtransport Flow[77a24699a449b988:0,rtp(none)]; Layer[dtls]: PacketReceived(1107)
2017-04-28 19:42:54.994322 UTC - [Socket Thread]: V/mediapipeline Successfully unprotected an SRTP packet of len 1097
2017-04-28 19:42:54.994324 UTC - [Socket Thread]: V/mediapipeline 77a24699a449b988| Receive video[5a185332-fa99-4e49-a109-c7fc472fb434] received RTP packet.
2017-04-28 19:42:54.994658 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyRealtimeTrackData() listener=0x7f42b14e9a80, offset=0, duration=128
2017-04-28 19:42:54.994669 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyQueuedChanges()
2017-04-28 19:42:54.997637 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyRealtimeTrackData() listener=0x7f42b14e9a80, offset=0, duration=128
2017-04-28 19:42:54.997649 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyQueuedChanges()
2017-04-28 19:42:54.997666 UTC - [AudioProxy #1]: D/signaling [AudioProxy #1|WebrtcAudioSessionConduit] AudioConduit.cpp:607: SendAudioFrame 
2017-04-28 19:42:54.999217 UTC - [Main Thread]: E/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:711: 0x7f42e27de000 SetRemoteSSRC: 0x24a7fa63
2017-04-28 19:42:54.999221 UTC - [Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1989: StopReceiving Engine Already Receiving . Attemping to Stop 
2017-04-28 19:42:54.999338 UTC - [Main Thread]: E/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:412: 0x7f42e27de000 DeleteRecvStream: Deleting stream 0x7f42b8f67e00
2017-04-28 19:42:55.000542 UTC - [Main Thread]: E/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:457: 0x7f42e27de000 CreateRecvStream: Created stream 0x7f42b8f67e00
2017-04-28 19:42:55.000544 UTC - [Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:2004: StartReceiving Attemping to start... 
2017-04-28 19:42:55.000603 UTC - [Socket Thread]: D/signaling [Socket Thread|WebrtcVideoSessionConduit] VideoConduit.cpp:1887: operator(): seq# 52463, Len 160 
2017-04-28 19:42:55.000653 UTC - [Socket Thread]: D/signaling [Socket Thread|WebrtcVideoSessionConduit] VideoConduit.cpp:1887: operator(): seq# 52464, Len 160 
2017-04-28 19:42:55.000666 UTC - [Socket Thread]: D/signaling [Socket Thread|WebrtcVideoSessionConduit] VideoConduit.cpp:1887: operator(): seq# 52465, Len 160 

...

2017-04-28 19:45:51.970435 UTC - [Main Thread]: V/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:313: ConfigureCodecMode 
2017-04-28 19:45:51.970452 UTC - [Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:493: ConfigureSendMediaCodec for VP8
2017-04-28 19:45:51.970455 UTC - [Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:505: ConfigureSendMediaCodec for VideoConduit:0x7f42e147b000 stream count:1
2017-04-28 19:45:51.972720 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyRealtimeTrackData() listener=0x7f42b14e9a80, offset=0, duration=128
2017-04-28 19:45:51.972732 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyQueuedChanges()
2017-04-28 19:45:51.972752 UTC - [AudioProxy #1]: D/signaling [AudioProxy #1|WebrtcAudioSessionConduit] AudioConduit.cpp:607: SendAudioFrame 
2017-04-28 19:45:51.975454 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyRealtimeTrackData() listener=0x7f42b14e9a80, offset=0, duration=128
2017-04-28 19:45:51.975467 UTC - [Unnamed thread 0x7f42cf116860]: D/signaling [|WebrtcAudioSessionConduit] AudioConduit.cpp:672: GetAudioFrame 
2017-04-28 19:45:51.975642 UTC - [Unnamed thread 0x7f42cf116860]: D/signaling [|WebrtcAudioSessionConduit] AudioConduit.cpp:772: GetAudioFrame GetAudioFrame:Got samples: length 640 
2017-04-28 19:45:51.975650 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline Audio conduit returned buffer of length 640
2017-04-28 19:45:51.975724 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyQueuedChanges()
2017-04-28 19:45:51.977051 UTC - [Unnamed thread 0x7f42e1ccf160]: D/signaling [|WebrtcVideoSessionConduit] VideoConduit.cpp:2044: SendRtcp : len 24 
2017-04-28 19:45:51.977078 UTC - [Unnamed thread 0x7f42e1ccf160]: D/signaling [|WebrtcVideoSessionConduit] VideoConduit.cpp:2053: SendRtcp Sent RTCP Packet 
2017-04-28 19:45:51.977129 UTC - [Socket Thread]: V/mediapipeline Successfully protected an SRTCP packet of len 38
2017-04-28 19:45:51.977134 UTC - [Socket Thread]: V/mediapipeline 77a24699a449b988| Receive video[mixedlabelvideo0] sending RTCP packet
2017-04-28 19:45:51.977150 UTC - [Socket Thread]: V/mtransport Flow[77a24699a449b988:0,rtp(none)]; Layer[ice]:  SendPacket(38) succeeded
2017-04-28 19:45:51.978540 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyRealtimeTrackData() listener=0x7f42b14e9a80, offset=0, duration=128
2017-04-28 19:45:51.978555 UTC - [Unnamed thread 0x7f42cf116860]: V/mediapipeline MediaPipeline::NotifyQueuedChanges()
2017-04-28 19:45:51.979620 UTC - [Main Thread]: E/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:711: 0x7f42e27de000 SetRemoteSSRC: 0x96189d8b
2017-04-28 19:45:51.979626 UTC - [Main Thread]: D/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:1989: StopReceiving Engine Already Receiving . Attemping to Stop 
2017-04-28 19:45:51.980171 UTC - [Main Thread]: E/signaling [main|WebrtcVideoSessionConduit] VideoConduit.cpp:412: 0x7f42e27de000 DeleteRecvStream: Deleting stream 0x7f42b8f67e00

And then it hits the assertion. 0x96189d8b is the valid SSRC from the signaling. 0x24a7fa63 is the SSRC for another conduit. I don't quite get how the incoming RTP packet which gets assigned to the correct media pipeline, then can suddenly call setRemoteSSRC() on the wrong conduit.

Comment 17

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8863073 [details]
Bug 1338521: don't over-write remote SSRC with random value

https://reviewboard.mozilla.org/r/134908/#review138146

r+, though I'm not sure it solves the assertion we hit occasionally - that may be a different bug that was exposed by reconfiguring on every call start

Comment 18

[Nils Ohlmeier [:drno]]

Attachment: Bug 1338521: delete send/recv video stream when destroying.

Review commit: https://reviewboard.mozilla.org/r/135260/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/135260/

Comment 19

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8863472 [details]
Bug 1338521: delete send/recv video stream when destroying.

https://reviewboard.mozilla.org/r/135260/#review138218

Per IRC, this double-frees

Comment 20

[Nils Ohlmeier [:drno]]

So I found issue which could cause all kind of issue, although I doubt it causes the assertion in the DestroyVideoReceiveStream() to kick in.

Jitsi negotiates several video m-sections depending on how many participants are in a call and if parties recently joined or dropped from a call. All these video m-sections use the same PT number (usually 100 for VP8). Now sometimes Jitsi sends RTP packets with an unnounced SSRC, for example an RTP packet from a previously active m-section comes in after said m-section has been de-activated.
Now with our current code this causes a problem, because the incoming RTP packet will be "offered" to all active media pipeline. None of them has a matching SSRC for the packet, but all video pipeline fall back to match by payload type and accept the packet. With the latest addition of restarting the video receivers on unanounced SSRC's this causes restarts on all video streams.

I think what we need to do to solve this issue is stop offering an incoming RTP packet to all registered pipelines. Instead we should build a demuxer which knows about SSRC's and PT's (and MID's and RID's in the future) and routes packets to a single pipeline. If that demuxer gets an RTP packet with an unknown PT it can route the packet to the correct pipeline if the PT is unique among all pipelines. But if it's a non-unique PT it should drop the packet instead of forwarding it to multiple pipelines and causing trouble.

Comment 21

[Byron Campen [:bwc]]

(In reply to Nils Ohlmeier [:drno] from comment #20)
> So I found issue which could cause all kind of issue, although I doubt it
> causes the assertion in the DestroyVideoReceiveStream() to kick in.
> 
> Jitsi negotiates several video m-sections depending on how many participants
> are in a call and if parties recently joined or dropped from a call. All
> these video m-sections use the same PT number (usually 100 for VP8). Now
> sometimes Jitsi sends RTP packets with an unnounced SSRC, for example an RTP
> packet from a previously active m-section comes in after said m-section has
> been de-activated.
> Now with our current code this causes a problem, because the incoming RTP
> packet will be "offered" to all active media pipeline. None of them has a
> matching SSRC for the packet, but all video pipeline fall back to match by
> payload type and accept the packet. 

We have code that is supposed to find the payload types that are unique to each m-section, so this should not be happening.

Comment 22

[Nils Ohlmeier [:drno]]

(In reply to Byron Campen [:bwc] from comment #21)
> We have code that is supposed to find the payload types that are unique to
> each m-section, so this should not be happening.

Well Jitsi appears to be always the offerer and they send us offers like this:

v=0
...
a=group:BUNDLE audio-499738133 audio-1749551224 video-2011480974 video-2518195595 data audio-2993297918 video-614988387
m=audio 1 RTP/SAVPF 111 103 104 126
...
m=audio 1 RTP/SAVPF 111 103 104 126
...
m=video 1 RTP/SAVPF 100 107 101
...
m=video 1 RTP/SAVPF 100 107 101
...
m=application 1 DTLS/SCTP 5000
...
m=audio 1 RTP/SAVPF 111 103 104 126
...
m=video 1 RTP/SAVPF 100 107 101
...

We can choose unique PT's for what we are sending. But we can't make them choose unique PT's, or?

Comment 23

[Randell Jesup [:jesup] (needinfo me)]

We can choose unique PT's for what we're *receiving* - and so we could side-step the "can't tell which pipeline an unknown SSRC is for" because no two will share PTs.  It is annoying... but will work*.  Note that if they offer bundles (without mid) with the same PT, if we change SSRC they will have to somehow deal with it.  

* Assuming of course they properly handle non-matching (send/recv) PTs for a stream, which they should....

Comment 24

[Nils Ohlmeier [:drno]]

(In reply to Randell Jesup [:jesup] from comment #23)
> We can choose unique PT's for what we're *receiving*

With the sample in comment #22 how can we choose to receive unique PT's?

Note: PT 100 is an all three m-lines VP8, 107 is H264 is all three, and 101 is VP9 in all three.
Are you suggesting we should choose different codecs for the m-lines to get to unique PTs here?

I totally agree that we can offer for all three m-lines different PT's for our outgoing video (assuming we would actually send multiple outgoing videos). But they are still going to send all three VP8 streams with PT 100, if we stick to the default choice of VP8 in all three cases.

Comment 25

[Byron Campen [:bwc]]

(In reply to Nils Ohlmeier [:drno] from comment #22)
> (In reply to Byron Campen [:bwc] from comment #21)
> > We have code that is supposed to find the payload types that are unique to
> > each m-section, so this should not be happening.
> 
> Well Jitsi appears to be always the offerer and they send us offers like
> this:
> 
> v=0
> ...
> a=group:BUNDLE audio-499738133 audio-1749551224 video-2011480974
> video-2518195595 data audio-2993297918 video-614988387
> m=audio 1 RTP/SAVPF 111 103 104 126
> ...
> m=audio 1 RTP/SAVPF 111 103 104 126
> ...
> m=video 1 RTP/SAVPF 100 107 101
> ...
> m=video 1 RTP/SAVPF 100 107 101
> ...
> m=application 1 DTLS/SCTP 5000
> ...
> m=audio 1 RTP/SAVPF 111 103 104 126
> ...
> m=video 1 RTP/SAVPF 100 107 101
> ...
> 
> We can choose unique PT's for what we are sending. But we can't make them
> choose unique PT's, or?

We can choose unique PTs for receive, sure. But what I was saying was that we have code that looks at the PTs we're receiving, calculates which ones are unique to their m-section, and allows only those through the filter if we don't know about the ssrc. Non-unique PTs should always be dropped by our code if we don't know about the SSRC, but it sounds like you caught it letting that stuff through?

Comment 26

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Nils Ohlmeier [:drno] from comment #24)
> (In reply to Randell Jesup [:jesup] from comment #23)
> > We can choose unique PT's for what we're *receiving*
> 
> With the sample in comment #22 how can we choose to receive unique PT's?
> 
> Note: PT 100 is an all three m-lines VP8, 107 is H264 is all three, and 101
> is VP9 in all three.
> Are you suggesting we should choose different codecs for the m-lines to get
> to unique PTs here?

No - use different numbers for vp8 in each m-line -- no overlap in PT values (within a bundle).  We'd need a per-bundle PT allocator, and ask it for each payload needed for a number.  Note that on an offer this might get dicey or even impossible to avoid overlaps, in which case we simply live with it and start duplicating, OR we take advantage of the ability in SDP to re-assign "static" payloads to dynamic codecs (make PT 7 into VP8, for example).  (There's a limited space for PTs, and a lot of (ancient, irrelevant) static definitions to codecs we don't support).

> I totally agree that we can offer for all three m-lines different PT's for
> our outgoing video (assuming we would actually send multiple outgoing
> videos). But they are still going to send all three VP8 streams with PT 100,
> if we stick to the default choice of VP8 in all three cases.

You mean incoming video... SDP offers (and answers) say "send to me using these PTs".  If the other side puts VP8 on PT 100 for all 3, then we'll send three streams with PT 100 in the bundle, and if we change SSRC they have to deal with it.  Not our problem.  By offering (or answering) with different PTs for every payload in the bundle, we can never be confused about where to route an incoming packet with a new SSRC to.  The limit on the number of PTs is why we decided to use MID, but we can get away with this ok until MID is generally supported.

Comment 27

[Nils Ohlmeier [:drno]]

Forgot to update: I looked at the PTs form the wrong perspective.

I'll double check later today, if unique PT's work with Jitsi as expected.

Comment 28

[Nils Ohlmeier [:drno]]

So the code for unique PT's does not result in unique PT's. In a three participant call I had an answer with PT 100 for all three video m-sections.

Comment 29

[Nils Ohlmeier [:drno]]

For future reference this is the URL to crash-stats for searching for the crashes caused by hitting the webrtc.org assertion in case of renegotiation: 
https://crash-stats.mozilla.com/signature/?product=Firefox&proto_signature=~SetRemoteSSRC&signature=mozalloc_abort%20%7C%20abort%20%7C%20rtc%3A%3AFatalMessage%3A%3A~FatalMessage&date=%3E%3D2016-05-13T19%3A51%3A00.000Z&date=%3C2017-05-12T19%3A51%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_sort=-date&page=1#reports

25 crashes so far. Good news is that this will allow us to track if the existing patch changes the frequency of that crash.

Comment 30

[Nils Ohlmeier [:drno]]

I think we are facing multiple problems here. And I'm not sure we have identified all of them yet.

But I also think we should land attachment 8863073 [details], which is definitely causing problems. Plus it is small enough to be uplifted.

There is more work to be done in bug 1365797 (which is probably to big to be uplifted). And we should keep an eye on the crash stats for the SetRemoteSSRC from comment #29.

I'll try if I can repro the SetRemoteSSRC assertion in RR.

Comment 31

[Nils Ohlmeier [:drno]]

Comment on attachment 8863073 [details]
Bug 1338521: don't over-write remote SSRC with random value

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/134908/diff/1-2/

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ad21dd422ad2
don't over-write remote SSRC with random value r=jesup

Comment 33

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/ad21dd422ad2

Comment 34

[Ryan VanderMeulen [:RyanVM]]

Please request Beta & ESR52 approval on this when you get a chance.

Comment 35

[Randell Jesup [:jesup] (needinfo me)]

Nils - please request beta/esr52

Comment 36

[Nils Ohlmeier [:drno]]

Comment on attachment 8863073 [details]
Bug 1338521: don't over-write remote SSRC with random value

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration: Avoid WebRTC video rendering issues for ESR users.
User impact if declined: Late or no rendering of the remote video in WebRTC calls.
Fix Landed on Version: 55
Risk to taking this patch (and alternatives if risky): Very low risk (see below in beta request).
String or UUID changes made by this patch: N/A

See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more info.

Approval Request Comment
[Feature/Bug causing the regression]: bug 1337777
[User impact if declined]: WebRTC calls might experience no or delayed video
[Is this code covered by automated tests?]: Yes
[Has the fix been verified in Nightly?]: Only manually with a local build
[Needs manual test from QE? If yes, steps to reproduce]: Only visible in log files. As we still have other issues with meet.jit.si is probably not feasible to verify this fix, besides a slightly better call establishing when doing calls on meet.jit.si.
[List of other uplifts needed for the feature/fix]: N/A
[Is the change risky?]: No
[Why is the change risky/not risky?]: We only avoid starting the video renderer with a wrong random number initially.
[String changes made/needed]: N/A

Comment 37

[Gerry Chang [:gchang]]

Track 54+/55+ as regression.

Hi Iulia, could you help verify if this issue was fixed as expected on a latest Nightly build? Thanks!

Comment 38

[Nils Ohlmeier [:drno]]

(In reply to Gerry Chang [:gchang] from comment #37)
> Track 54+/55+ as regression.
> 
> Hi Iulia, could you help verify if this issue was fixed as expected on a
> latest Nightly build? Thanks!

Just to be clear: the patch here should improve the situation. But it's not going to fix all the outstanding issues we have with Jitsi. Its going to be hard to differentiate all of the outstanding issues.

Comment 39

[Iulia Cristescu, QA [:JuliaC] (away, please needinfo? [e-mail])]

Hello! Investigated again this issue and I can confirm that it is still reproducible on latest Nightly 55.0a1 (2017-05-23), using Ubuntu 16.04 x64 and Windows 10 x64. Afetr refresh, the remote video does not appear at all and the self-views still work.

Comment 40

[Gerry Chang [:gchang]]

Comment on attachment 8863073 [details]
Bug 1338521: don't over-write remote SSRC with random value

Slightly improve the situation. Beta54+. Should be in 54 beta 11.

Comment 41

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/9beb51c85160

Comment 42

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8863073 [details]
Bug 1338521: don't over-write remote SSRC with random value

The fix seems low risk and just moving the initialization to an earlier stage, let's uplift to ESR52.

Comment 43

[Ryan VanderMeulen [:RyanVM]]

This needs a rebased patch for ESR52.

Comment 44

[Ryan VanderMeulen [:RyanVM]]

Upon closer inspection, it turns out that ESR52 wasn't affected after all (per IRC discussion with Randell).

Comment 45

[Julien Cristau [:jcristau]]

Comment on attachment 8863073 [details]
Bug 1338521: don't over-write remote SSRC with random value

clearing esr52 approval flag since this turned out not to be needed

Comment 46

[Iulia Cristescu, QA [:JuliaC] (away, please needinfo? [e-mail])]

(In reply to Nils Ohlmeier [:drno] from comment #38)
> (In reply to Gerry Chang [:gchang] from comment #37)
> > Track 54+/55+ as regression.
> > 
> > Hi Iulia, could you help verify if this issue was fixed as expected on a
> > latest Nightly build? Thanks!
> 
> Just to be clear: the patch here should improve the situation. But it's not
> going to fix all the outstanding issues we have with Jitsi. Its going to be
> hard to differentiate all of the outstanding issues.

Verified again the bug on 54.0 (20170605134926), using Windows 10 x64, Ubuntu 16.04 x64 and Mac OS X 10.11.6. The main issue is still reproducible, as mentioned in comment 39. However, a small improvement can be noticed: sometimes the remote video stream is working, but the audio stream is broken or vice versa.

Comment 47

[Alexandru Simonca [:asimonca], Release Desktop QA]

Hello,
We've tested the Jitsi service again on the latest Nightly 56.0a1 build (id: 20170710030203) and this issue is still reproducible on the following OSs:

- Ubuntu 14.04 x86 + Mac OS X 10.13 Beta
- Mac OS X 10.13 Beta + Windows 7 x64

Comment 48

[Iulia Cristescu, QA [:JuliaC] (away, please needinfo? [e-mail])]

Also tested the issue again on 55.0b13 build1 (20170727114534), using:
- Windows 10 x64 + Mac OS X 10.11.6
- Windows 10 x64 + Ubuntu 16.04 x64
and it is still reproducible, like stated in comment 46.