1430476 - Jitsi freezes for the remote party when using FireFox

Status: RESOLVED WORKSFORME

(Core :: WebRTC: Signaling, defect, P1)

Description

[Gurenko Alex]

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
Build ID: 20180104170325

Steps to reproduce:

Start a Jitsi call with FF 57 on your side and Chrome on the other side


Actual results:

Remote party complains that they do not receive video shortly after the call starts. If the page is refreshed same happens, video briefly works and then freezes and do not unfreeze unless the page is refreshed. 


Expected results:

Since Jitsi does not show that there is any problem and the same call can be continued without any problem on Chrome something needs to be changed/fixed in FF57

Comment 1

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

User Agent 	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
Build ID 	20180115100056

I was able to reproduce the issue on the latest Nightly 59.0a1 on Windows 10 x64 and Mac 10.12.6.

Comment 2

[Nils Ohlmeier [:drno]]

As discussed I would first check for RTP packets still "flowing". And as a next step check with chrome://webrtc-internals and/or Fx RTP logger what is going on on the RTCP level.

Comment 3

[Michael Froman [:mjf]]

I can reproduce this running Fx57 on my linux box to Chrome on my OSX box.  The connection info box in the preview window in the upper right side of the jitsi screen on Fx57 shows:
Connection: Nonoptimal
Bitrate: 178 Kpbs (down) 354 Kpbs (up)
Packet loss: 1% (down) 0% (up)
Resolution: N/A
Frame rate: 29

For Fellow Jitser info:
Connection: Inactive
Bitrate: N/A (down) 71 Kpbs (up)
Packet loss: 0% (down) 2% (up)
Resolution: N/A
Frame rate: 25


On Chrome, connection info box in the preview window in the upper right side of the jitsi screen:
Connection: Nonoptimal
Bitrate: N/A (down) 228 Kpbs (up)
Packet loss: 0% (down) 0% (up)
Resolution: 320x256
Frame rate: 29

For Fellow Jitser info:
Connection: Nonoptimal
Bitrate: 70 (down) 350 Kpbs (up)
Packet loss: 2% (down) 0% (up)
Resolution: 0x0
Frame rate: N/A

Comment 5

[Michael Froman [:mjf]]

I can reproduce this about 50% of the time.  When it doesn't reproduce, I don't see any NACKs in the packets logged.  When it does reproduce (freeze) I _do_ see NACKs in the packets logged.

After some rtp packing logging, I can see Firefox sending video RTP packets, and responding to NACKs by retransmitting those packets.  Interesting facts:
- Most of the lost packets end up requiring multiple retransmissions (usually 2)
- I see multiple NACKs (usually around 4) before a retransmission happens
- Even after waiting a couple minutes in the failing case, I do not see any PLI RTCP packets.

Comment 6

[Nils Ohlmeier [:drno]]

George, from our perspective we don't see anything going wrong on the Firefox side of things. Can you help us with some logging on the Jitsi bridge to figure out why the retransmissions don't fix a frozen video rendering?

Comment 7

[Michael Froman [:mjf]]

Attachment: log for a working call

This is a working jitsi call from Fx on linux to Chrome on OSX.

Comment 8

[Michael Froman [:mjf]]

Attachment: log for a failing call

This is a failing jitsi call from Fx on linux to Chrome on OSX.

Comment 9

[Michael Froman [:mjf]]

If anyone is interesting in creating pcap files from the logged packets in those logs, I used this command:
egrep '(RtpLogger)' log.txt | cut -d '|' -f 2 | cut -d ':' -f 3- | text2pcap -D -n -l 1 -i 17 -u 1234,1235 -t '%H:%M:%S.' - rtp.pcap

(adapted from https://blog.mozilla.org/webrtc/debugging-encrypted-rtp-is-more-fun-than-it-used-to-be/ )

Comment 10

[George Politis [:gp]]

There are currently a couple of nasty bugs that impact https://meet.jit.si/; so these freezes may not be directly related to FF. Alex, are you able to repro if you use https://beta.meet.jit.si/ (this is our staging env, please don't use for everyday work)?

Comment 11

[Michael Froman [:mjf]]

One thing I've noticed, is when I see the freeze (with the message on the Chrome side that video has been turned off to save bandwidth), on the Fx side we're only showing one PeerConnection in about:webrtc.

Comment 12

[Michael Froman [:mjf]]

In the freeze case, the incoming video in Fx is super low resolution.  When it doesn't freeze, the resolution is pretty good.  In every case that I have noticed, when the freeze happens, the "me" connection info in the upper right of the Chrome window is showing either yellow or red.  I'll attach the about:webrtc for a working session and a freeze session that shows 2 vs 1 PeerConnection.

Comment 13

[Michael Froman [:mjf]]

Attachment: aboutWebrtc-working.html

working call showing 2 PeerConnections both moving bytes

Comment 14

[Michael Froman [:mjf]]

Attachment: aboutWebrtc-freeze.html

call with freeze showing only 1 PeerConnection (that is still moving bytes).  In this case, Fx is still getting video from Chrome although it is very low res.

Comment 15

[Michael Froman [:mjf]]

I've also see the same freeze on https://beta.meet.jit.si/

Comment 16

[Gurenko Alex]

(In reply to George Politis [:gp] from comment #10)
> There are currently a couple of nasty bugs that impact https://meet.jit.si/;
> so these freezes may not be directly related to FF. Alex, are you able to
> repro if you use https://beta.meet.jit.si/ (this is our staging env, please
> don't use for everyday work)?

 I will try it hopefully today and will let you know, but I think it was the same behaviour last time I've tried it with FF57 when it was in beta.

Comment 17

[Michael Froman [:mjf]]

FYI: I'm not seeing this at all this morning on beta.meet.jit.si.

Comment 18

[Michael Froman [:mjf]]

Do you have any updates on this?

Comment 19

[Gurenko Alex]

Attachment: ff-logs-jitsi.log

(In reply to Michael Froman [:mjf] from comment #18)
> Do you have any updates on this?

 Sorry that it took so long. Did some testing today and both production and beta jitsi shows same issues. Did an hour call in a same session with Chrome after that with no issues at all.

 I've noticed repeating messages in a console in FF57, that are not present in Chrome's console during the call:

XML Parsing Error: not well-formed
Location: 
Line Number 1, Column 168:

I've grabbed some console output during the connectivity issues, I'm not sure whether it helps in any way.

Comment 20

[Gurenko Alex]

(In reply to Michael Froman [:mjf] from comment #17)
> FYI: I'm not seeing this at all this morning on beta.meet.jit.si.

I'm wondering, can you reproduce this problem now? And after all, is it something that jitsi implemented in non-standard way or FF problem?

Comment 21

[Michael Froman [:mjf]]

On non-beta jitsi, yes. I can reproduce, but in the cases where I see it happen I'm seeing packets loss (greater than 10%).
On beta, I did not see the issue.

Comment 22

[Gurenko Alex]

(In reply to Michael Froman [:mjf] from comment #21)
> On non-beta jitsi, yes. I can reproduce, but in the cases where I see it
> happen I'm seeing packets loss (greater than 10%).
> On beta, I did not see the issue.

So my log does not help? Anything I can capture to help you troubleshoot?

Comment 23

[Michael Froman [:mjf]]

(In reply to Gurenko Alex from comment #22)
> (In reply to Michael Froman [:mjf] from comment #21)
> > On non-beta jitsi, yes. I can reproduce, but in the cases where I see it
> > happen I'm seeing packets loss (greater than 10%).
> > On beta, I did not see the issue.
> 
> So my log does not help? Anything I can capture to help you troubleshoot?

Nils?

Comment 24

[Nils Ohlmeier [:drno]]

The log you provided I think are browser console messages from Jitsi. We don't know how interpret these. My educated guess from looking at it is that you are trying to send 1080p HD while the network quality goes down. Strangely the logs indicate no packet loss.

George: can you help to interpret the provided console log?

Gurenko: can I ask you to check if you can repro the problem with Firefox 60? We recently fixed a problem which resulted in packet loss. So I'm wondering if this is related.

Comment 25

[Gurenko Alex]

(In reply to Nils Ohlmeier [:drno] from comment #24)
> The log you provided I think are browser console messages from Jitsi. We
> don't know how interpret these. My educated guess from looking at it is that
> you are trying to send 1080p HD while the network quality goes down.
> Strangely the logs indicate no packet loss.
> 
> George: can you help to interpret the provided console log?
> 
> Gurenko: can I ask you to check if you can repro the problem with Firefox
> 60? We recently fixed a problem which resulted in packet loss. So I'm
> wondering if this is related.

 I've tried Nightly (FF60) and it kind of works, still showing errors in connectivity and b&w picture, but recovering from it. Stable Jitsi still freezes almost at the beginning and not recovering unless page is refreshed. Unfortunately in exactly same conditions Chrome still works perfectly fine not showing any kind of connectivity issues.

Comment 26

[Gurenko Alex]

So, I've tried yesterday new Jitsi and the problem still exists.

Comment 27

[Gurenko Alex]

If it still matters, the issue is still here to this day with FF62.0.3... =( Still have to use Chrome for Jitsi calls...

Comment 28

[Gurenko Alex]

Still happening on FF64

Comment 29

[Gurenko Alex]

I've opened a topic on Jitsi's forum as well to try and find out the root cause: https://community.jitsi.org/t/jitsi-users-jitsi-meet-freezes-on-firefox-but-not-on-chrome/16135

Comment 30

[Cristian Comorasu, QA [:ccomorasu], Releae Desktop QA - Please contact [e-mail]]

This issue is still reproducible.

Comment 31

[Gurenko Alex]

(In reply to Cristian Comorasu [:ccomorasu], Release Desktop QA from comment #30)

This issue is still reproducible.

Is the root cause still unknown? Is there still something I can help with to pin point it?

Comment 32

[Cristian Comorasu, QA [:ccomorasu], Releae Desktop QA - Please contact [e-mail]]

I am not sure.
Michael, any thoughts?

Comment 33

[Michael Froman [:mjf]]

Sorry - I haven't looked at this since comment 24 asking for some input from George. I'll redirect this to Nils since I'm currently working on other things.

Comment 34

[Gurenko Alex]

Did a quick test yesterday with FF65, seems like the issue is fixed now. Didn't see previous errors in the web console, and video didn't freeze for ~5-7 video. Will do more testing over next few days with longer calls, but it's a very good first sign.

Comment 35

[Gurenko Alex]

I want to confirm that the issue per se is gone in FF65. Made a 3h call with no interruptions on any side. Performance is worse than on Chrome, but finally I can use FF for Jitsi calls!

Comment 36

[Gurenko Alex]

Interesting that the issue is only gone on beta.meet.jit.si not on production, so still unclear if that FF's only issue or still a combination. I'm still confused if I'm the only one with such issue?

Comment 37

[Chris Cameron]

Using Nightly 68.0a1 on macOS 10.13.6, and I believe I'm experiencing this problem. In calls with three different people (independently) and they report that my video alternates between very low quality and a Jitsi message telling them I'm experiencing network connectivity issues. On my side the video quality is very poor, but doesn't freeze often. My network monitor shows outgoing traffic has a very definite saw tooth pattern.

When I switch to Safari video quality looks great and consistent on both sides and I'm told there are no more notices for the other party that I'm having network connectivity issues.

I say this not knowing much about WebRTC, but, I wonder if it's related to the peer-to-peer functionality. I notice that Safari succeeds in connecting to a very nearby IP, whereas Firefox is stuck going through Virginia (which is not nearby).

Comment 38

[Gurenko Alex]

(In reply to Chris Cameron from comment #37)

Using Nightly 68.0a1 on macOS 10.13.6, and I believe I'm experiencing this problem. In calls with three different people (independently) and they report that my video alternates between very low quality and a Jitsi message telling them I'm experiencing network connectivity issues. On my side the video quality is very poor, but doesn't freeze often. My network monitor shows outgoing traffic has a very definite saw tooth pattern.

When I switch to Safari video quality looks great and consistent on both sides and I'm told there are no more notices for the other party that I'm having network connectivity issues.

I say this not knowing much about WebRTC, but, I wonder if it's related to the peer-to-peer functionality. I notice that Safari succeeds in connecting to a very nearby IP, whereas Firefox is stuck going through Virginia (which is not nearby).

I'm wondering if beta.meet.jit.si would work for you too?

Comment 39

[Gurenko Alex]

Some annual update; As of FF72.0.1 general performance is much better as it used to be, but hangs are still occurring, but FF seems to recover now, although the experience is still not comparable to Chrome's experience on a same machine.

Comment 40

[Dan Minor [:dminor]]

From Comment 39, I'm going to call the issue as filed fixed. We still run into problems with jitsi, e.g. due to not supporting transport-cc for bandwidth estimation which might explain the ongoing symptoms here, for which we have Bug 1606823.