1464448 - cisco teams app leads to frozen media

Status: UNCONFIRMED

(Core :: WebRTC: Networking, defect, P3)

Description

[arun ganeshan]

Attachment: Firefox_Failed_no_Media

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.48 Safari/537.36

Steps to reproduce:

This issue happens with https://teams.webex.com/ there is a constant video freeze in a particular scenario with firefox to firefox 

Scenarios:
1. Alice and bob have login on web client (Both Firefox).
2. Alice dials [e-mail]  it will ask if you are a 'host' or 'non host' click not a host.
3. Bob tries to dial [e-mail] now he selects as host then enters PIN '1829'.
4. Alice gets connected to bob. 



Actual results:

Alice video freezes , sometimes frames come back after 30 sec then freezes again , Sometimes the video works also not sure why the video freeze happens randomly . But it happens a lot 

Here are the logs 



Expected results:

The video freeze should not happen . The call should go smooth

Comment 1

[Alex Chronopoulos [:achronop]]

Can you please mention the operating system and the versions that you are testing with.

Comment 2

[arun ganeshan]

(In reply to Alex Chronopoulos [:achronop] from comment #1)
> Can you please mention the operating system and the versions that you are
> testing with.

Its OSX (10.13.4 ) on both the sides. And version of firefox is 60.0.1.

Comment 3

[arun ganeshan]

To add you can go one step ahead and start sharing screen if you dont see the video freeze during the initial call setup

Comment 4

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Alex, I saw you have a webex account. Can you call me on webex tomorrow to try to repro this? Thanks.

Comment 5

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I should say I wasn't able to repro this by myself. It never asked me if I was a host. I don't use webex often, so please let me know if I'm doing it wrong.

Comment 6

[arun ganeshan]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #5)
> I should say I wasn't able to repro this by myself. It never asked me if I
> was a host. I don't use webex often, so please let me know if I'm doing it
> wrong.

Can you use https://teams.webex.com to login then dial the PMR info provided below .
you dont need your own PMR to test this 

2. Alice dials [e-mail]  it will ask if you are a 'host' or 'non host' click not a host.
3. Bob tries to dial [e-mail] now he selects as host then enters PIN '1829'.


You can 1:1 me or paul johns if you are not able to reproduce this issue

Comment 7

[arun ganeshan]

jan we didnt see any error on the decoder while the firefox was frozen. Can you confirm if the below code is sufficient to verify the logs

FIREFOX_EXEC=/Applications/Firefox.app/Contents/MacOS/firefox

set -e

# Signaling logs (SDP):
export MOZ_LOG_FILE=~/Desktop/NSPR.log
export WEBRTC_TRACE_FILE=~/Desktop/WebRTC.log
export MOZ_LOG=MediaManager:3,signaling:5,webrtc_trace:3,webrtc_trace:3,timestamp:3,GetUserMedia:3,mtransport:3,rtplogger:3,jsep:3 # signaling and signaling:5mtransport: 1-9 mtransport:5,GetUserMedia:4
# ICE/STUN/TURN logs:
export R_LOG_LEVEL=4
export R_LOG_DESTINATION=stderr
export R_LOG_VERBOSE=1  # include the module name generating the message


$FIREFOX_EXEC

Comment 8

[Nils Ohlmeier [:drno]]

(In reply to arun ganeshan from comment #7)
> jan we didnt see any error on the decoder while the firefox was frozen. Can
> you confirm if the below code is sufficient to verify the logs
> 
> FIREFOX_EXEC=/Applications/Firefox.app/Contents/MacOS/firefox
> 
> set -e
> 
> # Signaling logs (SDP):
> export MOZ_LOG_FILE=~/Desktop/NSPR.log
> export WEBRTC_TRACE_FILE=~/Desktop/WebRTC.log
> export
> MOZ_LOG=MediaManager:3,signaling:5,webrtc_trace:3,webrtc_trace:3,timestamp:3,
> GetUserMedia:3,mtransport:3,rtplogger:3,jsep:3 # signaling and
> signaling:5mtransport: 1-9 mtransport:5,GetUserMedia:4

In general this should work. But this is an excessive list which probably is going to slow down Firefox.

For this scenario I would recommend something like this:

MOZ_LOG=signaling:5,webrtc_trace:5,timestamp,rtplogger:5,jsep5:signaling5

> # ICE/STUN/TURN logs:
> export R_LOG_LEVEL=4
> export R_LOG_DESTINATION=stderr
> export R_LOG_VERBOSE=1  # include the module name generating the message

You probably don't need to log ICE in this scenario.

From looking at some Wireshark traces which Arun provided me it looks like there are quite big gaps where 5-20 RTP packets are missing. And I don't see retransmissions of these missing packets.

I would recommend to use Firefox RTP logging https://blog.mozilla.org/webrtc/debugging-encrypted-rtp-is-more-fun-than-it-used-to-be/ to capture clear text RTP packets in Firefox logs and then extract them from the log into a pcap file and search for NACK, PLI etc.

Comment 9

[arun ganeshan]

(In reply to Nils Ohlmeier [:drno] from comment #8)
> (In reply to arun ganeshan from comment #7)
> > jan we didnt see any error on the decoder while the firefox was frozen. Can
> > you confirm if the below code is sufficient to verify the logs
> > 
> > FIREFOX_EXEC=/Applications/Firefox.app/Contents/MacOS/firefox
> > 
> > set -e
> > 
> > # Signaling logs (SDP):
> > export MOZ_LOG_FILE=~/Desktop/NSPR.log
> > export WEBRTC_TRACE_FILE=~/Desktop/WebRTC.log
> > export
> > MOZ_LOG=MediaManager:3,signaling:5,webrtc_trace:3,webrtc_trace:3,timestamp:3,
> > GetUserMedia:3,mtransport:3,rtplogger:3,jsep:3 # signaling and
> > signaling:5mtransport: 1-9 mtransport:5,GetUserMedia:4
> 
> In general this should work. But this is an excessive list which probably is
> going to slow down Firefox.
> 
> For this scenario I would recommend something like this:
> 
> MOZ_LOG=signaling:5,webrtc_trace:5,timestamp,rtplogger:5,jsep5:signaling5
> 
> > # ICE/STUN/TURN logs:
> > export R_LOG_LEVEL=4
> > export R_LOG_DESTINATION=stderr
> > export R_LOG_VERBOSE=1  # include the module name generating the message
> 
> You probably don't need to log ICE in this scenario.
> 
> From looking at some Wireshark traces which Arun provided me it looks like
> there are quite big gaps where 5-20 RTP packets are missing. And I don't see
> retransmissions of these missing packets.
> 
> I would recommend to use Firefox RTP logging
> https://blog.mozilla.org/webrtc/debugging-encrypted-rtp-is-more-fun-than-it-
> used-to-be/ to capture clear text RTP packets in Firefox logs and then
> extract them from the log into a pcap file and search for NACK, PLI etc.

Thanks for the information nils.
I tried to collect the pacap and webrtc logs as mentioned above here is the link for the pcap and webrtc logs 

https://cisco.box.com/s/9d4si8tf1sdhpwg7ira8t7xd0gnn87sz

There was a small change in the script to add year month format

Comment 10

[arun ganeshan]

Hi our server team had a look at the issue . They had a question on firefox behavior when a key frame back from the server was partially not received. Will the firefox keep sending PLI ??. We saw that browser didnt send a PLI request back if the key frame coming in doesnt reach the browser

Comment 11

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I don't know the answer on PLI. Nils?

Comment 12

[arun ganeshan]

Scenario : Firefox --> PLI --> Sever 
           Server ---> Key Frame --> As the key frame was large partially it got lost 
           Firefox --> PLI with in 30 ms --> Server skips it as it was too quick
           Firefox --> ? Doesnt sent any PLI any more 

Above is the scenario . Our SFU currently skips any PLI with in a sec of last PLI .wanted to check the 1) rate at which the PLI request is sent if the key frame is not received. 2) What happens in the above mentioned scenario 

note: we currently dont support rtcp nack so little hard to recover the lost packet

Comment 13

[Nils Ohlmeier [:drno]]

To restate the problem:

This is with a special media server which doesn't support NACK, but tries to work reliable with PLI and FEC. But Firefox doesn't support FEC yet. Thus any packet loss will always result in PLI request, which is a big waste of bandwidth. And the fact that a single packet loss results in requesting a new key frame probably results in even more packet loss.

As you can see here https://searchfox.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/video_coding/video_receiver.cc#98 there is timer protection against processing too many PLI's.

It's my understanding that problem happens when there are multiple RTP streams. So my current guess is that is might be the same problem as in bug 1465473.

Arun can you check if the PLI's from your server are by any chance attached to sender reports?

Comment 14

[arun3528]

Our media server doesnt send compound RTCP with PLI mainly in the case of bundling. Does firefox support it ?

Comment 15

[Paul Adenot (:padenot)]

Byron, jib, can someone followup here?

Comment 16

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Not sure how to unstuck this. Byron, do you know the answer to comment 14?

Comment 17

[Byron Campen [:bwc]]

Our negotiation code supports pli, and will negotiate it by default. Our bundle code does not attempt to filter RTCP, and leaves it up to webrtc.org to match up RTCP packets with the appropriate RTP stream.