1090865 - WebRTC Video / Peer Data Connection latencies start OK, quickly become unreasonable

Status: RESOLVED DUPLICATE of bug 1087605

(Core :: WebRTC: Networking, defect)

Description

[Mark Goodwin [:mgoodwin]]

I was playing around with WebRTC on Desktop and Mobile (fennec and FxOS) - I found a problem that, though intermittent, happens fairly frequently (on Fennec at least).

Issue:
* On connection, video and data connection latencies start low and quickly degrade (round trip up to 300 seconds has been observed).
* This has been observed (frequently) on Fennec (nightly 2014-10-28, Samsung Galaxy Nexus, Android 4.3) and possibly (I think, once) on Firefox OS (2.1 on kk base image, non-engineering build. flame)

Steps to reproduce:
* I adapted the WebRTC codelab to provide a test. You can get this from this branch of my fork of the WebRTC Codelab: https://github.com/mozmark/WebRTC-codelab/tree/fennectest
** You'll need node.js with node-static and socket.io modules installed
** This application sends messages periodically from the initiator, to whatever connects to it. The connecting peer echoes the messages back. The initiator sends a timing to console.log
* Run the server (cd complete/step7; node server.js)
* Open a desktop browser, open the devtools, navigate to the test server (http:<server>:2013)
* Open fennec, navigate to the test server (http://<server>:2013)

For each test you'll need to connect the peers in this order.

Some of the time, the connection is established, the video stays smooth and the latency stays low.

Intermittently (frequently, not all of the time), the latency on the video stream will increase, the round trip timings in the console will increase. The device can become unusable when this happens (I wonder if there's a related leak?).

The issue seems to happen less frequently if the desktop does not share a video stream with the mobile device. I can't back this up with numbers.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

There are two points of interest:

One is to check the thread priorities; bug 1087605 has a patch that just landed.  It *may* only affect B2G, but Sotaro says it should affect Android as well.  This was causing the MainThread and a bunch of other threads to get bumped to pri -14 on B2G.

Another is the bandwidth adaptation code.  Saving webrtc internal logs (NSPR_LOG_MODULES=webrtc_trace:65535,signaling:5 WEBRTC_TRACE_FILE=whatever logs) may help disagnose this.  You can send the logs to the destination 'nspr' (meaning NSPR logs), but that can cause problems if it's being shipped over ADB (too much data).  On mobile wireless, I would imagine link issues might occasionally cause delay (RTT), but a) it should go away, b) if it's a link bandwidth issue for long enough, the adaptation code will dial down the bandwidth used to drain the network drivers and router buffers of delay buildup.

A Wireshark trace from the desktop (wired, please) will help as well.

Gian-Carlo should be able to comment here as well about WebRTC on Fennec

Comment 2

[Mark Goodwin [:mgoodwin]]

(In reply to Randell Jesup [:jesup] from comment #1)
> There are two points of interest:

I've got some caps / logs; I don't want to have to analyse them for private data or sanitize them so I've put them behind persona auth (PM me on IRC let me know what identity to use, if not your mozilla one).

https://people.mozilla.org/~mgoodwin/bugdata/logs_and_cap.tar.gz

I have a suspicion that logging all the things is affecting the test; I never see a smooth / low latency connection with logging enabled on android.

Comment 3

[Gian-Carlo Pascutto [:gcp]]

(In reply to Randell Jesup [:jesup] from comment #1)
> There are two points of interest:
> 
> One is to check the thread priorities; bug 1087605 has a patch that just
> landed.  It *may* only affect B2G, but Sotaro says it should affect Android
> as well.  This was causing the MainThread and a bunch of other threads to
> get bumped to pri -14 on B2G.

./ps -A -F -T -w -l | grep fennec_morbo 
with patch
http://pastebin.mozilla.org/7110076

with e82273c162bf backed out
http://pastebin.mozilla.org/7110113

Sotaro was right, I and the Android docs are wrong: it is possible for Fennec to raise it's own priority, so after that patch a number of Firefox threads now have different priorities.

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

Can we retest now since the patch should have landed everywhere?

Comment 5

[Mark Goodwin [:mgoodwin]]

I've tried for 10 minutes or so to reproduce the issue and am unable to; this is certainly a vast improvement.

Thanks!