906057 - Delay on WebRTC calls

Status: RESOLVED INCOMPLETE

(Core :: WebRTC: Audio/Video, defect)

Description

[Ada [:adalucinet]]

While testing in the following environments - https://wiki.mozilla.org/Releases/Firefox_23/Test_Plan#WebRTC - we found this delay issue:

Firefox 23.0.1 on Windows 7 (caller) to Firefox 23.0.1 on Windows 7 (callee)
Caller: - 4 sec delay; increases during conversation
Callee: - At the moment of the call there are no delays. 4 sec delay after more then 5 min


Firefox 22.0 on Windows 7 (caller) to Firefox 23.0.1 on Windows XP (callee)
Caller: - 4 sec delay
Callee: - At the moment of the call there are no delays. 4 sec delay after more then 5 minutes


Chrome Stable on Windows 7 (caller) to Firefox 23.0.1 on Windows 8 (callee)
Caller: - 2 sec Delay
Callee: - All working perfectly on Windows 8.


Firefox 23.0.1 on Windows 7 (caller) to Firefox 22.0 on Mac OS X 10.8 (callee)
Caller: - OK
Callee: - 1.5 sec delay from beginning


Firefox 23.0.1 on Windows 7 (caller) to Chrome Stable on Ubuntu 12.10 (callee)
Caller: - No delay
Callee: - 1.5 sec delay from beginning, 3 sec delay after 5 minutes


Firefox 22.0 on Windows XP (caller) to Firefox 23.0.1 on Windows 7 (callee)
Caller: - 5 sec delay
Callee: - 5 sec delay


Firefox 23.0.1 on Windows XP (caller) to Firefox 22.0 on Windows XP (callee)
Caller: - 4 sec Delay
Callee: - OK

Firefox 23.0.1 on Windows XP (caller) to Chrome Stable on Windows 8 (callee)
Caller: - 8 sec delay
Callee: - OK


Chrome Stable on Windows XP (caller) to Firefox 23.0.1 on Mac OS X 10.8 (callee)
Caller: - OK
Callee: - OK


Firefox 22.0 on Windows XP (caller) to Firefox 23.0.1 on Ubuntu 13.04 (callee)
Caller: - 2 sec delay
Callee: - OK


Chrome Stable on Windows 8 (caller) to Firefox 23.0.1 on Windows 7 (callee)
Caller: - OK
Callee: - OK


Firefox 23.0.1 on Windows 8 (caller) to Chrome Stable on Windows XP (callee)
Caller: - 6 sec Delay
Callee: - OK


Firefox 22.0 on Windows 8 (caller) to Firefox 23.0.1 on Windows 8 (callee)
Caller: - 6 sec delay
Callee: - OK


Firefox 23.0.1 on Windows 8 (caller) to Firefox 22.0 on Windows 8 (callee)
Caller: - 3-4 sec delay
Callee: - OK


Firefox 22.0 on Windows 8 (caller) to Firefox 23.0.1 on Mac OS X 10.8 (callee)
Caller: - 2 sec delay
Callee: - OK


Chrome Stable on Windows 8 (caller) to Firefox 23.0.1 on Ubuntu 13.04 (callee)
Caller: - OK
Callee: - OK


Firefox 23.0.1 on Mac OSX 10.8 (caller) to Firefox 22.0 on Windows 7 (callee)
Caller: - 3-4 sec delay after 5 min + jerky sound
Callee: - Low quality image


Chrome Stable on Mac OSX 10.8 (caller) to Firefox 23.0.1 on Windows XP (callee)
Caller: - delay increases after 5 min (5 min - 1 sec;  10 min -2 sec)
Callee: - OK


Firefox 22.0 on Mac OSX 10.8 (caller) to Firefox 23.0.1 on Windows 8 (callee)
Caller: - delay 1 sec after 5 min
Callee: - OK


Firefox 23.0.1 on Mac OSX 10.8 (caller) to Chrome Stable on Mac OS X 10.8 (callee)
Caller: - delay 1 sec after 5 min
Callee: - OK


Firefox 23.0.1 on Mac OSX 10.8 (caller) to Firefox 22.0 on Ubuntu 13.04 (callee) 
Caller: - delay 1,5 - 2 sec after 5 min + jerky sound
Callee: - Low quality image


Firefox 23.0.1 on Ubuntu 12.04 (caller) to Chrome Stable on Windows 7 (callee)
Caller: - OK
Callee: - OK


Firefox 22.0 on Ubuntu 12.04 (caller) to Firefox 23.0.1 on Windows XP (callee)
Caller: - 1sec delay after 5 min
Callee: - OK


Chrome Stable on Ubuntu 12.04 (caller) to Firefox 23.0.1 on Windows 8 (callee)
Caller: - OK
Callee: - OK


Firefox 23.0.1 on Ubuntu 12.04 (caller) to Firefox 22.0 on Mac OS X 10.8 (callee)
Caller: - 2 sec delay after 5 min
Callee: - OK


Firefox 23.0.1 on Ubuntu 12.04 (caller) to Chrome Stable on Ubuntu 13.04 (callee)
Caller: - OK
Callee: - OK

Comment 1

[Eric Rescorla (:ekr)]

Can you clarify this a bit: when you say "caller" do you mean media from the callee to the caller?

Comment 2

[Bogdan Maris, Desktop QA]

Just to clarify: In our testing CALLER is the person who entered first on apprtc.webrtc.org and gives a link to the CALLEE to join the conference.

The delays are present on Firefox 23.0 and older builds see bug 886886 so it happened before the backout in 23.0.1 as well.

Comment 3

[Eric Rescorla (:ekr)]

Bogdan,

In apprtc, the caller (i.e., he who calls CreateOffer()) is actually the second person into the room. However, that doesn't answer my question, I don't think. When you say that X has delay, do you mean that the media X is sending is delayed or that the media that X is receiving is delayed?

Comment 4

[Bogdan Maris, Desktop QA]

(In reply to Eric Rescorla (:ekr) from comment #3)
> Bogdan,
> 
> When you say that X has delay, do you mean that the media X is
> sending is delayed or that the media that X is receiving is delayed?

The audio that X is receiving is delayed.
Eg: 

> Firefox 23.0.1 on Ubuntu 12.04 (caller) to Firefox 22.0 on Mac OS X 10.8 (callee)
> Caller: - 2 sec delay after 5 min
> Callee: - OK

In this case the caller receives the audio from callee with delay.
If there is need for more clarification please let me know.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

This would be useful to retest using Fx25

Comment 6

[Bogdan Maris, Desktop QA]

We encountered some delays during our testing on Firefox 27 beta 7 using https://apprtc.webrtc.org/ on a call between:

Caller: Firefox 23.0 on Windows 7	 
Callee: Firefox 27.0b7 on Windows XP - (1 sec delay)
__________________________________________________
Caller: Firefox 25.0 on Windows XP - (1 sec delay)
Callee: Firefox 27.0b7 on Windows 7
____________________________________________________
Caller: Firefox 27.0b7 on Windows XP - (3 sec delay)
Callee: Firefox 24.0 on Windows XP - (1 sec delay)
_________________________________________________________________
Caller: Firefox 23.0 on Windows XP - (2 sec delay + burble sound)
Callee: Firefox 27.0b7 on Ubuntu 13.04

Comment 7

[u279076]

Randell, what further information do you need to investigate this?

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

Delay against 23 or 24 isn't very interesting (any more) as a major improvement to delay (in particular delay buildup) was landed in 25.  In general, delay against earlier than current_production_release minus 1 (so currently 25) isn't that large a concern.

That said, mostly for the one case where 25 is involved, we need more info in a report of delay:

Is the delay static or increasing?  Does it start from the beginning of the call?  Is it repeatable?  Headsets or not? Network: same lan or not?  Does the traffic leave the local lan or not?  If it does, does it use TURN or not?  (admittedly this is hard to determine currently - the logs don't show it well, though in 29 the about:webrtc page will let users know more about it).

CPU use in-call on each end?  (And number of cores/speed, since you can be flat-out at 25% use on a 4-core machine, depending on how many threads are being used.) (My core i7 laptop will get delay buildup and quality issues when I "call" myself because it will overheat (bad heat paste) and drop from 2.4GHz to 800MHz and fall behind, though CPU use was around 70%.)

"burble sound" - can this be recorded?  Repeatable?  

Any delay in 27b7 to 27b7?

We will be adding delay measurements to about:webrtc going forward.

Comment 9

[Bogdan Maris, Desktop QA]

(In reply to Randell Jesup [:jesup] from comment #8)

> Is the delay static or increasing?  
The delay is increasing as the time goes on, but after 5-10 minutes the delay is consistent. In a call of 30 minutes long, I got 1 sec delay.

> Does it start from the beginning of the call?  
At the beginning there is no delay at all.

> Is it repeatable?  
Yes

> Headsets or not? 
A4tech and Apple headsets.

> Network: same lan or not?  
Yes, but on XP I used a proxy (TunnelBear - Germany channel).

> Does the traffic leave the local lan or not?  
Yes

> If it does, does it use TURN or not?  (admittedly this is hard to determine currently - the logs don't show it well, though in 29 the about:webrtc page will let users know more about it).
I captured a dump using Wireshark but did not see anything related to TURN. (frankly I don`t now where exactly to look)
Attachment: http://goo.gl/1m8YYy

> CPU use in-call on each end? (And number of cores/speed, since you can be
> flat-out at 25% use on a 4-core machine, depending on how many threads are
> being used.) (My core i7 laptop will get delay buildup and quality issues
> when I "call" myself because it will overheat (bad heat paste) and drop from
> 2.4GHz to 800MHz and fall behind, though CPU use was around 70%.)
Windows 7 x32:
AMD FX 8320, 8GB RAM.
CPU - 7% - Memory 200MB

Windows XP x32:
Intel Core Duo E7500, 4GB RAM.
CPU 25-30% - Memory 150MB

> "burble sound" - can this be recorded?  Repeatable?  
I get the same sound as https://bug935617.bugzilla.mozilla.org/attachment.cgi?id=828136 from bug 935617 and yes It is repeatable.

> Any delay in 27b7 to 27b7?
Same behavior using the same OS`s. The delay is increasing as the time goes on, but after 5-10 minutes the delay is consistent. In a call of 30 minutes long, I got 1 sec delay.

Comment 10

[Maire Reavy [:mreavy]]

This data is no longer actionable given the number of changes since then,