Sudden audio delay while in a good call

RESOLVED DUPLICATE of bug 884365

Status

()

Core
WebRTC: Audio/Video
RESOLVED DUPLICATE of bug 884365
4 years ago
4 years ago

People

(Reporter: mreavy, Assigned: jesup)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Report from Mihai Morar, QA:

Huge audio delay while testing on FF 24b1 on following URLs :

1:1 call across 2 different machines on same network:
http://freshtilledsoil.com/the-future-of-web/webrtc-video/

3-4 call across different machines from different networks
http://codassium.com/

In both cases in first 3-5 minutes everything was fine, after that a huge audio delay appeared (about 15-20 sec)

Mihai -- Can you report what Operating Systems you are testing on?
(Reporter)

Comment 1

4 years ago
Mihai -- Did you see this bug when Firefox 24 was in Aurora, or is it new with the first Beta build? (I'd like help nailing down a regression window.)  

When you see this delay, using headphones, can the tester showing the delay unmute his/her self image (right click the self image and select "unmute") and check if his/her outgoing audio is delayed?  (This test will help us determine if the delay is on the sender or receive side.)  

Also, since audio issues can be very OS specific, reporting the OS used on each test is important and helpful to tracking down the issue.  Thanks.
Flags: needinfo?(mihai.morar)
We we're on Windows 7 x64 all.
Flags: needinfo?(mihai.morar)
(In reply to Maire Reavy [:mreavy] from comment #1)
> Mihai -- Did you see this bug when Firefox 24 was in Aurora, or is it new
> with the first Beta build? (I'd like help nailing down a regression window.)
I didn't test Aurora 24, only Beta 23 and only 1:1 call, which worked well. 

> When you see this delay, using headphones, can the tester showing the delay
> unmute his/her self image (right click the self image and select "unmute")
> and check if his/her outgoing audio is delayed?  (This test will help us
> determine if the delay is on the sender or receive side.)  
Not sure I'll test this case ASAP.

> Also, since audio issues can be very OS specific, reporting the OS used on
> each test is important and helpful to tracking down the issue.  Thanks.
Only Win 7 x64 but I'll test on Ubuntu and Mac too.
(Reporter)

Comment 4

4 years ago
(In reply to Mihai Morar, QA (:MihaiMorar) [ PTO 08/09 - 08/26] from comment #3)
> (In reply to Maire Reavy [:mreavy] from comment #1)
> > Mihai -- Did you see this bug when Firefox 24 was in Aurora, or is it new
> > with the first Beta build? (I'd like help nailing down a regression window.)
> I didn't test Aurora 24, only Beta 23 and only 1:1 call, which worked well. 

Can you test the older Firefox 24 builds (when Firefox was in Aurora)?  Start with the oldest Firefox 24 build you can find and see if the problem is there.  If not, then we have the start of a regression window.
Tested on FF 24b1:

1. Using Win 7 and Ubuntu 13.04 across 2 different machines on different networks:
Results: Huge audio delay and interruptions even from the begining of the conversation.

2. Using Win 7 across 2 different machines on different networks:
Results: Everything was fine for about 2-3 minutes. After that a couples of interruptions and audio delays have appeared.

Tested on first Aurora 24 build: 2013/06/30

1. Using Win 7 across 2 different machines on different networks:
Results: Everything was fine during the entire conversation.

2. Using Win 7 and Ubuntu 13.04 across 2 different machines on different networks:
Results: Everything was fine for about 2-3 minutes. After that a couples of interruptions and audio delays have appeared.

We'd muted/unmuted the machines but there we're same results as mentioned above. Mute/Unmute options do not influence the conversation in any way.
(Reporter)

Comment 6

4 years ago
(In reply to Mihai Morar, QA (:MihaiMorar) [ PTO 08/09 - 08/26] from comment #5)
> Tested on FF 24b1:
> 
> 1. Using Win 7 and Ubuntu 13.04 across 2 different machines on different
> networks:
> Results: Huge audio delay and interruptions even from the begining of the
> conversation.

The interruptions are likely bug 901527.

> 
> 2. Using Win 7 across 2 different machines on different networks:
> Results: Everything was fine for about 2-3 minutes. After that a couples of
> interruptions and audio delays have appeared.

Can you describe the interruptions and delays?  How long did the interruptions last?  How long were the delays?  Did the delays last or catch up?  Were the delays and interruptions on both sides of the call or just one?

> 
> Tested on first Aurora 24 build: 2013/06/30
> 
> 1. Using Win 7 across 2 different machines on different networks:
> Results: Everything was fine during the entire conversation.

Can you repeat this test a few more times?  I want to make sure that we were fine on the 6/30 build because delay may not show up on every call even though the build has a bug like this in it.

> 
> 2. Using Win 7 and Ubuntu 13.04 across 2 different machines on different
> networks:
> Results: Everything was fine for about 2-3 minutes. After that a couples of
> interruptions and audio delays have appeared.

Can you describe the interruptions and delays?  How frequent were the interruptions? Was video affected also?   How long were the delays?  Did the delays persist or did we "catch up"?  Were the delays and interruptions on both sides of the call or just one?  How did these results with the first Aurora build compare to the test with FF24b1?

> 
> We'd muted/unmuted the machines but there we're same results as mentioned
> above. Mute/Unmute options do not influence the conversation in any way.

No, they won't influence the conversation.  What I need you to do is listen to your own audio when you unmute.  When you unmute and hear your audio, does your audio sound delayed to you?  This should be done by the person whose audio sounds delayed to the other side.
Flags: needinfo?(mihai.morar)
Sure, I'll make these investigation tomorrow morning.
Flags: needinfo?(mihai.morar)
(Assignee)

Comment 8

4 years ago
Some data to capture:

I'm seeing two types of latency slips (on windows), when they happen:

1) "slow" steady slip (0.5-3 seconds/min).  In SysInternals' Process Explorer, if I view the graph of that firefox process's performance, I see a slow rise in memory use correlated with delay increase (as data is buffered in MediaStreamGraph).  CPU use is steady and not extreme (20-30% on my W520)

2) "Bursty" slip associated with high CPU use (55-65%).  The slip is still continuous, but MUCH faster (as much as 15 seconds in 35-40 seconds).  This may continue, or I've seen a weird pattern where it happens for 1:05 to 1:15 (one minute and a bit), starting every 4:30 to 4:45.    During this, there's increased system CPU time, though still low.  I'm not sure I've ever seen an instantaneous slip (sudden without-warning large latency change).  Note these bursts continued (sometimes) without any other browser running (see below for context).

If we do see an "instantaneous" slip, or nearly so, that will be interesting.  There still could be two different bugs here.

Note: Both of these are FAR more likely to occur when another instance of Firefox is running (in my case an Aurora with several hundred tabs, but only a few open - using 4-8% CPU, and a gig or two (on a 16GB system with 8GB memory free).

(It's useful to use Process Explorer to watch CPU and memory of the specific  Firefox process; all slips are directly visible in the memory use graph as a ramp up at a specific slope relative to the rate of slip (since MSG is buffering).)

So cases with and without another browser running are interesting - perhaps they're contending for a shared system lock/resource.  I should not I've *only* seen this on Windows - reproducing on another platform would be interesting, and useful in debugging.

My regression testing (by downloading nightlies) for constant latency increase pointed at July 27th or 28th, but these may have been partially impacted by running a specific other browser while doing my tests (idle, but there).  Overnight it got worse, but when I quit Aurora, my Nightly of July 29th suddenly stopped having a constantly building delay.

I have speculations as to the causes, but I'm not ready to commit to them yet, or to block people from thinking about other possible solutions.  If you *must* know now, contact me.
I did not have any luck attemting to narrow this down today. Mihai, please see if you or someone on your team can find a regression window for this overnight tonight.
QA Contact: mihai.morar
I have tested on following builds across 2 or more different machines on same and different networks too.

Nightly from: 2013.07.01, 2013.07.26, 2013.07.29, 2013.08.08

For all these builds mentioned above there was a huge audio delay (about 20-30sec)

While testing on same network, in the begining it was fine, no audio delay or interruptions for about 2-3 minutes. After that both audio delays and interruptions have appeared.

While testing on different networks, audio delay and interruptions appeared from the begining of the conversation.

For all these testes mentioned about video was displayed in real time without any delay, hang or interruption. Also my CPU usage was incresed to 80-90% for all tests mentioned above.
Can you identify a version that *doesn'* have a delay? Maybe go back to 6/1?
(In reply to Eric Rescorla (:ekr) from comment #11)
> Can you identify a version that *doesn'* have a delay? Maybe go back to 6/1?

As I discussed with Randell on IRC, I'd tested Aurora 24 from 6/30 and Aurora 23 from 6/1:

1:1 call across 2 different machines from different networks:

1.For 6/30 Aurora 24 build:
a) With another browser opened: CPU Usage: 30-35%
b) With no other browsers opened: CPU Usage: 20-25%

2.For 6/1 Aurora 23 build:
a) With another browser opened: CPU Usage: 30-35%
b) With no other browsers opened: CPU Usage: 20-25%

The problem appears when a third person is joining the conversation. In that moment CPU Usage increses to about 50-60% in 100% cases for both builds mentioned above.

My Machine Config: Intel(R) Core(TM) i5-3470 CPU @ 3.40GHz  3.6GHz with 4GB DDR 3 1600 and Intel HD Graphics 2500.

OS Used: Windows 7 x64
(In reply to Eric Rescorla (:ekr) from comment #11)
> Can you identify a version that *doesn'* have a delay? Maybe go back to 6/1?

In 6/1 Aurora where was a small delay (about 1 sec) in 1:1 call after a couple of minutes. When the third person joined the conversation, a huge delay had appeared (about 20-30 sec) This is the delay I was talking about in Comment 10.
Can you clarify that this only happens with 3-person calls? If so, that seems like
something we haven't really tested yet, so it's good if it works, but not really a regression if not.
On most builds that I tested, this issue is happening 100% cases on 3+ person calls. In 1:1 calls as i said in Bug 886886 video call works fine (just some small delays less than 1sec). The only URL that is causing huge delay on 1:1 call is:

http://freshtilledsoil.com/the-future-of-web/webrtc-video/
(Assignee)

Comment 16

4 years ago
(In reply to Mihai Morar, QA (:MihaiMorar) [ PTO 08/09 - 08/26] from comment #12)
> (In reply to Eric Rescorla (:ekr) from comment #11)
> > Can you identify a version that *doesn'* have a delay? Maybe go back to 6/1?
> 
> As I discussed with Randell on IRC, I'd tested Aurora 24 from 6/30 and
> Aurora 23 from 6/1:
> 
> 1:1 call across 2 different machines from different networks:
> 
> 1.For 6/30 Aurora 24 build:
> a) With another browser opened: CPU Usage: 30-35%
> b) With no other browsers opened: CPU Usage: 20-25%
> 
> 2.For 6/1 Aurora 23 build:
> a) With another browser opened: CPU Usage: 30-35%
> b) With no other browsers opened: CPU Usage: 20-25%
> 
> The problem appears when a third person is joining the conversation. In that
> moment CPU Usage increses to about 50-60% in 100% cases for both builds
> mentioned above.

3-way calls weren't the issue (though it's interesting).  Our focus for now is 1:1 calling.

The question is does delay build up (slowly or quickly) with 1:1 calls with a single browser running, and does it build up in 1:1 calls with two browsers running.  (and by "build up", I mean increase compared to the very start of the call - any added delay will tend to 'stick', and delay should be well under 1 second - preferably under 200ms.)
(Assignee)

Comment 17

4 years ago
And by "two browsers running" I mean a second instance of firefox running (different profile), but idle.  It doesn't have to be the same build.
(In reply to Randell Jesup [:jesup] from comment #16)

> The question is does delay build up (slowly or quickly) with 1:1 calls with
> a single browser running, and does it build up in 1:1 calls with two
> browsers running.  (and by "build up", I mean increase compared to the very
> start of the call - any added delay will tend to 'stick', and delay should
> be well under 1 second - preferably under 200ms.)

In first 2-3 minutes everything is fine, no delay or interruptions, after that suddenly they both appear but not in same time. Sometimes delay appear first, sometime the interruptions, and then they are both present until you close your conversation. This are happening in both cases (haveing or not a second browser opened) 

I don't think the another browser opened or not is causing this, because I'd tested on Chrome, having both Firefox and Opera opened, both with heavy flash content in used until my CPU usage get to about 95% and it worked fine.
(In reply to Maire Reavy [:mreavy] from comment #0)
> Report from Mihai Morar, QA:
> 
> Huge audio delay while testing on FF 24b1 on following URLs :
> 
> 1:1 call across 2 different machines on same network:
> http://freshtilledsoil.com/the-future-of-web/webrtc-video/
> 
> 3-4 call across different machines from different networks
> http://codassium.com/
> 
> In both cases in first 3-5 minutes everything was fine, after that a huge
> audio delay appeared (about 15-20 sec)
> 
> Mihai -- Can you report what Operating Systems you are testing on?

In both cases I can see a great improvement on Latest Nightly BuildID: 20130903030201 .No video/audio delay occured. Based on this I can say it was fixed but not sure if Bug 901527 fixed this issue or not.
(Assignee)

Comment 20

4 years ago
I suspect strongly that bug 884365 is masking the problem (or if you prefer, solving it for calls).  If you were to un-mute your local image (possible in some apps, like herokuapp), you may find you still get delays in on local destinations if there's an overload or other "slip" in the MediaStreamGraph.  So this wasn't fixed by bug 901527 (and this wasn't caused by landing bug 886886).  This likely existed before bug 886886 landed and caused bug 901527 (and we'd had reports of audio delay buildups).

So I think this can be dupped against bug 884365, and also helps us verify that fix
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 884365
You need to log in before you can comment on or make changes to this bug.