904754 - Quality of calls in video+audio between Android ↔ Android is unusable with Mozilla wifi involved

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect)

Description

[Aaron Train [:aaronmt]]

Possibly bug 901831? On trunk (mozilla-central) 08/13 builds on Android (4.3/4.2) both Jason and I are witnessing a very bad regression in the quality of audio on both ends. There is a large build-up of audio that plays all at once, either that or audio would 'chopped' up by each syllable as best to describe it.

Comment 1

[Jason Smith [:jsmith]]

Putting tracking-fennec here because the WebRTC audio performance for FxAndroid -> FxAndroid calls right now is incredibly difficult, if not unusable, to use right now.

Comment 2

[Maire Reavy [:mreavy]]

(In reply to Aaron Train [:aaronmt] from comment #0)
> Possibly bug 901831? On trunk (mozilla-central) 08/13 builds on Android
> (4.3/4.2) both Jason and I are witnessing a very bad regression in the
> quality of audio on both ends. There is a large build-up of audio that plays
> all at once, either that or audio would 'chopped' up by each syllable as
> best to describe it.

What build are you comparing today's Android Nightly (8/13) to?   When was the last good Android build?  I'm trying to nail down a regression window.

Comment 3

[Jason Smith [:jsmith]]

Aaron - Can we get someone from softvision to get a regression range here?

Comment 4

[Jason Smith [:jsmith]]

This looks like we're seeing bug 901527 here, but this is after that patch landed. That implies the patches that landed on 24, 25, and 26 don't work on Android.

Comment 5

[Jason Smith [:jsmith]]

Spoke with tchung about this - Mobile QA views this as a blocker to prevent shipping WebRTC on Fx24 for Android. The minimal acceptance criteria for ship was that:

* Bare minimum: 1 min call works reliably over video/audio over multiple runs
* Recommended: 5 min call works reliably over video/audio over multiple runs

We aren't hitting the bare minimum due to this bug.

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

I did a bunch of tests with an S4 and a mac (on the same residential wifi network).

first, the 901527 'burble' is definitely gone in nightly on Android.

Second, I found that audio quality was generally good excluding echo (I was in speakerphone mode, so the mac heard echo), unless I started actively moving around, in which case I got some audio dropouts (0.5 to 2 seconds) with distinctive Opus/PLC (packet loss concealment) audio artifacts at the edges of the dropout.  Motion affecting it implies network issues - in particular, the jitter buffer may have been running very low depths and underflowed if the WiFi network buffered some packets, or it simply lost a bunch.  More analysis would be needed of the mobile wifi case, and debugs.

As Jason was doing his tests on the Mozilla/MV Wifi network, there may be some challenging network issues.  In addition, the network configuration and firewall settings may change significantly from day to day, or even within a day.

Working well in a challenging/complex WiFi environment would be a goal, but no specific work has been done on it yet.  Laptops generally have higher power and much better/more-diverse antennas, and mobile devices (phones/tablets) are more likely to use power-saving features that may cause temporary buffering.  

To test for a regression on nightly earlier than bug 901527 landed in nightly, you'll need to go back before bug 886886 landed (7/22 IIRC i.e. there's a window from bug 886886 landing to bug 901527 landing where the quality will be bad on all platforms).

It would be very interesting to know if it's an actual regression, compared in the same network environment. (preferably a -> b -> a -> b testing since the network environment can change over time in a corporate setting).

A log of a debug build showing the dropouts would be useful (if there's a way to capture a network trace on the Android side, even better (in addition to logs).

Comment 7

[Jason Smith [:jsmith]]

(In reply to Randell Jesup [:jesup] from comment #6)
> I did a bunch of tests with an S4 and a mac (on the same residential wifi
> network).
> 
> first, the 901527 'burble' is definitely gone in nightly on Android.

The burble is probably gone in the residential wifi case because it's a low traffic network. Here the testing was done with a high traffic wifi office network (Mozilla wifi - Mountain View) to a low traffic wifi network (home wifi - Toronto). 

> 
> Second, I found that audio quality was generally good excluding echo (I was
> in speakerphone mode, so the mac heard echo), unless I started actively
> moving around, in which case I got some audio dropouts (0.5 to 2 seconds)
> with distinctive Opus/PLC (packet loss concealment) audio artifacts at the
> edges of the dropout.  Motion affecting it implies network issues - in
> particular, the jitter buffer may have been running very low depths and
> underflowed if the WiFi network buffered some packets, or it simply lost a
> bunch.  More analysis would be needed of the mobile wifi case, and debugs.
> 
> As Jason was doing his tests on the Mozilla/MV Wifi network, there may be
> some challenging network issues.  In addition, the network configuration and
> firewall settings may change significantly from day to day, or even within a
> day.

That might imply why we saw occasional cases where the latency was acceptable, but most others were barely usable at all on the Mozilla wifi. That does imply that we're likely failing to meet the developer use case of if the developer is building a WebRTC application on an office network.

> 
> Working well in a challenging/complex WiFi environment would be a goal, but
> no specific work has been done on it yet.  Laptops generally have higher
> power and much better/more-diverse antennas, and mobile devices
> (phones/tablets) are more likely to use power-saving features that may cause
> temporary buffering.  

That sounds like a problem. Product indicated that the target shipping goal needs to meet developer quality. We know that developers work on office networks. If we don't support the office network use case, then exactly how are web developers supposed to build and test their own WebRTC applications.

> 
> To test for a regression on nightly earlier than bug 901527 landed in
> nightly, you'll need to go back before bug 886886 landed (7/22 IIRC i.e.
> there's a window from bug 886886 landing to bug 901527 landing where the
> quality will be bad on all platforms).
> 
> It would be very interesting to know if it's an actual regression, compared
> in the same network environment. (preferably a -> b -> a -> b testing since
> the network environment can change over time in a corporate setting).

I'll try to look into this today.

Comment 8

[Aaron Train [:aaronmt]]

I can't reproduce this locally using two devices on my same Wi-Fi Wireless-N home network on trunk (08/15). Jason, I used http://mysecondwebrtc.appspot.com/ and connected two devices and all audio was audible without delay.

Comment 9

[Jason Smith [:jsmith]]

(In reply to Aaron Train [:aaronmt] from comment #8)
> I can't reproduce this locally using two devices on my same Wi-Fi Wireless-N
> home network on trunk (08/15). Jason, I used
> http://mysecondwebrtc.appspot.com/ and connected two devices and all audio
> was audible without delay.

That's not surprising. The issue being discussed above is talking about the case when one of the wifi networks involved is high traffic (Mozilla Mountain View wifi). A home wifi network would be considered a low traffic network.

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

> --- Comment #7 from Jason Smith [:jsmith] <[e-mail]> 
> (In reply to Randell Jesup [:jesup] from comment #6)
>> I did a bunch of tests with an S4 and a mac (on the same residential wifi
>> network).
>>
>> first, the 901527 'burble' is definitely gone in nightly on Android.
>
> The burble is probably gone in the residential wifi case because it's a low
> traffic network. Here the testing was done with a high traffic wifi office
> network (Mozilla wifi - Mountain View) to a low traffic wifi network (home wifi
> - Toronto). 

To be clear - bug 901527 is fixed, including in Android.  Network dropout and packet loss concealment may sound similar, depending on how bad it is, but it is not the same bug.  901527 would cause constant failure in all situations, residential or otherwise, and would not be affected by the network.  Since I could talk for minutes with no significant loss (while stationary), this tells me 901527 is indeed fixed on Android.

You are hearing issues; the question is the source of those, if it's a regression, and how bad/prevalent they are.

>>
>> Second, I found that audio quality was generally good excluding echo (I was
>> in speakerphone mode, so the mac heard echo), unless I started actively
>> moving around, in which case I got some audio dropouts (0.5 to 2 seconds)
>> with distinctive Opus/PLC (packet loss concealment) audio artifacts at the
>> edges of the dropout.  Motion affecting it implies network issues - in
>> particular, the jitter buffer may have been running very low depths and
>> underflowed if the WiFi network buffered some packets, or it simply lost a
>> bunch.  More analysis would be needed of the mobile wifi case, and debugs.
>>
>> As Jason was doing his tests on the Mozilla/MV Wifi network, there may be
>> some challenging network issues.  In addition, the network configuration and
>> firewall settings may change significantly from day to day, or even within a
>> day.
>
> That might imply why we saw occasional cases where the latency was acceptable,
> but most others were barely usable at all on the Mozilla wifi. That does imply
> that we're likely failing to meet the developer use case of if the developer is
> building a WebRTC application on an office network.

One thing I noted in your first paragraph was that the actual test was MV WiFi to Toronto residential WiFi, which hadn't been mentioned before.  Was this testing in mobile->desktop in either direction?  were the same issues seen in both directions?  Was there any verification that a desktop->desktop connection across the same networks?

I.e. network performance (especially across the wild internet and in a corporate network that can have wildly varying performance) needs to be considered and controlled for; comparing to desktop is a good first step, as is comparing to a isolated/idle lan/wifi.  It doesn't mean we don't work over the internet, but a given internet connection might not support a call well if saturated or may be over-saturated in bursts, or there may be wifi issues.  (Ex: Try a call across a router that resets due to heat every 1-2 minutes.... I've done it; it's not fun.)

>
>>
>> Working well in a challenging/complex WiFi environment would be a goal, but
>> no specific work has been done on it yet.  Laptops generally have higher
>> power and much better/more-diverse antennas, and mobile devices
>> (phones/tablets) are more likely to use power-saving features that may cause
>> temporary buffering.  
>
> That sounds like a problem. Product indicated that the target shipping goal
> needs to meet developer quality. We know that developers work on office
> networks. If we don't support the office network use case, then exactly how are
> web developers supposed to build and test their own WebRTC applications.

I'm not saying we don't support office environments; just that they're different and that may impact tests and repeatability of tests, and it needs to be measured/controlled-for/considered.  Thus far, we haven't done anything specific for corporate networks or specific (network-wise) for Android.

>> To test for a regression on nightly earlier than bug 901527 landed in
>> nightly, you'll need to go back before bug 886886 landed (7/22 IIRC i.e.
>> there's a window from bug 886886 landing to bug 901527 landing where the
>> quality will be bad on all platforms).
>>
>> It would be very interesting to know if it's an actual regression, compared
>> in the same network environment. (preferably a -> b -> a -> b testing since
>> the network environment can change over time in a corporate setting).
>
> I'll try to look into this today.

Thanks!  And I'd love to see some actual TCPdump traces (on the phone!) and/or logcat dumps when this happens (even for a non-debug build, but a debug build would be more interesting); perhaps signaling logs to start, and/or webrtc_trace:65535 logs (I don't know how to get them into logcat; they go to a file normally)

Comment 11

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Jason Smith [:jsmith] from comment #9)
> (In reply to Aaron Train [:aaronmt] from comment #8)
> > I can't reproduce this locally using two devices on my same Wi-Fi Wireless-N
> > home network on trunk (08/15). Jason, I used
> > http://mysecondwebrtc.appspot.com/ and connected two devices and all audio
> > was audible without delay.
> 
> That's not surprising. The issue being discussed above is talking about the
> case when one of the wifi networks involved is high traffic (Mozilla
> Mountain View wifi). A home wifi network would be considered a low traffic
> network.

Note that we only have indirect evidence it's related to the network, though Aaron's report would seem to reinforce mine.  We also didn't have a/b comparison to a desktop on the same network/AP immediately before or afterwards.  (Note: don't use 5GHz for a laptop and 2.4 for a android phone!  Can be very different, and 2.4 networks often have to run in 802.11b compatibility mode if there's a device in-range with only b support.  And more collisions, more "hidden node" problems, etc.)

Comment 12

[Jason Smith [:jsmith]]

(In reply to Randell Jesup [:jesup] from comment #10)
> 
> >>
> >> Second, I found that audio quality was generally good excluding echo (I was
> >> in speakerphone mode, so the mac heard echo), unless I started actively
> >> moving around, in which case I got some audio dropouts (0.5 to 2 seconds)
> >> with distinctive Opus/PLC (packet loss concealment) audio artifacts at the
> >> edges of the dropout.  Motion affecting it implies network issues - in
> >> particular, the jitter buffer may have been running very low depths and
> >> underflowed if the WiFi network buffered some packets, or it simply lost a
> >> bunch.  More analysis would be needed of the mobile wifi case, and debugs.
> >>
> >> As Jason was doing his tests on the Mozilla/MV Wifi network, there may be
> >> some challenging network issues.  In addition, the network configuration and
> >> firewall settings may change significantly from day to day, or even within a
> >> day.
> >
> > That might imply why we saw occasional cases where the latency was acceptable,
> > but most others were barely usable at all on the Mozilla wifi. That does imply
> > that we're likely failing to meet the developer use case of if the developer is
> > building a WebRTC application on an office network.
> 
> One thing I noted in your first paragraph was that the actual test was MV
> WiFi to Toronto residential WiFi, which hadn't been mentioned before.  Was
> this testing in mobile->desktop in either direction?  were the same issues
> seen in both directions?  Was there any verification that a desktop->desktop
> connection across the same networks?

Here's the results that came out of testing recently in relation to this.

AppRTC

* FxAndroid (Mozilla wifi) in Mountain View --> Desktop Firefox (Mozilla wifi) in Mountain View - Call established, very poor audio/video latency
* FxAndroid (Home wifi) in Toronto --> FxAndroid (Mozilla wifi) Mountain View - Call established, very poor audio/video latency
* Chrome Beta for Android (Mozilla wifi) in Mountain View --> FxAndroid (Home wifi) in Toronto - Call established, some audio/video latency (but not as bad as the above two)
* Desktop Firefox (Mozilla wifi) in Mountain View --> FxAndroid (Home wifi) in Toronto - Call failed to establish on first try, 2nd try established with low audio/video latency
* FxAndroid (Home wifi) in Toronto --> Chrome Desktop (Mozilla wifi) in Mountain View - Call established with low audio/video latency

We also tested against http://mysecondwebrtc.appspot.com/ as well for similar combinations above with the affected STUN server, which revealed similar results and confirmed that the RFC patch provides a net gain.

Comment 13

[Maire Reavy [:mreavy]]

Can you try audio-only WebRTC calls on Android and describe the audio quality/delay relative to the video+audio call tests you've already done?  Thanks.

Comment 14

[Jason Smith [:jsmith]]

(In reply to Maire Reavy [:mreavy] from comment #13)
> Can you try audio-only WebRTC calls on Android and describe the audio
> quality/delay relative to the video+audio call tests you've already done? 
> Thanks.

Note - This was tested with http://mysecondwebrtc.appspot.com/ on today's FxAndroid Nightly using Mozilla wifi.

Audio to audio call on FxAndroid --> FxAndroid on Mozilla wifi actually worked quite well. Latency was quite low - I could hear the other person not too long after the person spoke.

Video & Audio to Video & Audio call on FxAndroid --> FxAndroid on Mozilla wifi however hit the exact problems in this bug. Chopped up syllables are heard and the audio falls behind quite quickly in the call, rendering it unusable quite quickly.

Comment 15

[Jason Smith [:jsmith]]

Attachment: Top Output During Call - Galaxy Nexus, Mozilla wifi

Comment 16

[Jason Smith [:jsmith]]

Attachment: Logcat with Signaling NSPR Logs Included - Galaxy Nexus, Mozilla Wifi

Comment 17

[Jason Smith [:jsmith]]

I think the analysis above actually has concluded that this isn't a regression caused by bug 886886, so I'm pulling the regressionwindow-wanted keyword. I've got two sets of logs here, but still need trace logs. tcpdump logs were also requested, but the devices I'm using here are non-rooted.

Comment 19

[Jason Smith [:jsmith]]

Attachment: Logcat with WebRTC Trace Logs Enabled - Galaxy Nexus, Mozilla Wifi

Comment 20

[Mark Finkle (:mfinkle) (use needinfo?)]

I am hesitant to block on problems with Mozilla Wifi. It sucks in general.

Comment 21

[Jason Smith [:jsmith]]

(In reply to Mark Finkle (:mfinkle) from comment #20)
> I am hesitant to block on problems with Mozilla Wifi. It sucks in general.

Mozilla wifi worked fine on desktop when we shipped it for long running calls. It additionally works with audio only calls and video only calls with FxAndroid, so it's not a general wifi problem. The problem here is the call is unusable at all - it isn't usable for a small period of time a developer would need to build an app. Add on the fact that this is going block any useful QA testing as well in the Mountain View office in video/audio calls, unless we do all testing in Haxxor, which isn't really a realistic network to use (i.e. it's a low activity network).

I'm hesitant to push this out the door given that this indicates our network performance is not up to par in the mobile space to even be usuable under office network scenarios for a small period of time a developer would need. I'm even more concerned that makes testing in the Mountain View office very limited.

We could do more testing on other office networks as a point of comparison to see how widespread this problem here is to get a better assessment.

Comment 22

[Randell Jesup [:jesup] (needinfo me)]

Comparison to Android Chrome 29 (29.0.1547.59 - latest in Play Store) on two Galaxy S4's (i.e. hot new phones), residential Wifi with minor other traffic and no other AP's on-channel:

FF-FF (Nightly and Beta) using nightly-gupshup and apprtc: connected first time.  Video good.  Audio small delay on one side, one or two dropouts in one call.  Quite usable. 

Chrome-Chrome (29.0) using apptrc: Horrible problems getting them to connect.  On 10th try got a call.  Video pretty good.  No audio in one direction.  Chipmunk  audio in the other (sounded like 2x frequency).

Comment 23

[Jason Smith [:jsmith]]

(In reply to Randell Jesup [:jesup] from comment #22)
> Comparison to Android Chrome 29 (29.0.1547.59 - latest in Play Store) on two
> Galaxy S4's (i.e. hot new phones), residential Wifi with minor other traffic
> and no other AP's on-channel:
> 
> FF-FF (Nightly and Beta) using nightly-gupshup and apprtc: connected first
> time.  Video good.  Audio small delay on one side, one or two dropouts in
> one call.  Quite usable. 
> 
> Chrome-Chrome (29.0) using apptrc: Horrible problems getting them to
> connect.  On 10th try got a call.  Video pretty good.  No audio in one
> direction.  Chipmunk  audio in the other (sounded like 2x frequency).

I think we already know that this works fine on residential wifi. The problems being talked about on this bug involves higher traffic wifi situations.

The comparison analysis I'm looking for is situations like:

- Other office networks other than the Mountain View Mozilla office
** What happens in the SF office?
** What happens in the Portland office?
** What happens in the Toronto office?
- Hotel wifi networks
- Coffeehouse wifi networks
- Airport wifi networks
- Data plans (3G, 4G)

Comment 24

[Jason Smith [:jsmith]]

Putting qawanted to address comment 23.

Comment 25

[Maire Reavy [:mreavy]]

(In reply to Jason Smith [:jsmith] from comment #23)
> I think we already know that this works fine on residential wifi. The
> problems being talked about on this bug involves higher traffic wifi
> situations.

What surprised me was how poorly Chrome performed relative to Firefox in the same environment (same devices, same network, comparing Chrome 29 to Firefox 24).

Perhaps that info should have been captured in a separate bug, but it's worth capturing somewhere in bugzilla.

> 
> The comparison analysis I'm looking for is situations like:
> 
> - Other office networks other than the Mountain View Mozilla office
> ** What happens in the SF office?
> ** What happens in the Portland office?
> ** What happens in the Toronto office?
> - Hotel wifi networks
> - Coffeehouse wifi networks
> - Airport wifi networks
> - Data plans (3G, 4G)

All of those scenarios would be very helpful/useful to test -- especially if we can compare Firefox to Chrome 29 in each environment.

Comment 26

[Jason Smith [:jsmith]]

(In reply to Maire Reavy [:mreavy] from comment #25)
> (In reply to Jason Smith [:jsmith] from comment #23)
> > I think we already know that this works fine on residential wifi. The
> > problems being talked about on this bug involves higher traffic wifi
> > situations.
> 
> What surprised me was how poorly Chrome performed relative to Firefox in the
> same environment (same devices, same network, comparing Chrome 29 to Firefox
> 24).

Hmm okay. Interestingly enough, under the Mozilla wifi situation with an audio/video call, we did significantly worse than Chrome Beta - Aaron & I's tests showed latency to be significantly better when Chrome was used instead of Firefox. 

Perhaps each WebRTC implementation that exists (Chrome for Android & FxAndroid) is optimizing for different use cases right now? 

> 
> Perhaps that info should have been captured in a separate bug, but it's
> worth capturing somewhere in bugzilla.
> 
> > 
> > The comparison analysis I'm looking for is situations like:
> > 
> > - Other office networks other than the Mountain View Mozilla office
> > ** What happens in the SF office?
> > ** What happens in the Portland office?
> > ** What happens in the Toronto office?
> > - Hotel wifi networks
> > - Coffeehouse wifi networks
> > - Airport wifi networks
> > - Data plans (3G, 4G)
> 
> All of those scenarios would be very helpful/useful to test -- especially if
> we can compare Firefox to Chrome 29 in each environment.

Hotel wifi & airport wifi right now won't work with STUN, but per jesup's comments on the hotel wifi bug, this might be because we might need TURN in those networking environments.

I'll work with Aaron to get more data on latency for the other wifi/data networks mentioned above.

Comment 27

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Jason Smith [:jsmith] from comment #26)
> (In reply to Maire Reavy [:mreavy] from comment #25)
> > (In reply to Jason Smith [:jsmith] from comment #23)
> > > I think we already know that this works fine on residential wifi. The
> > > problems being talked about on this bug involves higher traffic wifi
> > > situations.
> > 
> > What surprised me was how poorly Chrome performed relative to Firefox in the
> > same environment (same devices, same network, comparing Chrome 29 to Firefox
> > 24).
> 
> Hmm okay. Interestingly enough, under the Mozilla wifi situation with an
> audio/video call, we did significantly worse than Chrome Beta - Aaron & I's
> tests showed latency to be significantly better when Chrome was used instead
> of Firefox. 
> 
> Perhaps each WebRTC implementation that exists (Chrome for Android &
> FxAndroid) is optimizing for different use cases right now? 

I suspect the answer is more random than that - with networks like this repeatability is low.  And the latency you heard could well be due to other things - I didn't get a good (audio) call with Chrome at all, and for whatever reason I found it far harder to actually connect.  debugging the 'why' would take some time.

> Hotel wifi & airport wifi right now won't work with STUN, but per jesup's
> comments on the hotel wifi bug, this might be because we might need TURN in
> those networking environments.

You can use them from there (probably), but many of them have blocked hairpinning (indirect p2p on their network) and direct LAN p2p traffic.  I.e. two devices in the *same* hotel network may not be able talk to each other without TURN.  If one's there, and one is on another network, they may be able to talk.

Comment 28

[Adrian Tamas (:AdrianT)]

On the SV WIFI with the ap configuration: security mode WPA2-Personel Mixed , WPA Algorithm : TKIP or AES, testing on http://apprtc.appspot.com/ with:
 * Samusng Galaxy Tab 2 (Android 4.1), Samsung Galaxy Note (Android 4.0) testing with Firefox Mobile 24 beta 6 and Chrome 29.0.1547.59
 * a desktop PC running Ubuntu 12.04 LTS with Firefox 24 beta 5 and Chrome 29.0.1547.57 
Here are my findings:
 * Firefox Mobile 24 beta 6 (Samsung Galaxy Tab 2) to Firefox Mobile 24 beta 6 (Samsung Galaxy Note) -> video freezes a lot and the audio is distorted at first and then starts to be more understandable but with ~2 min delay
 * Firefox Mobile 24 beta 6 (Samsung Galaxy Tab 2) to Firefox 24 beta 5 (Desktop, Linux) -> video freezes a lot and the audio is distorted at first and then starts to be more understandable but with ~2 min delay
 * Firefox Mobile 24 beta 6 (Samsung Galaxy Note) to Firefox 24 beta 5 (Desktop, Linux) -> video freezes a lot and the audio is distorted at first and then starts to be more understandable but with ~2 min delay
 * Chrome Mobile 29 (Samsung Galaxy Note) to Chrome 29 (Desktop, Linux) -> video freezes but not as much as with Firefox. Audio is not perfect but it is ok and does not have delays or time distortions
 * Chrome Mobile 29 (Samsung Galaxy Note) to Chrome Mobile 29 (Samsung Galaxy Tab 2) -> unable to connect in multiple tries
 * Firefox Mobile 24 beta 6 (Samsung Galaxy Note) to Chrome (Desktop,Linux) -> video freezes from time to time and audio is very time distorted(the audio is very laggy, with pauses and in slow motion) at the beginning. Later on it starts to be ok and can be understood but is delayed about 2 min

Comment 29

[Jason Smith [:jsmith]]

Hmm. The results in comment 28 seem to imply the following things:

* Mobile calls in general don't work well on SV Wifi with Chrome for Android or FxAndroid. Each test showed evidence of video freezes present during the call (with the exception of chrome for android --> chrome for android, in which the call failed entirely)

* Audio latency that was present in FxAndroid vs. Desktop Firefox was about the same

* Audio latency was only okay in Chrome Mobile to Chrome Desktop

Adrian - Can you provide more information on how good the SV wifi is generally? Do you guys have connectivity problems? Do you have slow wifi? Is it fast? slow?

Comment 30

[Adrian Tamas (:AdrianT)]

The wifi speed is ok. It is not extremely populated and only used to test mobile devices only. On a quick speedtest using the app in the appstore, connecting to a server in Bucharest (300+ km aways) both the upload speed and the download speed are about 20 Mbps.

Comment 31

[Randell Jesup [:jesup] (needinfo me)]

A comment made in bug 907213 implied to me that traffic is using TURN (in that apprtc apparently busted TURN in the last day or two, and suddenly their apprtc calls stopped working).  Using TURN on apprtc may involve relaying all audio/video traffic via a server in California (not sure exactly where it is)....

Can you verify in some way if these device-to-device calls are going locally (direct 192.168.* to 192.168.*), hair-pinning at the router (both sending to the external IP address of the router), or relaying via an external TURN server (both sending to different ports at a remote relay server)?

On a mobile->desktop call this should be reasonably easy by running Wireshark on the desktop and filtering for UDP packets.  If the network is otherwise idle, you may be able to check the router traffic/bandwidth monitors if it has them.

Also you were seeing a consistent 2-minute delay in audio?  one way, or both ways?  Did the delay increase or was it steady from the very start, or did it increase to 2min and then stay there?  If so, how long did it take?

How long did it take to connect when the second browser went to the link?

Can you do a single test of FxAndroid Nightly to Desktop Nightly and see what delay you get?

I'll note it took me 10 tries to get Chrome Android to call Chrome Android on apprtc, and then it had unusable audio. (Galaxy S4's, same Chrome rev you have).

Comment 32

[Aaron Train [:aaronmt]]

(In reply to Maire Reavy [:mreavy] from comment #25)
> (In reply to Jason Smith [:jsmith] from comment #23)
> > I think we already know that this works fine on residential wifi. The
> > problems being talked about on this bug involves higher traffic wifi
> > situations.
> 
> What surprised me was how poorly Chrome performed relative to Firefox in the
> same environment (same devices, same network, comparing Chrome 29 to Firefox
> 24).
> 
> Perhaps that info should have been captured in a separate bug, but it's
> worth capturing somewhere in bugzilla.
> 
> > 
> > The comparison analysis I'm looking for is situations like:
> > 
> > - Other office networks other than the Mountain View Mozilla office
> > ** What happens in the Toronto office?

> All of those scenarios would be very helpful/useful to test -- especially if
> we can compare Firefox to Chrome 29 in each environment.

A bit of testing this afternoon in-office Toronto office WiFi and to WiFi in California (T-Mobile Office). Currently omitting connectivity to Jason's 4G device (non of which we successfully connected to):

Mozilla Network (Toronto) (802.11A 5GHz, WPA2 Enterprise, RSSI -69, Transmit Rate: 24)

Fx 24.0 :: Samsung Galaxy SIV (Android 4.3) → Fx 24.0 :: LG Nexus 4 (Android 4.3)
↳ http://codeshare.io :: no audio or video latency issues, http://apprtc.appspot.com :: no audio or video latency issues

Fx 24.0 :: Samsung Galaxy SIV (Android 4.3) → Chrome 29 :: LG Nexus 4 (Android 4.3)
↳ http://codeshare.io :: no audio or video latency issues, http://apprtc.appspot.com :: no audio or video latency issues

Chrome 29 :: Samsung Galaxy SIV (Android 4.3) → Chrome 29 :: LG Nexus 4 (Android 4.3)
↳ http://codeshare.io :: no audio or video latency issues, http://apprtc.appspot.com :: no audio or video latency issues

Mozilla Network (Toronto) (802.11A 5GHz, WPA2 Enterprise, RSSI -69, Transmit Rate: 24) to T-Mobile WiFi (California)

Fx 24.0 :: Samsung Galaxy SIV (Android 4.3) → Fx 24.0 :: Samsung Galaxy Tab 3 (Android 4.2)
↳ http://codeshare.io, video was 'good'; audio heard from both recipients was consistently behind a 5-6 second delay. Video seemed to drop the occasional frame but our conclusion here was that video behaved much better than audio in this scenario

Comment 33

[Jason Smith [:jsmith]]

Tracking document btw for all of these comments is here - https://wiki.mozilla.org/QA/Fennec/WebRTC#Firefox_24_.28Networking.29.

Newest update is a test tchung & I did you can see in the tchung & jsmith portion of the wifi. Summary - FxAndroid did do better overall than Chrome for Beta in a T-Mobile Guest Wifi --> Haxxor ateam wifi call. FxAndroid could successfully establish the call, where Chrome for Android only got video running (not audio) for one person. During the FxAndroid call for a few minutes, we found that the video was smooth throughout, even I walked around. The audio however quickly became problematic - we saw evidence audio falling behind by up to 6 - 8 seconds with it becoming choppy as time passed. The choppy audio occasionally prevented the ability to understand the other person, but not always.

I think what the above results imply is that we've got a confirmed serious problem with not having an established network in the Mountain View office that we know webrtc calls will reliably work on. We may have done better than Chrome in this situation, but I'm surprised we saw choppy audio & audio falling behind on Haxxor's network (it's an isolated network locked away from the Mountain View wifi problems). We need a resolution path forward to establish a network we can rely on for testing in the Mountain View office for consistent results. Haxxor should be that candidate, but we apparently are not performant enough yet.

Comment 34

[Jason Smith [:jsmith]]

Talked this over with release management - the conclusion we've reached is that we'll go forward with the release for 24 with these latency issues known. We should however, push forward with an initiative to diagnose these latency issues clearly seen during testing to establish networks we can reliably use for webrtc calls. So a continued performance initiative needs to happen ongoing.

Comment 35

[Adrian Tamas (:AdrianT)]

(In reply to Randell Jesup [:jesup] from comment #31)
> A comment made in bug 907213 implied to me that traffic is using TURN (in
> that apprtc apparently busted TURN in the last day or two, and suddenly
> their apprtc calls stopped working).  Using TURN on apprtc may involve
> relaying all audio/video traffic via a server in California (not sure
> exactly where it is)....
> 
> Can you verify in some way if these device-to-device calls are going locally
> (direct 192.168.* to 192.168.*), hair-pinning at the router (both sending to
> the external IP address of the router), or relaying via an external TURN
> server (both sending to different ports at a remote relay server)?
> 
> On a mobile->desktop call this should be reasonably easy by running
> Wireshark on the desktop and filtering for UDP packets.  If the network is
> otherwise idle, you may be able to check the router traffic/bandwidth
> monitors if it has them.
I could only verify that from device to desktop (FF mobile 24 beta 6 to FF 24) are in the network (192.168.77.211 to 192.168.77.177) using wireshark on desktop and filtering UDP packages.

> Also you were seeing a consistent 2-minute delay in audio?  one way, or both
> ways?  Did the delay increase or was it steady from the very start, or did
> it increase to 2min and then stay there?  If so, how long did it take?
> 
> How long did it take to connect when the second browser went to the link?

Today, very early in the morning when almost no one was in the office, I managed a mobile to desktop connection where the quality was good. At one point there was a short delay introduced 1-2 sec. As people came in to the office and the network started to be used the calls became unusable. The video started freezing a lot and the sound became robotic, stuttered, only part of the words were played and was totally unusable. Also there was a delay introduced that got larger in time to about 30 seconds after ~3 min of active call

It seems that as the network is getting used the quality of the calls goes down exponentially. With Chrome to Chrome calls yesterday although it was hard to connect and the audio was also robotic and with missing parts it seemed better then firefox.

WebRTC does not work on Nightly Desktop at this time so I could not connect to test n nightly

Comment 36

[Adrian Tamas (:AdrianT)]

sorry it seems that the reply changed the tracking flag by accident

Comment 37

[Eric Rescorla (:ekr)]

(In reply to Adrian Tamas (:AdrianT) from comment #35)

> WebRTC does not work on Nightly Desktop at this time so I could not connect
> to test n nightly

This seems serious. Can you expand on this please?

Comment 38

[Adrian Tamas (:AdrianT)]

Please see https://bugzilla.mozilla.org/show_bug.cgi?id=907213#c9

Comment 39

[Jason Smith [:jsmith]]

This was likely fixed when jesup landed the perf improvement patch for audio latency of a remote stream. Closing out as such.