842631 - getUserMedia - The audio received on an Android device plays back delayed by about a second

Status: RESOLVED WORKSFORME

(Core :: WebRTC: Audio/Video, defect)

Description

[Florian Quèze [:florian]]

Steps to reproduce:
Do a webrtc audio+video call between desktop and android.

The sound from the desktop's microphone is played on the android device approximately 1s after the video.

The sound in the other way (from the android device's microphone to the desktop machine) doesn't seem significantly delayed (it's in sync or almost in sync with the video).

This was tested with Nexus 4 android phones, and reproduced by at least 2 people.

Comment 1

[Mark Banner (:standard8)]

Note: we're using builds based on Alder, which include the a/v sync bug fix. The desktop machines are all Macs.

We've also reproduced using S3, and also a galaxy note.

Comment 2

[Romain Gauthier [:tOkeshu]]

Reproduced with Linux Desktop (Ubuntu 12.10) and LG Nexus 4.

Comment 3

[Florian Quèze [:florian]]

Jesup asked me during the webrtc apps call if this is only with the mac builtin microphone or also with the headset's microphone. I've just tested, it also happens when using the headset's microphone.

Someone (Maire?) also suggested turning off echo cancellation. It didn't help either.

Comment 4

[Jason Smith [:jsmith]]

I just tested using the gum test page as well on Android - this actually has nothing to do with the full stack. It's a gUM bug in the Android space.

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

Added roc and kinetik, since we're down to a more restricted case of

<android audio drivers>
<OpenSL ES>
webrtc.org audio capture
MediaEngineWebRTCAudio::Process()
MediaStreamGraph()
Audio element
cubeb
<android audio drivers>

Comment 7

[Matthew Gregan [:kinetik]]

We're not using cubeb for audio output on Android yet (we need bug 698328, bug 837564, at least).  The sydneyaudio backend that we are currently using uses a 1s buffer on Android (and much less (~55ms) on Gonk).

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

That's very suspicious... 1 second is the apparent delay we're seeing.  Padenot or kinetik, can you figure out a way to find out if the output buffer is running a 1s delay when we hear this problem?  (Perhaps padenot's delay-monitoring stuff)

Comment 9

[Matthew Gregan [:kinetik]]

A quick test would be to try setting media.use_cubeb to true in about:config and seeing if it helps, that should be using a ~100ms buffer (+/- whatever Android ends up forcing you to use).

Comment 10

[Paul Adenot (:padenot)]

I'm going to investigate today.

Comment 11

[Florian Quèze [:florian]]

(In reply to Matthew Gregan [:kinetik] from comment #9)
> A quick test would be to try setting media.use_cubeb to true in about:config
> and seeing if it helps

I tested this. The audio and video streams are still out of sync.
I was under the impression that flipping that preference slightly improves the situation (with the delay reduced from one second to half a second), but as I have no way to actually time the delay, it's possible this improvement is just coming from my imagination or hopes.

Comment 12

[Jason Smith [:jsmith]]

(In reply to Florian Quèze [:florian] [:flo] from comment #11)
> (In reply to Matthew Gregan [:kinetik] from comment #9)
> > A quick test would be to try setting media.use_cubeb to true in about:config
> > and seeing if it helps
> 
> I tested this. The audio and video streams are still out of sync.
> I was under the impression that flipping that preference slightly improves
> the situation (with the delay reduced from one second to half a second), but
> as I have no way to actually time the delay, it's possible this improvement
> is just coming from my imagination or hopes.

Florian - How does this compare to when you test this on the gum test page with Audio and Audio+Video - http://mozilla.github.com/webrtc-landing/gum_test.html?

Comment 13

[Paul Adenot (:padenot)]

Attachment: Bring the latency down. r=

So, this brings the latency down by using the OpenSL cubeb backend, an
requesting a latency of 20ms. Also, it reduces the cubeb buffer size.

Comment 14

[Florian Quèze [:florian]]

(In reply to Jason Smith [:jsmith] from comment #12)

> Florian - How does this compare to when you test this on the gum test page
> with Audio and Audio+Video -
> http://mozilla.github.com/webrtc-landing/gum_test.html?

It seems similar. Audio and video are out of sync by almost 1s.

Comment 15

[Gian-Carlo Pascutto [:gcp]]

Isn't playback of WebRTC stuff done through the OpenSL backend in the WebRTC code? Or is that only used on the capture side?

Comment 16

[Randell Jesup [:jesup] (needinfo me)]

Capture side only.  Output is through assigning a stream to a media element.

Comment 17

[Dan Mosedale (:dmosedale, :dmose)]

Unfortunately, when I tried this patch on my optimized/non-DEBUG build, it didn't seem to effect the in-call latency much.  There was still about a 1-second delay. :-(  Try builds with the patch are available from the B links in <https://tbpl.mozilla.org/?tree=Try&rev=679457b301c3>.

Comment 18

[Paul Adenot (:padenot)]

I only tested gUM, and I saw a lot of improvement. I'll be testing with a real call today.

Comment 19

[Florian Quèze [:florian]]

(In reply to Dan Mosedale (:dmose) from comment #17)
> Unfortunately, when I tried this patch on my optimized/non-DEBUG build, it
> didn't seem to effect the in-call latency much.  There was still about a
> 1-second delay. :-(  Try builds with the patch are available from the B
> links in <https://tbpl.mozilla.org/?tree=Try&rev=679457b301c3>.

I've just tested these try builds. I see (or rather hear) a dramatic improvement, both on the gum page and during calls.
The audio latency is short enough that I likely wouldn't notice it if I didn't have the phone and the laptop on the same desk.

Comment 20

[Dan Mosedale (:dmosedale, :dmose)]

I wonder if the difference is that I didn't tweak anything echo canceller-related, either in the builds, the prefs, or on phone.  Florian, is it possible that those tweaks that you had been playing with before will still in effect?

Comment 21

[Dan Mosedale (:dmosedale, :dmose)]

s/will still/were still/

Comment 22

[Florian Quèze [:florian]]

(In reply to Dan Mosedale (:dmose) from comment #20)
> I wonder if the difference is that I didn't tweak anything echo
> canceller-related, either in the builds, the prefs, or on phone.  Florian,
> is it possible that those tweaks that you had been playing with before will
> still in effect?

I actually had media.peerconnection.aec_enabled set to false on my phone. I've just tested this again with the pref back to its default value, and I still get the same results as in comment 19.

Comment 23

[Florian Quèze [:florian]]

Another thing to mention here: there seems to be some audio drift. When the call starts, audio and video are in sync or almost in sync. After a few minutes the audio and video from the phone arrive out of sync on the desktop, and the delay video and audio keeps increasing over time. The sound and video from desktop are still in sync even after a while.

Comment 24

[Dan Mosedale (:dmosedale, :dmose)]

Having a Mac build without this patch call an Android build with it seems to usually / mostly be lower latency and sounds great.

However, if the Mac build also has this patch, the audio received on the Android phone from the Mac has regular irritating glitches.  I suspect most people who've tried so far weren't testing with both of their builds patched.

Tweaked patch forthcoming.

Comment 25

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: bring the latency down, v2 (checked in on alder)

This ifdefs the bufferLimit change to only happen on Android, and leaves it alone on other platforms.

Comment 26

[Dan Mosedale (:dmosedale, :dmose)]

Try server build in progress: 

https://tbpl.mozilla.org/?tree=Try&rev=8b3e029ff9fe

Comment 27

[Dan Mosedale (:dmosedale, :dmose)]

I'm not totally convinced that these changes are quite as good as before this patch on the sound quality front, though I'm not at all sure how that could be.  It may well indeed just be coincidental network quality issues.

However, the latency is different better with the try build on both ends.

On balance, this feels better.

Since if gather data in the next day or so that suggests this was the choice, we can back it out, or simply fall back to the previous build, I'm landing on alder.  (The reasoning mentioned here is what we agreed to go forward with earlier today).

Comment 28

[Dan Mosedale (:dmosedale, :dmose)]

Pushed to alder: 

https://hg.mozilla.org/projects/alder/rev/7f758499c51e

Comment 29

[Dan Mosedale (:dmosedale, :dmose)]

Something like this is likely needed for uplift, so I've added [android-trunk-needed].  How closely it relates to this patch is likely to depend on the high-level audio strategy for Android going forward...

Comment 30

[Dan Mosedale (:dmosedale, :dmose)]

Goodness, let's try comment 27 again:

Since if gathering data in the next day or so that suggests this was the wrong choice, we can back it out, or simply fall back to the previous build, I'm landing on alder.  

Typing is hard, I'm going shopping.

Comment 31

[Jason Smith [:jsmith]]

gcp - Was this fixed when we landed on central?

Comment 32

[Jason Smith [:jsmith]]

Actually, nevermind. This is definitely still reproducible, in fact it's a few seconds in delay on apprtc.

Comment 33

[Jason Smith [:jsmith]]

Putting into blocking? to run some tests to see how bad this is to figure out if we need to block or not

Comment 34

[Jason Smith [:jsmith]]

QA Wanted for comment 33

Comment 35

[Aaron Train [:aaronmt]]

Using Nightly (06/02) on Mac OS X (10.8.3) and Nightly (06/02) on Android (4.1) on an HTC One and Galaxy S4 (Android 4.2), and a Galaxy Note II (Android 4.2.2) and Chromium (29.0.1527.0 (203622)), I am unable to discern any troubling ~1s± delay in audio and video between clients using http://apprtc.appspot.com as a test application for connecting clients. The latency difference I saw between audio and video sync between desktop and mobile device was rather minuscule very much less than a <1s on desktop and Android.

Comment 36

[Jason Smith [:jsmith]]

Sounds good. Closing as wfm then.