1295193 - Garbled audio and delay buildup with WASAPI full_duplex with rate mismatch on 3.5mm output

Status: RESOLVED FIXED

(Core :: Audio/Video: cubeb, defect, P1)

Description

[Chad Pilkey]

Just updated to the 49-beta3 build and when testing a WebRTC audio call the audio that I playback has static and is distorted and the audio that I send becomes more and more delayed as the call progresses.

All tests were on Windows 7. I tested calls between Chrome and Firefox, Firefox and Firefox, and Firefox and FreeSWITCH. The same problems presented in all calls on the Firefox side.

I tested using https://apprtc.appspot.com/ for between browser calls, https://webrtc.github.io/samples/src/content/getusermedia/audio/ for a local loopback test, and BigBlueButton (our application http://demo.bigbluebutton.org/demo/demo2.jsp) for the test with FreeSWITCH.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Thanks for the report.  I need more details:

what version did/does it work with?  Have your tried with Nightly or Developer Edition/Aurora?
Can you try flipping media.navigator.audio.full_duplex and see if this changes.  (I advise restarting after flipping it).

This was win7 - can you tell me what the audio input and output devices were (HW and SW) and the sampling rates they are set to in Control Panel?  Can you try switching to different devices?  (built-in on laptop, USB headset, split (headset mic and laptop speakers; headset speakers and laptop mic, system speakers and USB webcam mic)?  Also, if the input/output rates are different, can you switch them to be the same?

Local loopback in some cases will drift if the input and output devices use different clocks (we expect to fix this soon).

It would be useful as well if you can enable local playback of audio that's delayed when received (perhaps using devtools - Inspect Element, Use in Console, and set muted to false is a common trick).  This lets you see if the delay is at the sending side or the receiving side.

As for the static/distortion - can you record it (even with a smartphone held up near the speakers)? 

Thanks!

Comment 2

[Chad Pilkey]

The previous version that I was testing with was 47 (don't know the version sorry). I just installed Nightly (51.0a1) and tested with that and the problems are there as well. I flipped the full_duplex flag from true to false and the audio playback was fine (FF 49).

I was testing with a headset with 3.5mm input and output. I just grabbed a USB headset and tested with that as well. I think I tested all of the combinations between the 2 sets of devices and the problems presented only with the 3.5mm as the default playback device (USB mic with 3.5mm out, and 3.5mm mic with 3.5mm out). The 3.5mm headset is set to 24bit, 48000 Hz. The 3.5mm mic is set to 2 channel, 16 bit, 44100 Hz. The USB mic has the rates greyed out in Microphone Properties. I changed the rate on the 3.5mm mic to 48000 Hz, then restarted Firefox 49, and the problems went away.

I didn't try the Dev Tools trick, but I know for sure the delay is on the sending because with FreeSWITCH I can see the activity coming into it and I know that the distortion is on the playback because I can play static audio files on the server and they are distorted on my local machine.

I recorded a sample of it and uploaded it to Soundcloud, https://soundcloud.com/chad-pilkey/garbledwebrtcaudio. Another interesting thing that I noticed while recording the sample is that if you save the media stream and reuse for multiple calls the delay continues to get worse even when it's not attached to an active call.

Comment 3

[Chad Pilkey]

Sorry forgot to put in my sound card information. It is a Realtek sound card built into my motherboard.

Comment 4

[Chad Pilkey]

Another update, sorry for not putting them together. I decided to try Stable (48) because I had skipped from 47 to 49. Neither of the problems (delay and quality) is present.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

Ok, thanks a lot for the info; this narrows it down a lot.

Very likely this is a bug in the full_duplex resampling code meant to handle recording/playback sample-rate mismatches.  Since full_duplex provides (to the rest of firefox) input and output streams that are synchronized and a single sample rate, the wasapi backend in cubeb does rate conversions.

I'll look at it, and perhaps kinetik can look too - if it's something simple, we may be able to fix-and-uplift; otherwise we'll want to pref-off full_duplex for Windows in beta (it's preffed-off for Mac in beta 49 as well), and look to fix and uplift to 50 before 50 goes to beta.  Padenot did the wasapi full_duplex code, and is on PTO for a couple of weeks.

I hear odd audio in my 3.5 jack (which has no mic input) regardless of full_duplex or not, so that may not be a good test.  System mic and 3.5 output were both 48000 hz, but no idea if they're on the same clock.

Comment 6

[ffdixon]

I'm encountering the same audio distortion when testing with my Windows 10 machine using the BigBlueButton server at 

  http://demo.bigbluebutton.org/.

With FireFox 49.0b5, I join the computer from Chrome and FireFox and share my microphone in both, then take turns muting.  Chrome comes through fine, but in FireFox 49.0b5 this is what I hear

  https://soundcloud.com/fred-dixon-4/firefox-490b4-audio-test/s-iRfwq

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

We just fixed a bug in 49 and newer where if you call getUserMedia for audio more than once (capturing to multiple streams at once), the audio data will be double-inserted into every track.  This is bug 1297083.  This hasn't merged to mozilla-central, so it may or may not be in tomorrow's nightly (Aug 23); it shoudl be in the next one and soon uplift to aurora and beta.  Please try it; the distortion and delay seems similar

Comment 8

[Matthew Gregan [:kinetik]]

There's also bug 1272386 which is OS X but the resampler is implicated, so probably a cross-platform bug.  It'd be worth testing a debug build to see if the same debug-only assertions are triggering when reproducing this bug.

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

Please retry with today's nightly; bug 1297083 merged to mozilla-central yesterday.

Comment 10

[Chad Pilkey]

Just tested the new build. I reproduced 1297083 on 49 on a different computer just to see what was happening with that one and there's a slight difference between that bug and this one. In 12907083 the audio that you send is the only that is impacted, but with this bug the audio that you send is impacted and the audio that you receive is as well. It's hard to tell the difference between the two when you're just looping from your mic to yourself, but I tested with someone else or static sound files played through the call there's a difference.

For this bug I can confirm that it still exists on Nightly. Only open one tab and it happens every time when my mic is at 44100Hz and my headphones are at 48000Hz. If I change them both to 48000Hz the problem goes away.

Comment 11

[Randell Jesup [:jesup] (needinfo me)]

Thanks; that seems pretty clear.  I presume bug 1297083 is fixed in the nightly you tried (open two https://mozilla.github.io/webrtc-landing/gum_test.html's and capture audio in both).

Comment 12

[Chad Pilkey]

Yeah I can no longer reproduce 1297083 in Nightly when the my mic and speakers have the same sample rate.

Comment 13

[Maire Reavy [:mreavy]]

Paul has a fix, but he's at TPAC and can't verify the fix until he returns to the office.

Comment 14

[Paul Adenot (:padenot)]

Fixed by 1306570. Please try again on a nightly that has https://hg.mozilla.org/mozilla-central/rev/c06670c5446b.