1375729 - Can not use soundflower when using webRTC

Status: RESOLVED FIXED

(Core :: Audio/Video: cubeb, defect, P1)

Description

[kdalli]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:53.0) Gecko/20100101 Firefox/53.0
Build ID: 20170518000419

Steps to reproduce:

1. Start garageband or similar audio manipulation software on a computer(computer A). Direct the output to soundflower.
2. Start a webRTC app and set the input from the get user media to what is being output from soundflower ch 2. 
3. have second computer (computer B) with no audio manipulation join the webRTC room. 


Actual results:

The computer sending the manipulated audio (computer A)successfully sent manipulated audio to computer B, however, no audio was heard on computer A. Computer B played manipulated audio and computer A played no audio. Instead of two way communication it became only one way communication. The computer A seemed to be receiving the audio channel just fine, however, no audio could be heard. Firefox even acknowledged that audio should be head as the little speaker icon was on the tab. Yet still no audio. The issue was replicated on several different computers. All of them running firefox 54.  This was not an issue on 53. When soundflower was not used both computer A and B played each others audio


Expected results:

the manipulated audio should be heard on computer B and regular audio should be head on computer A. 

This is a vital part of my companies teaching methods. We use audio manipulation along with webRTC for avatar mediated training.

Comment 1

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Hi kdalli, thanks for filing an issue!

Note that RTCPeerConnection is largely content-agnostic, so from your description, it looks like this has nothing to do with RTCpeerConnection, rather it seems like a local audio output problem whenever soundflower is used. Can you confirm?

Would it be possible to reduce the reproduce steps to isolate the problem?

Also, you mentioned this worked in 53. That interesting. If this is a regression, then it would increase its priority.

Would you mind running https://mozilla.github.io/mozregression/ to find a regression range?

Comment 2

[kdalli]

Hey there, 

Everything works fine on chrome and on firefox 53. It works fine when outputing anywhere but firefox 54. The audio output is in fact fine. The computer using soundflower and garageband sends its audio to the other computer just fine,,, for some reason it does not recieve the untampered audio. So compuater A DOES manipulate sound and computer B recieves and outputs that sound. Computer B does not manpulate sound (and is not supposed to) and computer A does NOT output the unmanipulated sound. In chrome, it does, in firefox 53 it does. When using the same methods with skype, it does. This can be reproduced on any webRTC application. 

Also I have noticed that even before, when using soundflower, if I created an multi output device on my mac, so I would be able to hear what should be output in my speaker as well as sending audio to soundflower, audio is not sent over firefox. But i believe this is a different issue concerning how firefox deals with multi output devices and agregate devices. but it may be related, so i mention it. 


It is hard to simplify the repoduction steps further. However, after testing I see the error occurs anytime one uses soundflower 2ch as the input device choice for get user media before a webRTC call. This can be reproduced in conjunction with many different softwares, like VLC player, garageband or anything else that lets you set soundflower as your output. The result is, the audio you sent to soundflower will be comunicated and the reciever hears it well and good. However, the recievers audio (the person not using soundflower) is not heard. I am not sure what audio input devices this will be true with because I dont have the oportunity right now to test with anything but soundflower and my internal microphone on my labptop (that works fine btw). 

So I did mozregression and came up with some interesting behaviour over the last few months of updates. I don't normally use firefox nightly so I only caught this problem now, but it seems like it has experienced several errors. 

The following is when the problems began : 

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=c807c80d954a7761c3aae2036d3a1c4d3b55ba51&tochange=1fbff6defb481238929060e04b330e5870b013be

Before this point everything worked perfectly. After this point when soundflower is chosen as an input type firefox crashes entirely. This seems to be when problems began. 


The following is when it stopped crashing entirely but got to the issues that can be seen in the current version of firefox(54):

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=a477e80f03b61be9961bc61770a2b55cce139b91&tochange=745f2d85212d7c4bc82240c5a43730d9ecd32125

At this point firefox no longer outright crashes. However we can see the issue i described above. The computer using soundflower sends audio fine, but does not play any audio it is recieving. 

Finally, the newest versions of firefox nightly display this error when trying to use get user media to input soundflower as an audio device. Error name was NotReadableError. Continuing without sending a stream.  
This happens here : 
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=00f292748b3e702f338589863b986f2ee50d55cf&tochange=73260f7c6dae11039cc4c738ac8714671d8cd1df

At this point choosing soundflower as a device just gives a webrtc/get user media error saying the device is unreadable. 

I have tested using appr.tc as well as my own webRTC application. This is consistent with both and on multiple machines. 

This method for sending manipulated audio works fine on chrome, earlier versions of firefox and skype. 

I have asked all my clients to version back to firefox 53 or requested they use chrome until further notice as we cant pause our work. Though our company usually exclusively uses firefox.

Comment 3

[kdalli]

BTW in case I didnt mention it, I have tested on multiple machines. Everything has been tested and is running fine on chrome, skype and earlier versions of firefox. I dont think this is a local issue. 

Regards, 

Kevin

Comment 4

[kdalli]

Reproduce steps

1. open software that can output audio to soundflower (VLC, Garageband...) and then output to soundflower.
2. open a webRTC app, when the get user media requests asks for input, choose soundflower. 
3. Have a second computer that is not using soundflower join the room. 

That's as simple as i can make it... 

I am not sure why this is happening so it is hard for me to suggest simpler ways to reproduce the error. 
I just have to stress that the manipulated audio is sent and outputed. The issue is that the audio from the computer that is not using soundflower is not played. 


Kevin

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

We've had problems with induced delay and other problems with soundflower.  It appears soundflower doesn't provide all the API guarantees or side-effects that the normal Mac audio chain does.  Note that even having soundflower installed, but not active, caused audio capture drift in the past.  Uninstalling it was required to stop the drift.   Are you certain there wasn't drift after minutes or hours of capture?   Also, did you do 10+ minute webrtc calls (without using soundflower as an input) in these older versions with no problems?

While it might not cover your usecase, you can modify audio extensively using WebAudio before sending it.

If soundflower works correctly as a virtual input device to the mac system, this shouldn't be happening, but my guess is that their implementation is incomplete and/or has bugs.

achronop: likely this is related to your changes, but it might not be something we can do much about.

Comment 6

[kdalli]

We have done many 10 + minutes webRTC calls with out using soundflower and we have had no problems. However it is true that after 40 minute sessions there was sometimes slight drift, only when using soundflower, but not when it was not used. In our use case after 30 minutes our sessions are usually finished. So we would restart soundflower and garageband and there would be no problems after that. This limitation does not bother us and we would like to continue using soundflower as we did before. We were unable to use soundflower with chrome for a very long time. This issue has now been fixed on there side and in fact has less drift issues (though still some). Also, this drift does not seem to be present when using soundflower with other compatible applications like audicity, OBS, WireCast and garageband. 

Our problem with using WebAudio is that it was not compatible with chrome when sending audio channels with get user media.  We were unable to use the Web Audio API to pitch shift audio and then send it over webRTC even with firefox. For some reason when modifying audio and then trying to send it through webRTC, nothing would be recieved by the receipient. As a work arround we set up the web audio api to pitch shift the recieved audio before it output it to the users speakers. However the web audio API could not produce the quality of a pitch shift the we need to successfully perform our sessions. So we use garageband to do this for us. As garageband can produce much better quality pitch shifts (aka the person does not sound like a robot) The web audio api just did not proove to provide the quality we need as of yet. We are quite happy with the solutions and methods we are using now and spent a lot of resources and efforts to find the best solution for our use case. The lack of compatibility was surprising to us. 

Should I expect to tell my clients we should permenantly switch to using chrome only or older versions of firefox or is this something that can be fixed ?

Comment 7

[Alex Chronopoulos [:achronop]]

Finally back to this one. The problem is the configure of aggregate device on duplex streams. Our backend system configures internally an aggregate device every time the input and the output operates together. That aggregate device is the combination of input Soundflower (2c), in our case, and the output - Internal Speakers. For some reason this combination produces no audible output in Firefox. When I switch back to the old mechanism making use of separate input and output the behavior is the expected. I need to investigate it more in order to understand if this is a generic problem of Aggregate Devices with Soundflower or it's the way we configure our aggregate device.

The NotReadableError error is not reproducible here. I am in latest Nightly. Can you please confirm that?

Comment 8

[kdalli]

Hey, 
I am traveling all day today, I will check about wether or not the NotReadableError is coming up on the latest nightly tomorrow when i am back in the office. Sorry for the delay.

Comment 9

[Alex Chronopoulos [:achronop]]

I believe I can fix it. The way that we configure the internal aggregate device seems to be the issue. I changed the properties for the default output stream and the problem is solved, at least locally. I will push the fix soon in order to be able to confirm it. When you have the chance to check the NotReadableError let me know. Thank you for the support.

Comment 10

[kdalli]

Hey ! 
So I ran a quick test with the new version of nightly and did not get the NotReadableError. However this was where I was getting the not readable error when I ran the regression range finder software. 

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=00f292748b3e702f338589863b986f2ee50d55cf&tochange=73260f7c6dae11039cc4c738ac8714671d8cd1df

Comment 11

[kdalli]

Great ! 

Thank you so much for looking into this for me !, My company relies on it ! :)

Comment 12

[Alex Chronopoulos [:achronop]]

Fix landed on cubeb repo, will be imported with next cycle.
https://github.com/kinetiknz/cubeb/commit/55ef19406ad0a945902eb0de449c9add578235cd

Comment 13

[kdalli]

Sounds good, I will update all our browsers upon cycle release ! 

Thanks again !