1261280 - Voice on WebRTC is recorded via phone's microphone instead of BT earphone.

Status: NEW

(Core :: WebRTC: Audio/Video, defect, P3)

Description

[Blake Wu [:bwu][:blakewu]]

STR: 
1. Pair a BT earphone with my Sony Z3C
2. Use Firefox to go to https://talky.io/ to create a room. 
3. Join the room and start to talk
4. Put phone away with a distance.

Actual behavior:
Receiver cannot hear the voice clearly. 

Expected behavior:
Receiver can still hear the voice clearly.

Comment 1

[Blake Wu [:bwu][:blakewu]]

After discussing with alwu, the possible root cause could be currently FireFox uses "music" type for audio instead of voice call type. 
alwu, 
Could you elaborate it more?

Comment 2

[Alastor Wu [:alwu]]

I don't understand the detailed mechanism about how BT records sound from its microphone.
However, I know we use the "music" stream type for Android's audio track [1], not sure if it would affect BT voice-recording.

[1] https://dxr.mozilla.org/mozilla-central/source/dom/media/AudioStream.cpp#346

Comment 3

[Blake Wu [:bwu][:blakewu]]

alwu, 
We can have a quick check that if we modify stream_type to be "CUBEB_STREAM_TYPE_BLUETOOTH_SCO"[1], would it work well or not? If yes, we can think how to pass the right type from the upper layer. 
Please take care of this bug.
Thanks! 

[1]https://dxr.mozilla.org/mozilla-central/source/media/libcubeb/include/cubeb.h#111

Comment 4

[Alastor Wu [:alwu]]

In this case, if we want to use BT device as input/output, we need to do two things.
(1) Change Android stream type to "STREAM_VOICE_CALL"
(2) Enable SCO via Android::AudioManager [1]

Although now I can route the input device to AUDIO_DEVICE_IN_BLUETOOTH_SCO_HEADSET (from dumpsys info), I can't hear the my voice which should be recorded by BT mic. 

[1] http://developer.android.com/intl/zh-tw/reference/android/media/AudioManager.html#startBluetoothSco%28%29

Comment 5

[Alastor Wu [:alwu]]

I guess the reason might be the input sample rate not match, the SCO only accepts 8kHz for its input.

Comment 6

[Alastor Wu [:alwu]]

Now I can use BT mic to record voice via WebRTC if we did the following works,
(1) Change output stream to "STREAM_VOICE_CALL"
(2) Change input sample rate to "8K"
(3) Enable SCO via Android::AudioManager

Comment 7

[Alastor Wu [:alwu]]

In addition, this bug is not firefox-only, it also happens in Chrome, but not in Opera.

Comment 8

[Alastor Wu [:alwu]]

Now the thing would be little complicate if we change the output stream to "Android::AudioManager::STREAM_VOICE_CALL".

First, the stream type of Android is used for the output device choosing which is implemented in Android::AudioPolicyManagerBase [1].

If we want to use BT mic, we must use this stream type for the cubeb audio-backend. When using this type, AudioPolicyManagerBase would choose receiver or headset as output device. [2] That means this stream don't suite for any case, we should only use it during WebRTC communication. Eg, if we use this type for music playing, the user would think the sound is in a low voice because it's playing out from receiver instead of speaker.

For above reasons, we should only use "STREAM_VOICE_CALL" for getUserMedia() during WebRTC communication, that is to create a "telephony" MediaStreamGraph with "STREAM_VOICE_CALL" Cubeb back-end.

However, it introduces another problem, that is the getUserMedia() might be used cross different MediaStreamGraph. 

In present codebase, we would have multiple MediaStreamGraph which are based on different audio channel types. The media stream can't cross different MediaStreamGraph. The media stream can only live on the one MediaStreamGraph. 

Here is another example, when I use talky.io [3], this website use an AudioContext and it want to have an stream from getUserMedia, so it create an "normal" type MediaStreamGraph for AudioConext. In other hand, we create a "telephony" type MediaStreamGraph if we want to use STREAM_VOICE_CALL for getUserMedia() when WebRTC and bluetooth both are enable. 

Boom! That is an unexpected behavior! The stream can't cross these two graph!

[1] http://androidxref.com/5.1.0_r1/xref/hardware/libhardware_legacy/audio/AudioPolicyManagerBase.cpp#2499
[2] http://androidxref.com/5.1.0_r1/xref/hardware/libhardware_legacy/audio/AudioPolicyManagerBase.cpp#2575
[3] https://talky.io

Comment 9

[Alastor Wu [:alwu]]

Hi, Paul,
I want to propose a method to solve the issue in comment8.
Could you give me some suggestion for that?

I would like to do two things,
(1) use singleton for MediaStreamGraph
We don't use audio channel to choose different MediaStreamGraph, only return the singleton for usage.
Although it has some advantage like choosing different audio output device via audio channel, but it's seldom used (now only be used in b2g) and it's only useful in Android-based platform. 

In desktop, it seems no any benefit to keep multiple MediaStreamGraph alive because the media stream can't cross different graphs.

(2) switch AudioCallbackDriver when using getUserMedia() 
The "CUBEB_STREAM_TYPE_MUSIC" backend is fine for most cases. However, we need to change to "CUBEB_STREAM_TYPE_VOICE_CALL" after starting WebRTC communication. So I need to switch the graph driver dynamically in getUserMedia().

Very appreciate your help!

Comment 10

[Paul Adenot (:padenot)]

Alastor, we're looking into this at the moment. I think it should be fixed with enumerateDevices [0], that would list the bluetooth headset, and having multiple MSG that can communicate. We have a plan for that at [1], I just wrote this a couple days ago. I think it would solve the issue here, what do you think ?

[0]: https://w3c.github.io/mediacapture-main/getusermedia.html#dom-mediadevices-enumeratedevices
[1]: https://docs.google.com/document/d/1H7Vr25Xzq-9I-r6xd1xCe0ZiRtX0OmgenJN1761xez8/edit?usp=sharing

Comment 11

[Alastor Wu [:alwu]]

Attachment: Set SCO stuffs in WebRtcAudioManager

Comment 12

[Alastor Wu [:alwu]]

Hi, Paul,
Could you help me explain the detail about how to solve this problem with enumerateDevices?
But even we have this feature, the problem would still happen on the websites which doesn't appoint the device for enumerateDevices, is that right?

Communicate across multiple MSG seems cool! but here I have some other questions, 

---

(1) "Prepare some work for the incoming implementation of MediaSession" in [1]
This might be not related with that? The kinds in MediaSession is used for defining different audio competing behaviors, it seems not related with device-choosing.

Eg, in Android, the types for "device-choosing" and "audio-competing" is independent two things.

---

(2) "stop using webrtc.org’s code for that, to instead rely on cubeb." in [1]
In this case, the preliminary solution I thought is to set something about SCO in WebRtcAudioManager.java [2]. That code is part of the webrtc.org. Does that means the cubeb would have something like that in our future plan? 

If so, in this case, how do you set the SCO related stuffs in cubeb? using JNI to call AudioManager or...?

---

(3) automatically transfer input device to BT
Like above said, we couldn't use BT mic if the website doesn't implement enumerateDevices in their code. Should we automatically detect BT related stuffs and setup them for user? 

---

Very appreciate your help!

[1] https://docs.google.com/document/d/1H7Vr25Xzq-9I-r6xd1xCe0ZiRtX0OmgenJN1761xez8/edit?usp=sharing
[2] comment 11

Comment 13

[Paul Adenot (:padenot)]

(In reply to Alastor Wu [:alwu] from comment #12)
> Hi, Paul,
> Could you help me explain the detail about how to solve this problem with
> enumerateDevices?
> But even we have this feature, the problem would still happen on the
> websites which doesn't appoint the device for enumerateDevices, is that
> right?

It is the responsibility of the website or the user to pick the right device for their use case. I don't think we can make the assumption that because there is a bluetooth device connected, the stream should be routed to it.

> (1) "Prepare some work for the incoming implementation of MediaSession" in
> [1]
> This might be not related with that? The kinds in MediaSession is used for
> defining different audio competing behaviors, it seems not related with
> device-choosing.
> 
> Eg, in Android, the types for "device-choosing" and "audio-competing" is
> independent two things.

Indeed, having multiple MSG allow us to do more things than just choosing devices.

> (2) "stop using webrtc.org’s code for that, to instead rely on cubeb." in [1]
> In this case, the preliminary solution I thought is to set something about
> SCO in WebRtcAudioManager.java [2]. That code is part of the webrtc.org.
> Does that means the cubeb would have something like that in our future plan? 

We're planning to not use the webrtc.org code for audio devices in the future yes. Alex Chronopoulos is working on the cubeb code for android (achronop on IRC).

> If so, in this case, how do you set the SCO related stuffs in cubeb? using
> JNI to call AudioManager or...?

Either that, or dlopen("libmedia.so") like we do in other places of the code (cubeb_opensl.c), I' sure you know better than me what is best !

> (3) automatically transfer input device to BT
> Like above said, we couldn't use BT mic if the website doesn't implement
> enumerateDevices in their code. Should we automatically detect BT related
> stuffs and setup them for user? 

No, we can't do that. We can't make the decision that because gUM is being used, we need to prefer the bluetooth device in phone mode, and not, say, in a2ps mode, or use the mic instead, or some sound card that is plugged in, there is the need to have an intent somewhere from the user or the app.

That said, if the user picked the bluetooth input in voice mode, we can probably route the output to bluetooth earpiece in voice mode as well.

mozAudioChannel was good for that, but it's not standard, so we have to fall back on `enumerateDevices`.

Comment 14

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.