1091417 - Investigate to design Bluetooth SCO connection (bluetooth voice call audio) API for WebRTC/VoIP/Voice Dialing use cases

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: AudioChannel, defect)

Description

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

There are two connection types when we use Bluetooth Handsfree headset.
When we saw Handsfree connection established from Bluetooth Setting menu, it means Signaling channel has established, we call it RFCOMM connection. However, we need to establish Bluetooth SCO connection when call is connected, and route voice call audio to Bluetooth headset. We cannot establish SCO connection permanently after RFCOMM connection established due to power consumption reason at both phone and Blueototh headset sides. So it's hard to let gecko decide when we can automatically establish SCO connection and route audio data.

By default audio (normal mode) will be routed to Bluetooth device, but only for stereo (PCM data such as Music/Ringtone) through A2DP profile. For VOIP/Voice Dialing/WebRTC use cases, those use cases need to transmit voice call, we currently don't have any API to establish BT SCO connection automatically for privilege apps.

Comment 1

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

I would like know if Bluetooth establish SCO connection in none in-call mode, can our audio system work correctly?

Comment 2

[Randy Lin [:rlin] - Be Mozillian on 2015/01/26]

dumpsys and found those logs. 
=======
voip

Outputs dump:
- Output 2 dump:
 Sampling rate: 48000
 Format: 00000001
 Channels: 00000003
 Latency: 50
 Flags 00000002
 Devices 00000020
 Stream volume refCount muteCount
 00     0.008     01       00
 01     0.000     00       00


AudioPolicyManager Dump: 0xb845c4a0
 Primary Output: 2
 A2DP device address: 00:1E:DC:A2:99:EF
 SCO device address: 00:1E:DC:A2:99:EF
 USB audio ALSA 
 Output devices: 000000a3
 Input devices: 0000018c
 Phone state: 3  <-- setPhoneState = AUDIO_MODE_IN_COMMUNICATION
 Force use for communications 3  <-- setForceUse(AUDIO_POLICY_FORCE_BT_SCO)
 Force use for media 0
 Force use for record 3 <-- setForceUse(AUDIO_POLICY_FORCE_BT_SCO)
 Force use for dock 8
 Force use for system 0

Inputs dump:
- Input 350 dump:
 Sampling rate: 8000  <-----
 Format: 1
 Channels: 00000010
 Devices 80000008
 Ref Count 1

SetPhoneState into communication mode and let bt sco device on and setForceUse to bt_sco could let voice sound go through BT device. But the recording sample rate may need to use 8k.

Comment 3

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Hi Bruce,
We need to address this problem, and work with WebRTC guys. Can we put this into v2.2 backlog? We already seen this requirement from v2.1, for Loop application and WebRTC.

Comment 5

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Carmen,
Can we have more use cases description here?
We also need to evaluate different permutations, 
ex:
1) During voice call, there is an incoming VOIP call
2) During VOIP call, there is an incoming voice call

Paul has concerns about DoS cases with VOIP.

More Loop application use case detail can help us to evaluate solution and risks.

Comment 6

[Carmen Jimenez Cabezas]

For Loop, I believe there are two different issues here:

*Audio output through the Bluetooth headset. This can be managed by the app (by making the bluetooth permission privileged/adding a moz-bluetooth permission just for loop/adding a privileged bluetooth permission that allows a subset of all the Bluetooth manager functionality). In fact, I have a working patch for that at bug 1087463. The patch manages bluetooth connections and disconnections during the call.

* Audio input (microphone) through the Bluetooth headset. Since Loop uses WebRTC for all the communication, this has to be managed internally by getUserMedia, at the platform/Gecko level, and cannot be managed by the app. Since the standard says that the input device cannot be changed during the life of a stream obtained from getUserMedia, this means that either we can't allow connecting/disconnecting a bluetooth headset during the call, or that Gecko has to treat internally the Bluetooth microphone and the headset microphone as the same (fictitious) device, so the switch from one to the other is transparent to the application. The other solution could be creating/destroying streams every time we switch from one device to the other, and I don't know how well that would work, performance wise. 

As for the use cases when a voip call interrupts a voice call, or the reverse, I guess we should do the same we do when we're using a headphone or the earphone/speaker:

 * When a new VoIP call is received, and the user accepts it, the voice call is put on hold, and the VoIP call takes ownership of the output and input devices for the duration of the call. After the call is finished, the voice call is resumed.
 * When a new voice call is received and accepted, the voip call is put on hold, and the voice call takes ownership of the output and input devices. 

Note that this is currently broken (there's a bug open for that) but that's the expected behavior.

What are Paul DoS worries?

Comment 7

[Paul Theriault [:pauljt] (no longer reading bugmail)]

(In reply to Carmen Jimenez Cabezas from comment #6)
> For Loop, I believe there are two different issues here:
> 
> *Audio output through the Bluetooth headset. This can be managed by the app
> (by making the bluetooth permission privileged/adding a moz-bluetooth
> permission just for loop/adding a privileged bluetooth permission that
> allows a subset of all the Bluetooth manager functionality). In fact, I have
> a working patch for that at bug 1087463. The patch manages bluetooth
> connections and disconnections during the call.

It would be nice to stop adding more moz- permissions and find a proper solution here if we can. (ie one that is safe to expose to all privileged apps). But the current API I don't think is safe - i.e. you have no guarantee that another app with bluetooth permission won't interfere with hello by changing settings unexpectedly. 

I wonder though if we should be exposing the full bluetooth API here, or we should just be exposing an API which allows apps to connect to bluetooth headsets. (ie provide something higher level, and take care of the bluetooth connection stuff inside gecko) Just a though, maybe not feasible in 2.2 time frame and we just need to go moz-bluetooth option.




> 
> * Audio input (microphone) through the Bluetooth headset. Since Loop uses
> WebRTC for all the communication, this has to be managed internally by
> getUserMedia, at the platform/Gecko level, and cannot be managed by the app.
> Since the standard says that the input device cannot be changed during the
> life of a stream obtained from getUserMedia, this means that either we can't
> allow connecting/disconnecting a bluetooth headset during the call, or that
> Gecko has to treat internally the Bluetooth microphone and the headset
> microphone as the same (fictitious) device, so the switch from one to the
> other is transparent to the application. The other solution could be
> creating/destroying streams every time we switch from one device to the
> other, and I don't know how well that would work, performance wise. 
> 
> As for the use cases when a voip call interrupts a voice call, or the
> reverse, I guess we should do the same we do when we're using a headphone or
> the earphone/speaker:
> 
>  * When a new VoIP call is received, and the user accepts it, the voice call
> is put on hold, and the VoIP call takes ownership of the output and input
> devices for the duration of the call. After the call is finished, the voice
> call is resumed.
>  * When a new voice call is received and accepted, the voip call is put on
> hold, and the voice call takes ownership of the output and input devices. 
> 
> Note that this is currently broken (there's a bug open for that) but that's
> the expected behavior.

This is all contigent on Hello playing nicely with the dialer right? (i.e. it wouldn't work if we allow third party apps to use bluetooth headsets?) 

> 
> What are Paul DoS worries?

I just want to understand what the threats are here. I don't know this technology well enough to have a clear idea, but the sorts of issues I would expect are:

1. an app routes audio to a speaker when the user wasn't expecting (privacy)
2. an app routes enables a bluetooth microphone to record audio (can the app get audio with ONLY the bluetooth permission, or do they still need access to getUserMedia to get the audio stream?)
3. An app modifies bluetooth settings in a way that interferes with other apps (maybe we can make it foreground only, but this was an issue for Loop/Hello in the past, since you want to answer calls)

What do you think? Are these realistic threats or are there mitigating controls? Are there any other threats you can think of?

Comment 8

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Paul Theriault [:pauljt] from comment #7)
> (In reply to Carmen Jimenez Cabezas from comment #6)
> > For Loop, I believe there are two different issues here:
> > 
> > *Audio output through the Bluetooth headset. This can be managed by the app
> > (by making the bluetooth permission privileged/adding a moz-bluetooth
> > permission just for loop/adding a privileged bluetooth permission that
> > allows a subset of all the Bluetooth manager functionality). In fact, I have
> > a working patch for that at bug 1087463. The patch manages bluetooth
> > connections and disconnections during the call.
> 
> It would be nice to stop adding more moz- permissions and find a proper
> solution here if we can. (ie one that is safe to expose to all privileged
> apps). But the current API I don't think is safe - i.e. you have no
> guarantee that another app with bluetooth permission won't interfere with
> hello by changing settings unexpectedly. 
I guess the problem you though is the case: 
1. App A call connectSco
2. App B call disconnectSco

App B might interfere App A by calling disconnectSco.
Maybe we can implement it adding conn_id. 
When App A call connectSco, we return conn_id to App A.
App A need to pass conn_id as a parameter to disconnectSco.
So if App A call connectSco, App B maliciously call disconnectSco, App B needs to know the conn_id first. connect_id can be UUID or random number.

Or we can maintain a table recording child id inside Gecko, checking if App B never request connectSco and call disconnectSco, directly return NS_FAIL.


If we have concerns about let VOIP privilege applications have bluetooth permission, maybe we should have different layers of bluetooth permission control. Currently bluetooth api v2 (bluetooth mgmt part), having bluetooth permission can only do 'Discovery', 'Set discoverable', 'Pair/UnPair'.

I think 'Set discoverable', 'Pair/UnPair' could be sensitive, maybe we should have bluetooth-mgmt (or bluetooth-admin) permission and apply permission on those functions.

So we still can expose bluetooth permission to privilege applications, and we don't need to worry about malicious apps can call api and touch additional Bluetooth settings. So we can still give apps ability to access bluetooth, such as VOIP/Voice Dialing/Siri-like applications through bluetooth handsfree.  
The same permission case we can apply to give bluetooth abilities to Music application to do AVRCP metadata api, as you previously raised questions we shouldn't give too much permission to Media related application.
'bluetooth' permission is more like Bluetooth profile-level permission so privileged apps (VOIP/Voice Dialer/Voice assistant/Music/Video applications) can use.

Comment 9

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Paul Theriault [:pauljt] from comment #7) 
> I wonder though if we should be exposing the full bluetooth API here, or we
> should just be exposing an API which allows apps to connect to bluetooth
> headsets. (ie provide something higher level, and take care of the bluetooth
> connection stuff inside gecko) Just a though, maybe not feasible in 2.2 time
> frame and we just need to go moz-bluetooth option.
Yeah, but Gecko still needs to get indication from Gaia, so Gecko Bluetooth can know when SCO connection needs to be established. Maybe we need to handle from Gecko audio module, let Gaia app call Gecko some audio api, which suggest Gecko audio path shall be changed to Bluetooth HFP and Gecko Audio internally informs Bluetooth module connectSco. But I really don't know who can make this decision.

Comment 10

[StevenLee[:slee]]

cc Randell, who is the tech lead of WebRTC.

Comment 11

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

This bug has been pending a while, but I think we shall ask UX first. Per Comment 6, I guess there is no use case to use bluetooth headset as the default input/output.

Comment 12

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Hi Jenny,
We currently don't have UX spec to define the use case about making WebRTC calls through bluetooth headset. Can you help on this part?

Comment 13

[Jenny Lee]

Hi Shawn,
Per discussion, for better UX, providing user option to choose which input/output source to use is needed. 
If the technical part is settled, I'd be glad to help with the spec. Thanks!

Comment 14

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Pass to HFP owner.

Comment 19

[Blake Wu [:bwu][:blakewu]]

As I can recall, a proposed solution discussed in Orlando work week is to use media element's sinkID[1] to pass the correct type and let audio path switch to BT.  


[1]https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/sinkId

Comment 20

[BMO Automation]

Firefox OS is not being worked on