1388586 - Implement the sampleRate microphone constraint

Status: NEW

(Core :: WebRTC: Audio/Video, enhancement, P3)

Description

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Spec[1]: "The sample rate in samples per second for the audio data."

[1] https://w3c.github.io/mediacapture-main/getusermedia.html#def-constraint-sampleRate

Comment 1

[Dan Minor [:dminor]]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #0)
> Spec[1]: "The sample rate in samples per second for the audio data."
> 
> [1]
> https://w3c.github.io/mediacapture-main/getusermedia.html#def-constraint-
> sampleRate

Does this mean we need to resample to any arbitrary sample rate provided by the user? There are assumptions in the webrtc.org code that sample rate is one of 8000, 16000, 32000, 44100 or 48000. Or am I misunderstanding the intent of this constraint?

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

I suspect if they ask for an exact rate that we don't support internally we can just return overconstrained ;-)

Comment 3

[Dan Minor [:dminor]]

For instance, the DTMF code assumes a fixed set of sample rates [1]. That's just the tone generation code which I'm sure could be redone in a more readable if perhaps less efficient way, but there might be other places in the code base that make similar assumptions.

[1] http://searchfox.org/mozilla-central/rev/e5b13e6224dbe3182050cf442608c4cb6a8c5c55/media/webrtc/trunk/webrtc/modules/audio_coding/neteq/dtmf_tone_generator.cc#38

Comment 4

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

As jesup touches on, the constraints model, and the spec, doesn't mandate any particular output at all, it just sets up the mechanism to query or discover what a mic can output. I.e. it doesn't mean we have to support arbitrary values. E.g. we could limit to outputting 8000, 16000, 32000, 44100 and 48000 if we find internal limitations, and still be spec compliant.

That said, I believe the plan is to support whatever cubeb can give us. We're also adding the AudioContext({sampleRate}) JS option (bug 1387454) which AFAIK might mean that peer connection needs to handle inputs of arbitrary sample rates. I don't know. cc Paul.

In either case, we should be sure to test what breaks. Thanks for the heads up!

Comment 5

[Paul Adenot (:padenot)]

Cubeb will get you anything in [8000, 192000] on any platform, for output or input, it just gives you what you ask for. 

We can resample anything to anything, at any point, so it's just a matter of making a decision.

It's really unclear what this constraint is for, because the only way to look into a MediaStream is to use the Web Audio API, and the Web Audio API will resample all inputs to its own sample-rate. One of the way to observe this is to notice that the high partials will be cut off because of the lower sample-rate I suppose.

Comment 6

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Reminder to check the privacy.resistFingerprinting pref as well as bug 1528042 when implementing this feature, since it might expose system sample rates through track.getSettings().

Comment 8

[mat.reiner]

(In reply to Paul Adenot (:padenot) from comment #5)

Cubeb will get you anything in [8000, 192000] on any platform, for output or
input, it just gives you what you ask for.

We can resample anything to anything, at any point, so it's just a matter of
making a decision.

It's really unclear what this constraint is for, because the only way to
look into a MediaStream is to use the Web Audio API, and the Web Audio API
will resample all inputs to its own sample-rate. One of the way to observe
this is to notice that the high partials will be cut off because of the
lower sample-rate I suppose.

Firefox does not support resampling MediaStream to an AudioContext (domexception: AudioContext.createMediaStreamSource: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported)
That I could find, it is currently not possible to reliably read raw data from microphone, since the sampleRates must match but there is no way to know the microphone's sampleRate
(This is actually seriously frustrating, MediaStream is a joke 😠)