591976 - (microphone) Expose raw audio from the microphone to web content

Status: RESOLVED FIXED

(Core :: WebRTC, enhancement)

Description

[David Humphrey (:humph)]

Now that we have an audio data API, it would be great to be able to work with audio coming out of the mic on the local machine.  I've seen some reference to this idea on lists, but could find no bug.

In addition to being able to stream the audio from the mic into the browser to be played, I want to be able to manipulate it, send it to remote boxes, store it locally, whatever.  If we can get the raw audio data to the media element's audio data API, this is pretty easy now.

From talking with khuey on irc,

< khuey|class> we can just plug into the same framework we get camera data from
< khuey|class> and then you can do "moz-device:mic" to get a stream

Hopefully this bug can serve as a starting point for discussion on doing this after ff4 timing.

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Security is the tricky bit here.  For the camera work we're using a <video> with src="moz-device:camera" but that's not exposed to content directly.

Comment 2

[David Humphrey (:humph)]

Flash does something similar to what we are doing for our geolocation stuff, namely, prompting for access.

Comment 3

[John Drinkwater (:beta)]

Would it make sense to make this block ‘webapi’ and become part of the WebAPI work?

Comment 4

[Ralph Giles (:rillian)]

Yes and no. It's device access, and so falls under the WebAPI umbrella, so blocking 'webapi' makes sense.

It's also a prerequisite for the MediaStream API proposal which is part of the larger Audio API and WebRTC/Conferencing work going on. There's quite a bit of overlap with projects outside the broader mission of the WebAPI group, as I understand it.

Comment 5

[Fabricio Campos Zuardi]

A MediaStream looks like the way to go IMHO, like in this audio recorder example from:

http://dev.w3.org/2011/webrtc/editor/webrtc-20111004.html#examples-1

Or example 4 from:

https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/webrtc-integration.html

Anyways, I am not sure if I am on the correct mozilla bug or not, but I am trying to implement an audio recorder app for gaia/b2g/firefoxOS and trying to figure out the current blockers, looks like it would be possible to create such an app once the different borwsers implement/fix some webrtc bugs.

I am gathering hints and references here for now https://kippt.com/fabricio/audio-recorder-webapp-blockers

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

Covered by the getUserMedia API