1029554 - Need a way to send the MSG's mixed audio output to (privileged) JavaScript

Status: NEW

(Core :: Audio/Video: MediaStreamGraph, defect, P3)

Description

[Paul Adenot (:padenot)]

This can be useful for testing.

With bug 989921, we get the ability to register arbitrary AudioMixer consumers, so we can send a copy of the data to JavaScript. This is very useful, because we will get exactly "what the person will hear". It can also help writing regression tests, for example for bug 1015519.

This could be something like:

partial interface Navigator {
  callback MixedDataAvailable = void (Float32Array or Uint16Array);
}

I'm happy to mentor someone on this one, although feel free to grab it if you need it for testing (but I can still help you figuring out AudioMixer stuff :-)).

Comment 1

[Nick Lebedev [:nl]]

Attachment: Add method to priviliged javascript that allows to receive mixed audio output data

Comment 2

[Nick Lebedev [:nl]]

Hi Paul,

could you check the WIP patch and clarify a few questions about this task and also let me know if i generally moving in right direction :) :

1. What is the good way to tie the navigator callback and the AudioDataJsExposer class? So that AudioDataJsExposer could invoke the callback anytime. 

2. Is it ok for AudioDataJsExposer to have the same lifetime as the AudioMixer class?

3. How to properly make the SetMixedDataAvailableHandler available only from the priviliged js ? via CheckPermissions attribute?

Comment 3

[Paul Adenot (:padenot)]

(In reply to Nick Lebedev [:nl] from comment #2)
> Hi Paul,
> 
> could you check the WIP patch and clarify a few questions about this task
> and also let me know if i generally moving in right direction :) :
> 
> 1. What is the good way to tie the navigator callback and the
> AudioDataJsExposer class? So that AudioDataJsExposer could invoke the
> callback anytime. 

I think we have to do it like this:
- When you set a callback in the navigator object, you send a message to the MediaStreamGraph (MediaStreamGraph::GetInstance()), effectively telling it to start sending audio data to the main thread. You also pass in a pointer to the AudioDataJSExposer, which has just been created. This is done using messages: http://mxr.mozilla.org/mozilla-central/source/content/media/MediaStreamGraph.cpp#1545, so we don't have to use locks. In practice, you need to implement a class that is an AudioCallbackReceiver, and call AddCallback on the mMixer (http://mxr.mozilla.org/mozilla-central/source/content/media/MediaStreamGraph.cpp#2738). This will register your new callback to the mixer, so it calls you back when there is data.

- When the mixer finishes mixing, it calls all the callbacks that have been set. When your new AudioCallbackReceiver callback is called, it should _copy_ the data into its own buffer, create a custom nsRunnable (let's say AudioMixerDispatchEvent), pass in the audio data and the pointer to the AudioDataJSExposer, and dispatch this runnable to the main thread (NS_DispatchToMainThread).

- When it is run on the main thread, you simply call the js function, passing the ArrayBuffer.

- I believe all the pointers to audio data should be UniquePtr, that should simplify lifetime issues.

> 2. Is it ok for AudioDataJsExposer to have the same lifetime as the
> AudioMixer class?

It should only be created if the user sets a callback, and destroyed when the callback is unset.

> 3. How to properly make the SetMixedDataAvailableHandler available only from
> the priviliged js ? via CheckPermissions attribute?
[ChromeOnly], like so: http://mxr.mozilla.org/mozilla-central/source/dom/webidl/CanvasRenderingContext2D.webidl#215

Comment 4

[Paul Adenot (:padenot)]

Comment on attachment 8488116 [details] [diff] [review]
Add method to priviliged javascript that allows to receive mixed audio output data

Review of attachment 8488116 [details] [diff] [review]:
-----------------------------------------------------------------

::: content/media/AudioMixer.h
@@ +52,5 @@
>        mChannels(0),
>        mSampleRate(0)
> +  {
> +    mJsExposer = new AudioDataJsExposer();
> +    mCallbacks.insertBack(new MixerCallback(mJsExposer));

This has to be lazy initialized when the callback is set. It should also probably live in the Navigator.

::: dom/webidl/Navigator.webidl
@@ +393,5 @@
> +callback MixedDataAvailable = void (ArrayBufferView array);
> +
> +partial interface Navigator {
> +  [Throws]
> +  void setMixedDataAvailableHandler (MixedDataAvailable callback);

The usual way to do this is an onmixeddataavailable handler, see: http://mxr.mozilla.org/mozilla-central/source/dom/webidl/AudioBufferSourceNode.webidl#29, for an example.

Comment 5

[Paul Adenot (:padenot)]

Once we have bug 1156472, we have this for free: it will just be a matter of, from chrome code, setting the pref to not prompt, and use the browser capture, then pipe it to Web Audio.

Comment 6

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.