615946 - Implement Audio Workers

Status: RESOLVED INCOMPLETE

(Core :: Web Audio, enhancement)

Description

[David Humphrey (:humph)]

Based on discussions over the summer about how to improve the Audio Write capabilities of the Audio API, Ben Turner wrote a first pass at a patch to support Worker Threads talking directly to the Audio Decoder.  From an email he sent:

----------------------------------------------------------
To use, specify the worker js file in the html like so:

    <audio id="audio"
           src="sound.ogg"
           mozDecodeWorkerURL="audioWorker.js">
    </audio>

Then your "audioWorker.js" file needs to look something like this:

    var volume = 1;

    function ondata(data) {
      dump(volume + "\n");
      for (var i = 0; i < data.length; i++) {
        data[i] *= volume;
      }
    }

    function onmessage(message) {
      volume = message
    }

Your can specify an "ondata" function to process SoundData arrays as they come in, and you can specify an "onmessage" function to receive structured cloned data from the main thread. To communicate with the audio worker you do something like this on the main page:

    <form>
      <label>Volume
        <input id="volume" type="text" oninput="changeVolume(event);" value="1">
      </label>
    </form>

    <script>
      var audio = document.getElementById("audio");

      function changeVolume(event) {
        var volume = parseFloat(event.target.value);
        if (!isNaN(volume)) {
          audio.mozDecodeWorkerPostMessage(volume);
        }
      }

    </script>
 
The integration with the audio element is rough, to say the least. There are definitely some fun use-after-free errors lurking here due to the way I hooked element destruction, and I'm sure the media team could do a much better job. Hopefully this will point you all in the right direction.
----------------------------------------------------------

I've been tied up with other work, and waiting to see what would happen with the API, Firefox 4, and Chrome's API, so haven't done any more with this.  However, smaug suggests we try this regardless.  I'll post the 3 patches next.

Comment 1

[David Humphrey (:humph)]

Attachment: Audio patch (apply 1st)

This is unlikely to work against the API in its present form, and will need to get updated to the shipping version.

Comment 2

[David Humphrey (:humph)]

Attachment: Worker patch (apply 2nd)

Comment 3

[David Humphrey (:humph)]

Attachment: postMessage patch (apply 3rd)

Comment 4

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Instead of mozDecodeWorkerURL, I think a better API would let us associate one or more workers with a media element. E.g.,

HTMLMediaElement:
  void addAudioListener(Worker);
  void removeAudioListener(Worker);
HTMLVideoElement:
  void addVideoListener(Worker);
  void removeVideoListener(Worker);

This would allow a single Worker to process multiple streams and do other things too. These APIs would cause mozAudioAvailable and mozVideoAvailable events to be delivered to the Worker.

Somehow you'd also have to pass over an object reference corresponding to an HTMLMediaElement that lets the Worker do mozWriteAudio.

Comment 5

[Olli Pettay [:smaug][[e-mail]]]

(In reply to comment #4)
> Somehow you'd also have to pass over an object reference corresponding to an
> HTMLMediaElement that lets the Worker do mozWriteAudio.
Somehow wrap HTMLMediaElement inside some thread safe object and make
sure that HTMLMediaElement is used only in the main thread.

Comment 6

[Olli Pettay [:smaug][[e-mail]]]

We could have some interface for mozWriteAudio. Then HTMLMediaElement could
implement that interface, but also something in worker.
I'm hoping we wouldn't need to pass the data to the main thread at all.

Perhaps then in the worker context there could be mozAudioStreams array.
That would contain all the audio objects connected to the worker.
Each item in the array would also implement EventTarget so that
one could use exactly the same API in DOM and in worker context.

Comment 7

[Olli Pettay [:smaug][[e-mail]]]

interface WorkerData : Worker
{
  attribute DataStream[] dataStreams;
}

interface DataStream : EventTarget
{
  // This could contain whatever json data is sent from main thread,
  // for example sample rate
  attribute Object json;

  attribute unsigned long long streamOffset;

  long writeData(jsval aData);
}

DataAvailable event would be dispatched to DataStream.


Then audio/video elements could have something like
.connectAudioData(<someworker>, json_message);
.connectVideoData(<someworker>, json_message);

Canvas could have
.connectImageData(<someworker>, json_message)
.notifyImageData(long x, long y,
                 unsigned long w, unsigned long h)


Would need to prototype some API and bring it to WhatWG.

Comment 8

[Olli Pettay [:smaug][[e-mail]]]

Attachment: DataBroker

Something like this would be rather generic.

Comment 9

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 517527 [details]
DataBroker

>  // The data will be written to DataBroker.
>  void connect(in DataBroker aBroker);
Actually this version of connect() shouldn't be needed.


There is atm some discussion to pass ArrayBuffer
to workers, but that would still cause some processing in main thread.
I'd like to have something which allows implementation to keep all (except
perhaps the initial setup) audio processing out of main thread.

Comment 10

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 517527 [details]
DataBroker

Hmm, I wonder how to transfer data from one worker to another without allocating so much all the time, yet keeping the data easy and fast to use in JS.

Perhaps data event should have a pointer to some object
which contains the data and which has also some method to mark
data reusable.
interface DataBuffer
{
  readonly attribute jsval buffer;
  readonly unsigned long long length; // Currently in-use of buffer
  readonly boolean locked;
  // After calling this implementation may or may not reuse .buffer.
  // It is just a hint for the implementation.
  void unlock();
}


Then in the example
dataBrokers["input"].ondata =
  function(event) {
    var buffer = event.data.buffer;
    // process effect here by modifying buffer
    // ...
    dataBrokers["output"].writeData(buffer);
    event.data.unlock();
  }

This all would just help with allocating/deallocating, not with
memcopy. Sharing the same objects between workers just isn't possible
(nor really should be, IMO.)

Comment 11

[David Humphrey (:humph)]

Probably wise to consider this in the context of roc's work here: https://wiki.mozilla.org/MediaStreamAPI

Comment 12

[Yury Delendik (:yury)]

Attachment: Audio object for the Web Worker (same interface as in Bug 490705)

(In reply to comment #6)
> We could have some interface for mozWriteAudio. Then HTMLMediaElement could
> implement that interface, but also something in worker.
> I'm hoping we wouldn't need to pass the data to the main thread at all.

Enables the audio write functionality that is equivalent to the functionality implemented in Bug 490705.

Comment 13

[David Humphrey (:humph)]

http://jsmad.org/ would be a good use case for this.

Comment 14

[Maire Reavy [:mreavy]]

Work is occurring in the WG now to spec this out; padenot is heavily involved