668449 - Need to be able to play independent sounds with low latency

Status: RESOLVED FIXED

(Core :: Audio/Video, defect)

Description

[Robert O'Callahan (:roc) (email my personal email if necessary)]

See http://www.phoboslab.org/log/2011/03/multiple-channels-for-html5-audio for some background.

Many games have simple audio needs: they just want to be able to trigger playback of various independent sound samples with low latency. They often want to be able to play back multiple instances of the same sound resource simultaneously.

A natural way for Web authors to do this would be to write code like this:
  <audio id="sound1" src="..." preload oncanplaythrough="..."></audio>
  <audio id="sound2" src="..." preload oncanplaythrough="..."></audio>
  ... possibly hundreds more sounds ...
... wait for all elements to fire canplaythrough before starting the game ...
  function playSound(n) {
    document.getElementById("sound" + n).cloneNode().play();
  }

I think we can make that work well without introducing new audio APIs.

This bug is about
a) writing a benchmark that tests such behavior, in particular measuring the latency incurred for each playback request and checking that we don't skip when playing. This may require instrumentation of a browser build or even the OS, although it should be possible to do a crude test manually.
b) tracking other bugs that we need to fix to make that work well.

We definitely need to make sure that the cloned node shares the same resource in the media cache as the original node (and its other clones); we can usually do that by taking advantage of the section "it is already being downloaded for other reasons" of the HTML resource fetch algorithm: http://www.whatwg.org/specs/web-apps/current-work/complete.html#fetch

We will probably also need to arrange for buffered decoded data to be cloned, and possibly more of the decoder state.

We may also have to reduce other overhead. This definitely depends on not needing multiple threads per media element (bug 592833). We may have to optimize DOM event dispatch more when there are no listeners. Fortunately, dispatching events to elements not in the DOM should be quite optimizable since only listeners set on the element itself can fire.

Comment 1

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to comment #0)
> We definitely need to make sure that the cloned node shares the same
> resource in the media cache as the original node (and its other clones); we
> can usually do that by taking advantage of the section "it is already being
> downloaded for other reasons" of the HTML resource fetch algorithm:
> http://www.whatwg.org/specs/web-apps/current-work/complete.html#fetch

I forgot to mention that the hard part of this is already done. The mozLoadFrom API already has a way to make elements share the compressed media resource; we just need to do it automatically when URIs match.

Comment 2

[Ralph Giles (:rillian)]

Attachment: sketch of a test framework

I started implementing the example in the description. This version uses four audio elements sharing the same source. Clicking the 'PLAY' text with trigger each of them in succession. It mostly works, but there are clearly some delays, especially the first time. Maybe it would be more representative to keep a list of times and call them from a 60 Hz play loop.

I also notice the status lines returning to loaded, which means canplaythrough is firing again after play() method is invoked.

If four different source files are used (bark, drip, glass and sonar from the gnome-media package) drip hangs without reaching the ended state for nearly a second, and doesn't respond to a play() during that period. Repeatedly clicking on play() to create a loop, one can hear it dropping out every other cycle.

Next I'll try to get some numbers of dispatch latency and footprint so we can see what we're doing.

Comment 3

[Matthew Gregan [:kinetik]]

If you're testing on Linux, you will likely be running into bug 582513.

Comment 4

[Ralph Giles (:rillian)]

Aha, thanks. That's likely the cause of the drop-out I was seeing. I ran the tests on Ubunutu Maverick (10.10) i686.

Comment 5

[Paul Adenot (:padenot)]

I've ran some experiments on this, measuring the dispatch latency. I'll attach a revised version of Ralph file tomorrow.

Comment 6

[Paul Adenot (:padenot)]

Attachment: Track latency between play(), and actual play.

Decompress and run index.html, then click play.

You will get the latency between the play() call and the "play" event.

Comment 7

[cajbir (:cajbir)]

The test file in comment 6 triggers bug 666132. I don't think it affects the test, just noting it here.

Comment 8

[Chris Pearce [:cpearce (Not reading bugmail)]]

So I've been collecting feedback from authors of HTML5 games such as [1],[2],[3], and the general consensus is that we (still) have a lot of audio latency. Chris Double tells me the last time he ran his measurements they showed that most of the lag was initializing nsAudioStream. libcubeb should have much lower latency, so making that blocking this.

[1] http://www.webworks.dk/enginetest/
[2] http://www.cuttherope.ie/?html5audio=true
[3] http://playbiolab.com/

Comment 9

[Matthew Gregan [:kinetik]]

(In reply to Chris Pearce (:cpearce) from comment #8)
> libcubeb should have much lower latency, so making that blocking this.

For playback, but not for initialization, since that code is much the same as we currently have.  To fix that, we'd either need to have cached streams, or provide lightweight streams with software mixing so that we don't need to open an OS-level audio device for every playing element.  The latter makes the most sense to me, so I plan to head in that direction.

Comment 10

[Martin Best (:mbest)]

We think this will be resolved by MediaStream Processing API.

Comment 11

[Randell Jesup [:jesup] (needinfo me)]

I think this covers a lot more than the MediaStreams portion - I think it should be re-opened.  MediaStreams are only one part of solving this.

Comment 12

[Matthew Gregan [:kinetik]]

Yup.

Comment 13

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

HTML5 <audio> is irrelevant here; we just need the capability, whether it's through Web Audio or Media Streams/Media Stream Processing/etc.

Comment 14

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #13)
> HTML5 <audio> is irrelevant here; we just need the capability, whether it's
> through Web Audio or Media Streams/Media Stream Processing/etc.

True, although since HTML5 <audio> is a spec and the others aren't (yet), it makes sense to support this through HTML5 <audio>.

Comment 15

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Dropping to p3 in favor of Web Audio

Comment 16

[Kelsey Gilbert [:jgilbert]]

Is this something Eideticker can track, since Eideticker is HDMI-based?

Comment 17

[David Bruant]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #15)
> Dropping to p3 in favor of Web Audio
I'm not sure I understand the "in favor of Web Audio" part. The concern of this bug may apply to Web Audio as well, can't it?

Comment 18

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Now that we're supporting Web Audio, I think we should just close this bug. If we have bugs where Web Audio is high latency, they're probably platform-specific and should just be filed as specific bugs.