Closed Bug 674080 Opened 13 years ago Closed 11 years ago

Simple multishot API to playback sounds

Categories

(Core :: Audio/Video, defect)

Product:
Core
Component:
Audio/Video
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED DUPLICATE of bug 779297

People

(Reporter: cjones, Unassigned)

References

(Depends on 1 open bug)

Details

Everything our games need is a reliable way to play back background music and multishot plenty of sound effects on top. Essentially, we want Audio.play(url), an extremely simple API that only does one thing, but does it well: Play the sound at the given url. In the background, the browser needs to optimize, cache the sound for multiple shots and intelligently allocate and de-allocate channels. Along to that, we'd want an API that tells us "your sound is ready for playback at the lowest latency. Go for it.". This could be Audio.load(url), that either returns a handle to which I can bind events, or simply comes with a callback.
(In reply to comment #0)
> Everything our games need is a reliable way to play back background music
> and multishot plenty of sound effects on top.

I totally agree.

> Essentially, we want
> Audio.play(url), an extremely simple API that only does one thing, but does
> it well: Play the sound at the given url. In the background, the browser
> needs to optimize, cache the sound for multiple shots and intelligently
> allocate and de-allocate channels.

I don't think you actually want the API to be this simple. What happens when you call Audio.play(url) the first time? The browser goes and fetches the URL, causing a delay before the audio is played, which is not what you want.

> Along to that, we'd want an API that
> tells us "your sound is ready for playback at the lowest latency. Go for
> it.". This could be Audio.load(url), that either returns a handle to which I
> can bind events, or simply comes with a callback.

We have this. You've just described the HTML5 audio element. ;)

Particularly relevant to your request here is the "canplaythrough" event which fires on the media element when the resource has loaded enough such that if the download proceeds at the current rate, we estimate that we'll be able to play through without playback needing to stop to buffer. Unfortunately this event not fired correctly in Chrome, so this isn't reliably interoperable. They know about this.

To handle your two cases you mention above the with existing API:

1. Play background music:
  var music = new Audio(musicUrl).play();

You'd acutally need to use HTMLMediaElement.canPlayType() to ensure the browser can play Ogg/WebM/Mp3/AAC or whatever, so it's not quite *that* simple. *sigh*

2. Multishot audio:
  var shortSound = new Audio(shot);
  // ... Some time later, every time we want to start playing an instance of
  // the sound (ideally after the "canplaythrough" event has fired on the element).
  shortSound.cloneNode().play();

cloneNode().play() will clone the audio node and share its cached data. It *should* be pretty fast, and you can easily play as many of these as you like. There's no need to track references to the clones either, it will GC naturally once the sound finishes. This cloneNode functionality is being refined in Bug 668449 to ensure we're not creating new connections - we currently don't share cached data on clones, so we re-download the resource, so this approach isn't yet low latency.

Once Bug 668449 is fixed (new hire Ralph Giles I think is slated to work on it), this second case should be low latency. Then we should start publicizing this approach loudly and strongly.

cjones: Does that seem sufficient?
(In reply to comment #1)
> (In reply to comment #0)
> > Everything our games need is a reliable way to play back background music
> > and multishot plenty of sound effects on top.
> 
> I totally agree.

Hi Chris, I'm the author of the issue, so I'm going to answer below :)

> 
> > Essentially, we want
> > Audio.play(url), an extremely simple API that only does one thing, but does
> > it well: Play the sound at the given url. In the background, the browser
> > needs to optimize, cache the sound for multiple shots and intelligently
> > allocate and de-allocate channels.
> 
> I don't think you actually want the API to be this simple. What happens when
> you call Audio.play(url) the first time? The browser goes and fetches the
> URL, causing a delay before the audio is played, which is not what you want.

Yeah, good point. I agree.

> 
> > Along to that, we'd want an API that
> > tells us "your sound is ready for playback at the lowest latency. Go for
> > it.". This could be Audio.load(url), that either returns a handle to which I
> > can bind events, or simply comes with a callback.
> 
> We have this. You've just described the HTML5 audio element. ;)

This request is born out out of despair when working with mobile devices. On both Android and iOS, it is currently unbearable to instantiate new audio elements for every triggered sound effect. Now, I'd be happy if that wasn't the case - if both Apple and Google can make instantiating trivial and not perf hungry, I'd be fine to use whatever spec they give me.

I think what we really need is a spec that has success criteria in the form of achieved performance goals. This goes as far as to the W3C level, might be something for the audio group to think about.

> 
> Particularly relevant to your request here is the "canplaythrough" event
> which fires on the media element when the resource has loaded enough such
> that if the download proceeds at the current rate, we estimate that we'll be
> able to play through without playback needing to stop to buffer.
> Unfortunately this event not fired correctly in Chrome, so this isn't
> reliably interoperable. They know about this.

Yeah, we're not using it because of two different issues. First, as you mentioned, it's broken in most browsers today. Second, we have to resort to using sound sprites today for runtime performance reasons (new Audio init time) and load optimization (too many http requests at startup for preloading sound effects). If we can fix both issues in another way, I'd be happy with whatever you give me. A beautiful spec is very desired, but I'd be happy to take the worst spec if it solves our perf bottlenecks.

> 
> To handle your two cases you mention above the with existing API:
> 
> 1. Play background music:
>   var music = new Audio(musicUrl).play();
> 
> You'd acutally need to use HTMLMediaElement.canPlayType() to ensure the
> browser can play Ogg/WebM/Mp3/AAC or whatever, so it's not quite *that*
> simple. *sigh*

We can live with that. It's a mess, but we're Zynga - we don't care if we encode the **** out of the files, into 20 different codecs. This is really bad for indie developers though, that don't have the underlying horsepower to host and convert plenty of different files, but that's a different discussion :)

> 
> 2. Multishot audio:
>   var shortSound = new Audio(shot);
>   // ... Some time later, every time we want to start playing an instance of
>   // the sound (ideally after the "canplaythrough" event has fired on the
> element).
>   shortSound.cloneNode().play();
> 
> cloneNode().play() will clone the audio node and share its cached data. It
> *should* be pretty fast, and you can easily play as many of these as you
> like. There's no need to track references to the clones either, it will GC
> naturally once the sound finishes. This cloneNode functionality is being
> refined in Bug 668449 to ensure we're not creating new connections - we
> currently don't share cached data on clones, so we re-download the resource,
> so this approach isn't yet low latency.

Ohh shiny. This is exactly what I'm talking about in one of the earlier sentences. If you make this work, we can close this issue ('make this work' refers to Android as well, but might require filing the bug on the Android bugtrackers I guess?). Looking forward to this!

> 
> Once Bug 668449 is fixed (new hire Ralph Giles I think is slated to work on
> it), this second case should be low latency. Then we should start
> publicizing this approach loudly and strongly.
> 
> cjones: Does that seem sufficient?

Yeah, awesome. Maybe you can link 668449 to this issue as a dependency, so I know when I can use the audio API as expected. Just one piece of the puzzle is left - load optimization. With a hundred sound effects, this still requires a hundred http requests at load time. Say you have an arcade shooter and start right into the game, with lots of creeps approaching your spaceship. There's no time to do deferred loading of sounds when they need to be applied - you need to preload all of them to trigger them with low latency.

That reminds me - it seemed to me there's no good way to preload a sound today without triggering play() and then stop() right after, which seems unnatural. I want to be able to precache the resource but not play it at load time.

Hope this helps!
(In reply to comment #2)
> That reminds me - it seemed to me there's no good way to preload a sound
> today without triggering play() and then stop() right after, which seems
> unnatural. I want to be able to precache the resource but not play it at
> load time.

Have you looked at the 'preload' attribute? Does that do what you want if set to 'auto'?
(In reply to comment #3)
> (In reply to comment #2)
> > That reminds me - it seemed to me there's no good way to preload a sound
> > today without triggering play() and then stop() right after, which seems
> > unnatural. I want to be able to precache the resource but not play it at
> > load time.
> 
> Have you looked at the 'preload' attribute? Does that do what you want if
> set to 'auto'?

I'm pretty sure it doesn't work in most browsers, but I can check again in Firefox and see if it works. Will also let Chris Martens chime in, who mostly worked on this part.
(In reply to comment #4)
> (In reply to comment #3)
> > (In reply to comment #2)
> > Have you looked at the 'preload' attribute? Does that do what you want if
> > set to 'auto'?
> 
> I'm pretty sure it doesn't work in most browsers, but I can check again in
> Firefox and see if it works. Will also let Chris Martens chime in, who
> mostly worked on this part.

I think Chrome may still support the old "autobuffer" attribute, which does the same thing as preload="auto". "autobuffer is the old specification, and Chrome hasn't updated to the current spec in this area.
(In reply to comment #2)
> Just one piece of the puzzle
> is left - load optimization. With a hundred sound effects, this still
> requires a hundred http requests at load time.

Firefox limits the number of connections to any one server to 15 (the value of the network.http.max-connections-per-server pref) so is this actually a problem? When the max connections per server is reached the pending connections in the other media should wait their turn. I don't know if other browsers do this.

One guy at Opera concatenated all his sounds into one file and seeks in the file when he wants to play it:
http://my.opera.com/emoller/blog/2011/06/13/emberwind-week-1-flash-free-audio

Not an ideal solution, but it's about the only way you could reduce the number of HTTP connections required; by reducing the number of physical files.
If the files are small you can embed them in the HTML directly using data URL's.
(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > (In reply to comment #2)
> > > Have you looked at the 'preload' attribute? Does that do what you want if
> > > set to 'auto'?
> > 
> > I'm pretty sure it doesn't work in most browsers, but I can check again in
> > Firefox and see if it works. Will also let Chris Martens chime in, who
> > mostly worked on this part.
> 
> I think Chrome may still support the old "autobuffer" attribute, which does
> the same thing as preload="auto". "autobuffer is the old specification, and
> Chrome hasn't updated to the current spec in this area.

Thanks for the info. We'll try it out.
(In reply to comment #6)
> (In reply to comment #2)
> > Just one piece of the puzzle
> > is left - load optimization. With a hundred sound effects, this still
> > requires a hundred http requests at load time.
> 
> Firefox limits the number of connections to any one server to 15 (the value
> of the network.http.max-connections-per-server pref) so is this actually a
> problem? When the max connections per server is reached the pending
> connections in the other media should wait their turn. I don't know if other
> browsers do this.

This works when you're on a fast connection. Thinking about Firefox mobile, it's an absolute no-go. HTTP roundtrips are very costly on bad 3G networks.

> 
> One guy at Opera concatenated all his sounds into one file and seeks in the
> file when he wants to play it:
> http://my.opera.com/emoller/blog/2011/06/13/emberwind-week-1-flash-free-audio
> 
> Not an ideal solution, but it's about the only way you could reduce the
> number of HTTP connections required; by reducing the number of physical
> files.

We are doing this today. It's not ideal, because it stops us from using canplaythrough (even when canplaythrough fires, you can't be sure you can jump to the last cue in the sprite). Additionally, it's difficult to multishoot many simple sounds - you gotta clone the whole sprite over and over.
(In reply to comment #7)
> If the files are small you can embed them in the HTML directly using data
> URL's.

We have given this some thought as well but haven't tried it yet. Might work.
There are really two problems here:
1) Load lots of small resource assets efficiently
2) Trigger playback of those assets with low latency

Here are a couple of solutions for #1:
a) SPDY or some evolution thereof. In Chrome, will probably be in Firefox soon. Will be a while before it's standardized and available cross-browser.
b) Use XHR to load the resources packed into a single stream, break up the data into Blobs, use document.createObjectURL() to mint URIs that you can use with media elements. Optionally, cache data in localStorage or IndexedDB. Should work in Firefox, Opera, Chrome today.

For #2, having a set of <audio preload src=...> elements and then doing audioElem.cloneNode().play() is a perfectly adequate API. The fact that it's poorly implemented today doesn't justify creating a new API that might be just as poorly implemented.

We need to fix bug 703379 in Gecko, and probably other bugs.
No longer depends on: 703379
Depends on: 703379
Depends on: 782507
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #11)
> We need to fix bug 703379 in Gecko, and probably other bugs.

Given bug 703370 is fixed and we are tackling the webaudio api, is this bug obsolete now?
I think we should close this when enough of the Web Audio API is landed, if not before.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Oops
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
@Robert I agree. This makes sense.
I'm just going to dup this to the Web Audio API bug.
Status: REOPENED → RESOLVED
Closed: 12 years ago → 11 years ago
Resolution: --- → DUPLICATE
Note that Web Audio is somewhat crippled in iOS Safari, similar to the way they crippled the <audio> element. But it doesn't make sense to keep coming up with new audio APIs in the hope that Apple won't cripple the next one :-).
IMHO it's good enough on iOS Safari – the only remaining issue is the user activation. I consider this ticket fixed if a stable Web Audio API has landed in Firefox (as the spec itself pretty much requires it to be high perf, low latency).
You need to log in before you can comment on or make changes to this bug.