Last Comment Bug 654787 - Looping <audio> files should be seamless
: Looping <audio> files should be seamless
Status: REOPENED
[games:p2]
:
Product: Core
Classification: Components
Component: Audio/Video: Playback (show other bugs)
: unspecified
: x86 Mac OS X
: -- normal with 10 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
http://forestmist.org/2010/04/html5-a...
Depends on: 782507 449157 664918
Blocks: gecko-games 1014243
  Show dependency treegraph
 
Reported: 2011-05-04 11:37 PDT by huskyr@gmail.com
Modified: 2015-11-05 10:47 PST (History)
24 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
backlog


Attachments
testcase - 440Hz sine (187 bytes, text/html)
2011-11-20 18:43 PST, Matthew Gregan [:kinetik]
no flags Details
loop.tar.gz (317.59 KB, application/gzip)
2014-05-16 07:33 PDT, Paul Adenot (:padenot)
no flags Details
loop.tar.gz (318.07 KB, application/gzip)
2014-05-16 07:39 PDT, Paul Adenot (:padenot)
no flags Details

Description huskyr@gmail.com 2011-05-04 11:37:06 PDT
User-Agent:       Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Build Identifier: 4.0.1

When looping an <audio> file there's a small, audible, gap when re-starting the audio file from the beginning. 

This makes it impossible to make something like an audio sequencer or drum loops.

Currently the 'loop' HTML5 property is not implemented (see #449157), but Javascript like this should have the same effect:

<audio id="audio" src="loop.wav" loop autoplay controls />

<script>
document.getElementById('audio').addEventListener('ended', function() {
    this.currentTime = 0;
}, false);
</script>

Reproducible: Always
Comment 1 Matthew Gregan [:kinetik] 2011-05-04 16:17:22 PDT
The current plan is to fix bug 449157 in such a way that the looping is seamless.  I'm not sure it's feasible to provide (or, at least, guarantee) seamless looping when script sets currentTime to the start of the media.
Comment 2 huskyr@gmail.com 2011-05-05 14:56:17 PDT
Great, thanks for taking that into account. I'm just wondering: is there any technical difference in setting 'loop' to  true or giving a 'currentTime = 0' on an ended event? Seems the same to me...
Comment 3 Matthew Gregan [:kinetik] 2011-05-05 15:17:45 PDT
There are, I forgot to mention them earlier.  One is that the loop attribute allows the decoder to know in advance that the media is looping, so it can prepare to handle that by not tearing down resources at the end of playback (such as the decode threads, audio device, etc.)

The second is that your example sets currentTime in an ended event handler.  The ended event is dispatched after playback ends, which is dispatched after we have drained and closed the audio device.  The event handler runs asynchronously on the main thread, so (in addition to the tearing down resources issue mentioned above) there is some delay between playback ending where the event is dispatched and the time the event handler runs.

To allow seamless looping, the ended event would need be to be dispatched early enough that the event handler would set currentTime at the correct moment, which given the indeterminate delays between the event dispatch and event handler running, isn't really possible.

Having said that, there are probably improvements that can be made to reduce the delay between dispatching the ended event and starting playback again.  It's just that it's not possible to guarantee anything close to seamless looping without the loop attribute.
Comment 4 Timothy B. Terriberry (:derf) 2011-05-05 15:43:53 PDT
(In reply to comment #3)
> It's just that it's not possible to guarantee anything close to seamless
> looping without the loop attribute.

Note also that when using lossy compression (such as Vorbis), even if the first sample of the file immediately follows the last, there may be glitches caused by differing quantization used on the two blocks. For truly seamless looping, you want to crosslap the end of the file with the beginning using the MDCT windows, which may require specially prepared files for best results (see http://xiph.org/vorbis/doc/vorbisfile/crosslap.html for details). This is something I expect would be done if the loop attribute is present, but may not be the behavior you would want on a normal seek.
Comment 5 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-05-10 15:49:26 PDT
Based on comment 1 I think it would be safe to add a dependency to bug 449157 or dupe it.
Comment 6 Matthew Gregan [:kinetik] 2011-05-10 15:54:46 PDT

*** This bug has been marked as a duplicate of bug 449157 ***
Comment 7 Matthew Gregan [:kinetik] 2011-11-20 18:07:23 PST
Reopening this to track seamless looping.  Bug 449157 now tracks basic (non-seamless) looping support.
Comment 8 Matthew Gregan [:kinetik] 2011-11-20 18:43:25 PST
Created attachment 575792 [details]
testcase - 440Hz sine
Comment 9 Alexander Brüning 2013-08-19 12:00:35 PDT
This still seems to be an issue on Linux at least.
Comment 10 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2013-08-19 12:30:17 PDT
(In reply to Alexander Brüning from comment #9)
> This still seems to be an issue on Linux at least.

No patches have landed to resolve this issue yet.
Comment 11 Jukka Jylänki 2014-05-16 07:22:35 PDT
Hey, any progress here? We are getting bit by this issue as well.

It sure looks like audio loop scheduling implementation in audio.loop attribute does not even attempt to perform sample-precise joining, but plays the next instance of the audio loop to play back way too late after the first.

When I have an audio clip that looks like this: https://dl.dropboxusercontent.com/u/40949268/Bugs/firefox_audio_loop/Screen%20Shot%202014-05-16%20at%204.45.56%20PM.png

we are getting audio playback which when recorded back looks like this:

https://dl.dropboxusercontent.com/u/40949268/Bugs/firefox_audio_loop/Screen%20Shot%202014-05-16%20at%204.44.00%20PM.png

There is about 80msec gap in the scheduled playback between loops. I don't think this is a question about lossy compression - definitely the loop="true" attribute should in itself be able to schedule sample-precise joins of the audio data, and then it's the question of the audio data source on how well it sounds like. In that image, it looks like there's about 80 msecs of delay, which at a source audio data rate of 48kHz means that the audio buffer scheduling was off by some 3840 samples.

Here is a minimal STR with the loop="true" attribute: https://dl.dropboxusercontent.com/u/40949268/Bugs/firefox_audio_loop/audio_loop.html

Some suggested using the audio 'onended' event to fire the next loop. Here is a handwritten STR #2 for that workaround: https://dl.dropboxusercontent.com/u/40949268/Bugs/firefox_audio_loop/audio_loop_onended.html
which produces the same results. I agree with the earlier comments that this is not a good way to code, since the browser cannot prepare for this in advance.

I know that Web Audio API implements this properly, so at first I thought to use that, but I hit two obstacles:

  - currently we use HTMLAudioElements to load up audio files, so that they're loaded via browser WAV and OGG codecs. Looking here, http://updates.html5rocks.com/2012/02/HTML5-audio-and-the-Web-Audio-API-are-BFFs , it seems that <audio> elements integrate with Web Audio, but still, the playback api is not as flexible as when playing back AudioBufferSourceNodes? I was not able to start the playback in the Web Audio API graph and control looping nor manually schedule there. Is it possible to work around the looping some way with Web Audio API?

  - my other thought was to use Web Audio API with AudioBufferSourceNodes, but because of the above limitation, that means I would have to implement WAV and OGG codecs in JavaScript, just to be able to drive the looping from the Web Audio API graph. Or is there some other way to get the raw audio buffer data from <audio> so that one could use the same API as AudioBufferSourceNodes to play back?

Are there known workarounds for this issue?
Comment 12 Paul Adenot (:padenot) 2014-05-16 07:33:02 PDT
Created attachment 8423887 [details]
loop.tar.gz

Web Audio API is perfectly suitable for this, see the attached demo (unzip, untar and run loop.html, the audio sample is included). I threw in a play/pause button, as it seemed to be a needed feature.

Note that you can put any codec that works in <audio> in the decodeAudioData call, it'll get decoded for you by the Web Audio API.

Jukka, does that work for you?
Comment 13 Jukka Jylänki 2014-05-16 07:36:33 PDT
Thanks, I downloaded the file, but I wonder if it has the wrong file? It did not contain a file loop.html.
Comment 14 Paul Adenot (:padenot) 2014-05-16 07:39:02 PDT
Created attachment 8423890 [details]
loop.tar.gz

Oops, went too fast, this has the right files.
Comment 15 Jukka Jylänki 2014-05-16 08:01:02 PDT
Perfect, that workaround is working great! Thanks Paul!
Comment 16 Jukka Jylänki 2014-05-20 02:55:16 PDT
Testing more, I am hitting this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1012801

How difficult would this bug be to fix? (I expect between this and 1012801, this bug would be the easier one to tackle?) I seem to be trading bugs now when trying to find a workaround.
Comment 17 Preeti Raghunath(:Preeti) 2014-05-27 10:17:53 PDT
Moving this to backlog.

This is too late to take in 1.4
Comment 18 Erin Lancaster [:elan] 2014-05-29 10:30:59 PDT
Note: For games we need either this bug fixed or Bug 1012801 to be able to do seamlessly looping audio. Likely target is 2.0.

Note You need to log in before you can comment on or make changes to this bug.