1731682 - Web Audio API decodeAudioData causes short playback freeze

Status: UNCONFIRMED

(Core :: Web Audio, defect)

Description

[maxligtenberg]

Attachment: decodeAudioData-freeze Audacity recording.zip

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36

Steps to reproduce:

On macOS:

1. Create a Web Audio API context.
2. Make an XMLHTTPRequest to retrieve an MP3-file.
3. Use Web Audio API's decodeAudioData to convert data.
4. Put the (converted) data in a BufferSourceNode and play buffer.
   Loop steps 2 - 4.

Online example:
https://www.evdh.net/DAD_investigations-master/
Example code:
https://github.com/sevdh/DAD_investigations

Actual results:

Occasionally the playback audibly freezes / glitches for a few ms. If a glitch occurs it is mostly during a decodeAudioData-process but can also happen during an XMLHTTPRequest.

See https://youtu.be/Aj1k9YdvD_Q for examples. Timecodes are in the description of the video. A zip-file is attached with a screenshot of an Audacity recording (audio recorded via SoundFlower) that shows the temporary freeze.

Expected results:

I expect that the playback does not hiccup and plays continuously.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[C.M.Chang[:chunmin] | PTO]

Hi Max, thanks for the report. Can this be reproduced when the audio is in a different format? Or it only happens when using mp3 ?

Comment 3

[maxligtenberg]

Hi Chang,

Thanks for your reply. I've tested .flac, .m4a (aac) and .ogg-files and they all have the same problem. Glitches / hiccups occur every once in a while mainly during decodeAudioData en sometimes during XHR.

Comment 4

[maxligtenberg]

Additional information:
We've tested it on Firefox on Windows and the problem does not occur on that platform.

Comment 5

[C.M.Chang[:chunmin] | PTO]

Hi Max, Would you mind testing this on a Linux machine?

Comment 6

[C.M.Chang[:chunmin] | PTO]

Paul, do you have an idea where we should start looking at it? The issue seems different from the bug 1731813 since it happens on all formats.

Comment 7

[Paul Adenot (:padenot)]

Reporter, please try to reproduce while taking a performance profile with the Firefox profiler with the "Media" preset, as shown in this video: https://blog.paul.cx/post/profiling-firefox-media-workloads/#the-media-preset (there are written instructions below the video as well).

You can then either upload the profile and give us the link (button on the top-right), or send it to me at <[e-mail]> (there is an option to save locally). Please have the checkbox "Include hidden threads" checked, so we can understand what's up.

The "raw data" from about:support can be useful to us as well, if you can attach that here.

Thanks!

Comment 8

[maxligtenberg]

Attachment: support-raw-data.txt

Comment 9

[maxligtenberg]

Performance profile:
https://share.firefox.dev/2YjyqFC

The "raw data" has been added as attachment.

Comment 10

[maxligtenberg]

And I'm not able to test it on a Linux machine unfortunately.

Comment 11

[Paul Adenot (:padenot)]

This works as intended, no performance issue there.

The code as it is cannot be made to work: it's relying on setTimeout, that doesn't have any scheduling guarantees. If you instead use the parameters of the start method on AudioBufferSourceNode, you will be able to achieve perfect playback. https://www.html5rocks.com/en/tutorials/audio/scheduling/ is a good introduction on the subject.

Comment 12

[Edwin van der Heide]

Dear Paul, we are aware that setTimeout is not accurate but that is not what's causing the problem in this context.

Did you have a look at the above referenced YouTube video that demonstrates the problem? (the link is given again at the bottom of this post)

The example code alternates between the playback of two sound files that overlap. For each sound an XHR and decodeAudioData are being executed.

The problem is that the playback of one buffer gets now and then interrupted by the decodeAudioData (and sometimes even the XHR) of a next buffer that is being prepared for playback.

The YouTube video that demonstrates the problem:
-> see https://youtu.be/Aj1k9YdvD_Q for examples. Timecodes are in the description of the video.

Working online example:
-> https://www.evdh.net/DAD_investigations-master/

Example code:
-> https://github.com/sevdh/DAD_investigations

Comment 13

[Edwin van der Heide]

Dear Paul, we are aware that setTimeout is not accurate but that is not related to the bug we are reporting.

The example code alternates between the playback of two sound files that partially overlap with each other. For each sound an XHR and a decodeAudioData are being executed.

The problem is that the playback of one buffer gets, (only) now and then, interrupted by the decodeAudioData (and sometimes even the XHR) of a the buffer that is being prepared for playback.

Please have a look at the video that demonstrates the problem (timecodes at which the hiccups occur are in the description of the video):
-> https://youtu.be/Aj1k9YdvD_Q

Working online example:
-> https://www.evdh.net/DAD_investigations-master/
(it sometimes takes some time a hiccup to occur).

Example code:
-> https://github.com/sevdh/DAD_investigations

Comment 14

[Paul Adenot (:padenot)]

I had a look, something strange is going on and I'm going to have a closer look.

Comment 15

[Edwin van der Heide]

Very happy to hear that!

I would guess that (aspects of) the decodeAudioData (and also the XHR) execution block either the 'regular audio generation process' of Firefox or the passing of audio from Firefox to the system. If so, it might be a threading or thread priority issue.

Let us know if and how we can help.

Comment 16

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:padenot, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 17

[Paul Adenot (:padenot)]

We're re-architecturing how real-time audio threads are in Firefox at the moment. I'll try again after this, as one of the goal of this re-architecturing is to reduce the number of context switches.

Comment 18

[Edwin van der Heide]

Very nice to hear this follow up!

Please let us know when it's time to help testing from our side.