1351291 - Audio FFT data delayed on mic input

Status: RESOLVED WORKSFORME

(Core :: Web Audio, defect)

Description

[uralzima]

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:45.0) Gecko/20100101 Firefox/45.0
Build ID: 20100101

Steps to reproduce:

I created an audio visualizer, based on FFT, following music from microphone input, where output of sound card is actually used, instead of microphone (configured in pulseaudio as 'monitor of sound card')


Actual results:

The data is very delayed. Visualizer looks ugly and unusable. 


Expected results:

The data must have less delay. In chrome it works perfectly. Personally, I like more firefox than chrome, but if this will not be resolved, I will be forced to recommend users use chrome instead.

Comment 1

[Alex Chronopoulos [:achronop]]

Can you please provide a link or a code sample that will help us to reproduce the case?

Comment 2

[uralzima]

https://github.com/ianharmon/visualizer

Comment 3

[Alex Chronopoulos [:achronop]]

Can you please check if it is a real bug?

Comment 4

[uralzima]

It is very easy to check.
Git clone that github repo, open index.html in chromium and firefox. 
Redirect output in pavucontrol to follow your output of sound card for both browsers.
Then start and stop music. You see, that in firefox visualiser levels go down with a 0.4 sec delay (after music was stopped), in comparison to chromium. Chromium reacts near immediately.

Comment 5

[uralzima]

I didn't checked it with an audio file, instead of mic input, maybe the delay will be same in firefox. I think it is something buffer-related, chromium have very low value and works real-time. I would like to see firefox to work realtime for both audio files and mic input, so I will recommend users use Firefox or Chromium, instead of chromium only.

Comment 6

[Paul Adenot (:padenot)]

I don't have a huge delay here, it's visually similar between Chrome and Firefox. I'm using PulseAudio 8, Ubuntu 16.04, Firefox 55. See the attached video, can you make a screencast ?

Comment 7

[Paul Adenot (:padenot)]

Attachment: bug1341291.mkv

Comment 8

[uralzima]

Uploaded here: https://www.sendspace.com/file/8zwaq9
Firefox 45.8.0. Disabling addons don't help. 
Intel i5 16GB Ram, cpu load < 40% while doing this.

Comment 9

[Paul Adenot (:padenot)]

Yeah, this is on a very old build (although still supported for two or three months). We've rewritten all this code since then. Use Firefox 52 or above. (current release or pre-release builds).

Also, you can get an optimized path, with improved (= reduced) latency, if you do this in your code:

> +++ b/js/visualizer.js
> @@ -37,7 +37,11 @@ function setup() {
>                              navigator.msGetUserMedia);
>  
>    if (navigator.getUserMedia) {
> -    navigator.getUserMedia({video: false, audio: true},
> +    navigator.getUserMedia({video: false, audio: {
> +      echoCancellation: false,
> +      mozNoiseSuppression: false,
> +      mozAutoGainControl: false
> +    }},
>        // success callback
>        function(stream) {
>          // initialize nodes

There are similar options for Chrome, and we are in the process of standardizing them (to remove the `goog` and `moz` prefix), but we're at the stage where we are trying to agree on how it should behave.

Without the patch above, Firefox will try to apply echo cancellation, noise suppression, gain control. Some of this add some latency and CPU load.

With those options passed, Firefox will bypass those processing steps, and directly do input device -> webaudio.

Let me know if those two things help !

Comment 10

[uralzima]

Thank you very much for suggestions. I tried your code, but the delay remained. 
About version, I have latest 'stable' version installed/compiled from portage on my gentoo. I have ~52 version available to compile. 

I am using that version, because most people use it and want the script to work with most-installed version. 
I'll try to compile version 52 tomorrow (takes very long) and will check.

Comment 11

[uralzima]

In Firefox 52 it is working much better, near real-time. But pulseaudio crashed several times and sound freezes while running it in firefox 52, so not usable for me. I am using system-wide pulseaudio 10. Don't know why, it is working well for rest apps. 

I hope 45 version will end soon and at least all m$**** users will have newer version. 
Some more questions: 
1. Do win users have some auto-updater? Or they will sit in installed version until reinstall?
2. I want to apply some semi-transparent canvas on top of hq video. When I tried importing frames into canvas, it was very slow. So I just put a canvas on top of video tag. It is working better, but system load is high and if I use FullHD - video freezes and becase slow. How to do this in best way? Does this approach get hardware-accelerated (I use nouveau without acceleration). I just need a semi-transparent canvas on top of a video.

Comment 12

[Paul Adenot (:padenot)]

(In reply to uralzima from comment #10)
> Thank you very much for suggestions. I tried your code, but the delay
> remained. 
> About version, I have latest 'stable' version installed/compiled from
> portage on my gentoo. I have ~52 version available to compile. 
> 
> I am using that version, because most people use it and want the script to
> work with most-installed version. 
> I'll try to compile version 52 tomorrow (takes very long) and will check.

This is not accurate. 52 is stable. 45 is the old extended support release version. 52 will also be the new extended support release.

Comment 13

[uralzima]

I have filled a bug report to Gentoo to unmask 52 version as stable.

Comment 14

[Paul Adenot (:padenot)]

(In reply to uralzima from comment #11)
> In Firefox 52 it is working much better, near real-time. But pulseaudio
> crashed several times and sound freezes while running it in firefox 52, so
> not usable for me. I am using system-wide pulseaudio 10. Don't know why, it
> is working well for rest apps. 
> 
> I hope 45 version will end soon and at least all m$**** users will have
> newer version. 
> Some more questions: 
> 1. Do win users have some auto-updater? Or they will sit in installed
> version until reinstall?

I think that all the versions we ship on windows to regular users auto-update. 

> 2. I want to apply some semi-transparent canvas on top of hq video. When I
> tried importing frames into canvas, it was very slow. So I just put a canvas
> on top of video tag. It is working better, but system load is high and if I
> use FullHD - video freezes and becase slow. How to do this in best way? Does
> this approach get hardware-accelerated (I use nouveau without acceleration).
> I just need a semi-transparent canvas on top of a video.

Nouveau is unfortunately very slow for now. I don't think you can reach the level of performance you want here using it (it also depends on the size of your screen in pixels). However I'm no specialist.

Using another chip/driver combo (open source intel or amd driver, or proprietary nvidia driver) will make this work better, and using another OS (OSX/windows) will allow the video decoding to be hw accelerated (depending on the codec).

Comment 15

[uralzima]

I know about nouveau being slow. My question was, if I am doing right or if any other ways exist in manipulating semitransparent objects on top of video?

Anyway, I am testing Firefox 52 last days and I like how it works, it uses even less resources, than chromium! 
So I am closing this. 
But for everyday usage I use torbrowser, because firefox by default is leaking webrtc ip address, etc... and torbrowser is still on 45 branch. Maybe once they will merge together :)

Good luck on your work! chrome sucks!