1375466 - WebAudio API latency

Status: UNCONFIRMED

(Core :: WebRTC: Audio/Video, defect, P5)

Description

[Z@]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36

Steps to reproduce:

I create Web Audio example where I directly connect input->output without any effects/processing. I need only audio, no video.

my example is here

https://jsfiddle.net/zlelik/xfq3ykp7/49/



Actual results:

Latency is very high 170-200 milliseconds.

I tried to use latency parameter like this (for 20 ms), but it did not help.

var p = navigator.mediaDevices.getUserMedia({ audio: { latency: 0.02, echoCancellation: false, mozNoiseSuppression: true, mozAutoGainControl: false} });


Expected results:

Expected latency is 20-30 ms and it should be configurable.

According to the official W3C specification https://www.w3.org/TR/webaudio/#latency they tell about 3-50 ms latency. 20-30 ms is what I need. I do not really need 3-5 ms.

Or this latency 200 ms is expected and nothing can be done?

Comment 1

[Alex Chronopoulos [:achronop]]

Latency constraint is not implemented yet. The only thing you can do is to remove all extra processing of the stream by keeping
echoCancellation: false, mozNoiseSuppression: false, mozAutoGainControl: false. When all three are off getUserMedia delivers the raw input which is the fastest you can get.

FYI you can read this blog post about those constraint: https://blog.mozilla.org/webrtc/fiddle-of-the-week-audio-constraints/

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

How do we perform here against e.g. Chrome?

Comment 3

[Z@]

(In reply to Jan-Ivar Bruaroey [:jib] from comment #2)
> How do we perform here against e.g. Chrome?

Thank, I will try. But I am afraid I will have this bug https://bugzilla.mozilla.org/show_bug.cgi?id=1367697

Could you please tell me, what is minimum latency I can expect in this example when I send input from the microphone to output directly?

In Google Chrome it looks the same.

Another question why in W3C specification (https://www.w3.org/TR/webaudio/#latency) they tell latency can be from 3-6 ms up to 20-50 ms latency? As I understand they mean 50 ms is maximum, but I have 200 ms, 4 times more. Does it mean that Firefox is not following W3C specification or they tell about different latency?

Comment 4

[Alex Chronopoulos [:achronop]]

It is on a windows platform, right? At least that is reported in the description. Paul can tell you about expected latency in that case and he is definitely familiar with the spec.

Comment 5

[Paul Adenot (:padenot)]

(In reply to Z@ from comment #3)
> Another question why in W3C specification
> (https://www.w3.org/TR/webaudio/#latency) they tell latency can be from 3-6
> ms up to 20-50 ms latency? As I understand they mean 50 ms is maximum, but I
> have 200 ms, 4 times more. Does it mean that Firefox is not following W3C
> specification or they tell about different latency?

This is non-normative, as indicated by the line "This section is non-normative." at the top of the text.

I would expect 30ms round trip latency on Windows 7 and up. I've seen some issue in the area, related to a fast path we have when disabling processing like echo cancellation.

I'll investigate this week.

Comment 6

[Z@]

(In reply to Paul Adenot (:padenot) from comment #5)
> (In reply to Z@ from comment #3)
> > Another question why in W3C specification
> > (https://www.w3.org/TR/webaudio/#latency) they tell latency can be from 3-6
> > ms up to 20-50 ms latency? As I understand they mean 50 ms is maximum, but I
> > have 200 ms, 4 times more. Does it mean that Firefox is not following W3C
> > specification or they tell about different latency?
> 
> This is non-normative, as indicated by the line "This section is
> non-normative." at the top of the text.
> 
> I would expect 30ms round trip latency on Windows 7 and up. I've seen some
> issue in the area, related to a fast path we have when disabling processing
> like echo cancellation.
> 
> I'll investigate this week.

Thanks for the answer. I tested with Windows 7 and 10.
Could you please tell me what is expected latency for different platforms like MacOS, Linux etc?

Comment 7

[Z@]

I could make it 100 ms (0.1 s) on Windows 7 with settings like this
var p = navigator.mediaDevices.getUserMedia({ audio: { latency: 0.02, echoCancellation: false, mozNoiseSuppression: false, mozAutoGainControl: false} });

and this example

https://jsfiddle.net/zlelik/xfq3ykp7/74/

if you help me to make it 30 ms, I will be absolutely happy :)

P.S. I got this 100 ms in both Firefox and Chrome.

Comment 8

[Paul Adenot (:padenot)]

Looks like you might have an issue with your system or hardware setup. I would expect (round trip) 30ms on Windows, maybe 60ms on linux, and less than 20ms on osx.

Comment 9

[Z@]

Attachment: Input_Output_Test.png

Comment 10

[Z@]

Initially, I tried with 3 different computers with Windows 7 and Windows 10 and everywhere it was 200 ms. With the latest approach audio: { latency: 0.02, echoCancellation: false, mozNoiseSuppression: false, mozAutoGainControl: false} I tried only on 1 PC with Windows 7, but I will try with other and let you know.

Is there any requirement to hardware, like embedded sound card is not good, or sound card should support some specific features?

Let me explain and attach screenshot how I did the measurements, maybe we are talking about different things.
I hit microphone with my finger and sound looks like a click or something like this, when I hear this sound from speakers. I recorded both sounds with an external device (actually with my phone) and then open this recording in sound editing software and check the distance between 2 clicks.
I have attached Input_Output_Test.png as a demonstration, how I measured latency.

Comment 11

[Z@]

Hi Again.
I made another test on another Windows 10 computer and uploaded it as Firefox_Latency_PC2.png. I got 70 ms. (In Chrome it was 100 ms!).

Can you make similar test and upload your results and what hardware you have?

Comment 12

[Z@]

Attachment: Firefox_Latency_PC2.png

Comment 13

[Alex Chronopoulos [:achronop]]

Is there anything we can do here?

Comment 14

[Z@]

Yes. Paul Adenot wrote on 27 June that expected latency is 30 ms on windows. I could not get it. I tested at least on 3 computers.

on windows 7 I got 200 ms, on windows 10 I got 70 ms, but not 30 ms.

What I would like to ask, which can be helpful.

- if someone from your team can make a test like I did and upload screenshots like I uploaded (Input_Output_Test.png, Firefox_Latency_PC2.png) it will be very helpful. And what is hardware and OS, where this test was done.
- Explain what is hardware requirements to get low latency.
- maybe there is another advice how can I get 30 ms latency in my example.

Comment 15

[Z@]

Sorry, on windows 7 I got 100 ms, not 200.

Comment 16

[Paul Adenot (:padenot)]

I have things I need to take care of first, but I'll do it eventually.

Comment 17

[Z@]

(In reply to Paul Adenot (:padenot) from comment #16)
> I have things I need to take care of first, but I'll do it eventually.

Thanks a lot. take your time whatever you need.

Comment 18

[Bulk Bug Changes for mreavy's org]

Mass change P4->P5 to align with new Mozilla triage process.

Comment 19

[jbstrater]

In answer to the question in comment #2: Chrome, Firefox, and Edge all have similar latency on Windows 10. It ought to be surprising that Edge isn't better - I hope that doesn't bode poorly for this effort here.

Microsoft claims substantial improvements in latency with Win10 here:
https://docs.microsoft.com/en-us/windows-hardware/drivers/audio/low-latency-audio

It says that true low-latency performance no longer requires exclusive mode or ASIO.  It looks like the iAudioClient3 interface allows better control of buffer sizes, and the ability to make them smaller for lower latency. This source is supposed to demonstrate the specifics:
https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/WindowsAudioSession

Is it possible to reduce this latency using these new buffer size features? Or is it up to the manufacturer to update their drivers?  Or both?  I have only one audio input device that lets me specify the buffer size, and is uses ASIO to do it.  If the device driver doesn't offer smaller buffer sizes, then there's nothing more you can do, right?

I would love to see this improved on Windows. If there's any way I might assist, including coding/testing, let me know. I have programmed in various languages on Windows - I'm not familiar with Firefox code at all.

ps - This issue has nothing to do with Web Audio API, and everything to do with getUserMedia(). It might benefit from a title change in that regard. I can hook the same stream to an HTML audio or video element and experience identical latency.  The latency is in the stream before it connects to a web audio context.

Comment 20

[Paul Adenot (:padenot)]

> Is it possible to reduce this latency using these new buffer size features?

I think so, yes.

> Or is it up to the manufacturer to update their drivers?  Or both? 

Clearly having better drivers will help here, but I think we can do some improvements by using IAudioClient3. This is something we want to do, but we don't have time to work on it. It's probably not very hard.

> I have only one audio input device that lets me specify the buffer size, and is uses ASIO to do it.  If the device driver doesn't offer smaller buffer sizes, then there's nothing more you can do, right?

It's unclear. It's probably device/driver dependent, but for some configuration, we should be able to go a bit lower. Because of the license of the ASIO SDK by Steinberg, implementing an ASIO backend for Firefox is out of the question. Something that is doable is to make the jack backend work on Windows, and to use Jack to talk to ASIO.

> ps - This issue has nothing to do with Web Audio API, and everything to do with getUserMedia().
> It might benefit from a title change in that regard. I can hook the same stream to an HTML audio
> or video element and experience identical latency.  The latency is in the stream before
> it connects to a web audio context.

In Firefox, regardless of what you're doing, we consider that when you're doing getUserMedia, AudioContext, or use MediaStream in any capacity, you want lower latency (note that AudioContext lets you pass in a "hint" [0] to change this assumption, to suite some use cases better, but we haven't implemented this at the minute).

If we're doing this optimization, then it will apply to audio input, audio output, and all object that are more real-time than HTMLMediaELement (say, getUserMedia, AudioContext, MediaStream, anything webrtc, including HTMLMediaElement that are receiving MediaStream from PeerConnections, etc.).

We're tracking the implementation of IAudioClient3 in bug 1412067, if people want to follow along. We also have some buffering happening for under-run protection, that we will able to remove when we've optimized a bit some other things.

We're always happy to mentor people working on the Firefox code base, let us know if it's something you can/want to do.

Comment 21

[jbstrater]

Paul,

Thanks for the detail. I want to implement low latency audio for getUserMedia(). I have programmed C++ in Windows, and used various Windows SDKs/APIs in the past (I assume the relevant Firefox code is in C++). If there is mentoring available, and I can be effective, I would gladly spend some time on this. It would be a relief to have at least one browser in Windows that has input latency low enough for musical play-along, which is what I've been working on this past week.  I'm mildly surprised that Edge doesn't do better at this, since its raison d'être is better performance on Windows, but Edge seems a bit haphazard in its support of features anyway.

Let me know if there's an opportunity to help enable this on Firefox.  You can contact me via the email address in my profile here.

Comment 22

[Paul Adenot (:padenot)]

The code in question is located at [0], and is maintained as a separate project (that can be compiled separately), vendored in Firefox. This github issue [1] describes what we want to do.

IAudioClient3 is being used for both output and input, so using it should lead to improved latency. We should be conservative when setting the buffer size, and do proper testing, it can lead to under-runs pretty quickly.

[0]: https://github.com/kinetiknz/cubeb/blob/master/src/cubeb_wasapi.cpp
[1]: https://github.com/kinetiknz/cubeb/issues/324

Comment 23

[jbstrater]

Paul,

Thanks for the links.  I have a MinGW setup for compiling C++ in Qt, so I'll try that. If not, I'll install the Community version of Visual Studio 2017.
For any other "mentoring" help I will contact you via email, as I don't want to clog up this issue with my naïve questions.
My understanding is that 256 is the smallest buffer size that is plausible for the Web Audio API. That's a ~6ms buffer at 44.1khz.  I'll try not to go below that size.
I'm sure I'll have at least one dumb question before I get it up and running for debugging/tinkering...

Thanks!

Comment 24

[jbstrater]

Update: I am now up and running on the cubeb project, and though I have not solved anything yet, I have discovered some things that are worth mentioning to you all:

1) IAudioClient3 and its low latency features will solve part of this problem, and should be able to reduce latency by ~22ms.  That still leaves ~60ms of improvement needed to get to 40ms round-trip latency, which is a reasonable maximum for musical live monitoring. I will attempt to find other places to improve this latency, and hopefully they won't interfere with other bits of code intended to prevent audio glitches. Latency vs. glitches is a balancing act.

2) Availability of the IAudioClient3 low latency features is limited on Windows 10:
    a) USB devices do not have access to these features.  I have filed a feature request with Microsoft, here is the official bug number:description - Bug 14725594: usbaudio.sys and usbaudio2.sys do not opt in to Win10 low latency (variable frame rates)
    b) Bluetooth devices do not have access to these features, according to Microsoft support.  I am unable to test this with my current hardware.
    c) PCI audio manufacturers must create their own device drivers, and are not subject to the restrictions of any Microsoft subsystems; but they must "opt in" to access these features, so those manufacturers must write the necessary code.  I have no idea which manufacturers have opted in at this time.
    d) So today, the only hardware with access to these features is built-in (on the motherboard) audio and PCI cards for which there are new, compatible drivers.  And, to get access to those features, you might have to switch your built-on audio device's drivers to the High Definition Audio Device generic drivers provided by Microsoft.  That is what I had to do.  It involves using Device Manager, which is a pretty low level tool for your average user, unfortunately.
    e) I haven't mentioned Thunderbolt audio, but it's not really worth mentioning on Windows.

So, we'll see what happens on the USB front. The developer with whom I've been communicating at Microsoft seemed eager for me to file the feature request, so maybe he'll start working on it soon.  Other than that it's not the brightest of outlooks, but hope is still alive :-)