899204 - Improve OpenSLES input latency calculation for webrtc

Status: RESOLVED WONTFIX

(Core :: WebRTC: Audio/Video, defect, P5)

Description

[Randell Jesup [:jesup] (needinfo me)]

Bug 899159 doesn't address the latency calculation for input; it should be based on getMinBufferSize and the sampling rate.  That requires either going to java or baking it into an existing interface we have (mwu suggests putting it in nsAudioManager for mozilla).

We need to be able to query based on frequency, channels and format.

To upstream this, it would need to include all the things needed to access the buffer size via java, so we may flag it to them but not try to upstream any patch we do here.

Comment 1

[Ivan Tsay (:ITsay)]

Dear Randell,

I am Ivan Tsay running Media Recording agile team for FFOS 1.2. Since audio recording feature depends on this bug, may I know is there any idea about how long to get this fixed?

Moreover, I am thinking this is the blocker for audio recording feature so that I flag it for koi+ for future tracking in FFOS 1.2. Feel free to correct me if my understanding is wrong about this bug. Thank you!

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

Michael - can we get the input latency exposed to us as a function on nsAudioManager?  (It needs to be a function, as the sampling frequency is an input to the min-buffersize call.)  Should I open a bug; if so which component/etc?

Thanks!

Comment 3

[Michael Wu [:mwu]]

Sure - Core::DOM: Device Interfaces might be the closest thing, though I also see a lot of people dump these audio bugs into B2G::General. Guess we need better components.

Comment 4

[Maire Reavy [:mreavy]]

Setting a deadline of 9/4 to get this into our code.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

Input latency only matters for AEC tuning, and AEC is only used in PeerConnections, so resolving this bug is not needed for 26 (and in fact would have no impact on B2G).

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

There's a total rewrite of the OpenSLES code in upstream (plus AEC delay and tail changes) which will be imported from 3.43, which we're targeting to land at the start of 28. This will be resolved in that release.

Comment 7

[Ivan Tsay (:ITsay)]

Plus it for v1.3 since this blocks the committed feature in v1.3

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

We landed 3.43 with the OpenSLES rewrite from upstream.

This changed the input latency *assumed* (not checked or queried) from a fixed 80ms to a fixed 25ms.  This decreases the chance that the value is too high (which causes major failure in an AEC), at the cost of "eating" some of the tail length.

This is not a "MUST" for mobile use; we can tweak values if needed, but that is basically a tuning issue.  At most I think we'd use a configurable offset, which is already available in AudioConduit as media.peerconnection.capture_delay (though we should verify it works for mobile) and tune it for the device.

So we should unmark this as blocking 1.3.  It's a nice-to-have, and if we get something *and* we test it out as working better than the fixed default, great.

Bumping blocking to 1.3? to ask for verification of this change.

Comment 9

[Jason Smith [:jsmith]]

Sounds good to me. Moving to blocking- then.

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

Is this still relevant?

Comment 11

[Gian-Carlo Pascutto [:gcp]]

https://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/audio_device/android/opensles_input.cc#333

Looks like the delay is roughly estimated with about 10ms accuracy (buffers are 10ms)?

Comment 12

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #11)
> https://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/
> modules/audio_device/android/opensles_input.cc#333
> 
> Looks like the delay is roughly estimated with about 10ms accuracy (buffers
> are 10ms)?

I wouldn't call it 'estimated:

TotalBuffersUsed()-0.5 = (num_fifo_buffers_needed_ + kNumOpenSlBuffers) - 0.5 =
(kNum10MsToBuffer + kNumOpenSlBuffers) - 0.5 = (8 + 2) - 0.5 = 9.5 * 10ms = 95ms

That doesn't sounds like a good estimate... or rather a good amount if it is accurate, which it may well be.

Comment 8 was mostly correct, it was lower in 3.43 ((3 + 2) - 0.5 = 4.5*10ms = 45ms, not 25ms) until Bug 974378 landed ("Make webrtc.org OpenLS ES output code optional.  Increase input buffers").   "Increasing the buffer to the value we had before the webrtc.org uplift makes audio OK on my SGS2. I also verified that this doesn't seem to increase latency."

As for this bug: it may be that this bug is irrelevant (in that it accurately reflects the buffering and delays inherent to the input buffers), but the real issue (if so) may be to accurately determine how many buffers we need for good quality audio.

Comment 13

[Bulk Bug Changes for mreavy's org]

Mass change P4->P5 to align with new Mozilla triage process.

Comment 14

[Randell Jesup [:jesup] (needinfo me)]

Paul, is this still in any way relevant? If not, close it, thanks

Comment 15

[Paul Adenot (:padenot)]

Yes, we have the number directly in cubeb now.