Open Bug 899204 Opened 8 years ago Updated 3 years ago

Improve OpenSLES input latency calculation for webrtc

Categories

(Core :: WebRTC: Audio/Video, defect, P5)

22 Branch
defect

Tracking

()

mozilla28
blocking-b2g -
Blocking Flags:

People

(Reporter: jesup, Assigned: jesup)

References

Details

(Whiteboard: [ft:webrtc])

Bug 899159 doesn't address the latency calculation for input; it should be based on getMinBufferSize and the sampling rate.  That requires either going to java or baking it into an existing interface we have (mwu suggests putting it in nsAudioManager for mozilla).

We need to be able to query based on frequency, channels and format.

To upstream this, it would need to include all the things needed to access the buffer size via java, so we may flag it to them but not try to upstream any patch we do here.
Dear Randell,

I am Ivan Tsay running Media Recording agile team for FFOS 1.2. Since audio recording feature depends on this bug, may I know is there any idea about how long to get this fixed?

Moreover, I am thinking this is the blocker for audio recording feature so that I flag it for koi+ for future tracking in FFOS 1.2. Feel free to correct me if my understanding is wrong about this bug. Thank you!
blocking-b2g: --- → koi+
Flags: needinfo?(rjesup)
Whiteboard: [FT: Media Recording, Sprint 2]
Michael - can we get the input latency exposed to us as a function on nsAudioManager?  (It needs to be a function, as the sampling frequency is an input to the min-buffersize call.)  Should I open a bug; if so which component/etc?

Thanks!
Flags: needinfo?(rjesup) → needinfo?(mwu)
Sure - Core::DOM: Device Interfaces might be the closest thing, though I also see a lot of people dump these audio bugs into B2G::General. Guess we need better components.
Flags: needinfo?(mwu)
Setting a deadline of 9/4 to get this into our code.
Target Milestone: --- → mozilla26
Assignee: nobody → rjesup
Depends on: 910959
Input latency only matters for AEC tuning, and AEC is only used in PeerConnections, so resolving this bug is not needed for 26 (and in fact would have no impact on B2G).
blocking-b2g: koi+ → 1.3?
Target Milestone: mozilla26 → mozilla27
There's a total rewrite of the OpenSLES code in upstream (plus AEC delay and tail changes) which will be imported from 3.43, which we're targeting to land at the start of 28. This will be resolved in that release.
Target Milestone: mozilla27 → mozilla28
Blocks: 923363
Plus it for v1.3 since this blocks the committed feature in v1.3
blocking-b2g: 1.3? → 1.3+
Whiteboard: [FT: Media Recording, Sprint 2] → [ft:webrtc]
Depends on: 933189
We landed 3.43 with the OpenSLES rewrite from upstream.

This changed the input latency *assumed* (not checked or queried) from a fixed 80ms to a fixed 25ms.  This decreases the chance that the value is too high (which causes major failure in an AEC), at the cost of "eating" some of the tail length.

This is not a "MUST" for mobile use; we can tweak values if needed, but that is basically a tuning issue.  At most I think we'd use a configurable offset, which is already available in AudioConduit as media.peerconnection.capture_delay (though we should verify it works for mobile) and tune it for the device.

So we should unmark this as blocking 1.3.  It's a nice-to-have, and if we get something *and* we test it out as working better than the fixed default, great.

Bumping blocking to 1.3? to ask for verification of this change.
blocking-b2g: 1.3+ → 1.3?
No longer blocks: 923363
Sounds good to me. Moving to blocking- then.
blocking-b2g: 1.3? → -
No longer depends on: webrtc_upstream_bugs
Is this still relevant?
backlog: --- → webRTC+
Rank: 45
Flags: needinfo?(gpascutto)
Priority: -- → P4
https://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/audio_device/android/opensles_input.cc#333

Looks like the delay is roughly estimated with about 10ms accuracy (buffers are 10ms)?
Flags: needinfo?(gpascutto)
(In reply to Gian-Carlo Pascutto [:gcp] from comment #11)
> https://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/
> modules/audio_device/android/opensles_input.cc#333
> 
> Looks like the delay is roughly estimated with about 10ms accuracy (buffers
> are 10ms)?

I wouldn't call it 'estimated:

TotalBuffersUsed()-0.5 = (num_fifo_buffers_needed_ + kNumOpenSlBuffers) - 0.5 =
(kNum10MsToBuffer + kNumOpenSlBuffers) - 0.5 = (8 + 2) - 0.5 = 9.5 * 10ms = 95ms

That doesn't sounds like a good estimate... or rather a good amount if it is accurate, which it may well be.

Comment 8 was mostly correct, it was lower in 3.43 ((3 + 2) - 0.5 = 4.5*10ms = 45ms, not 25ms) until Bug 974378 landed ("Make webrtc.org OpenLS ES output code optional.  Increase input buffers").   "Increasing the buffer to the value we had before the webrtc.org uplift makes audio OK on my SGS2. I also verified that this doesn't seem to increase latency."

As for this bug: it may be that this bug is irrelevant (in that it accurately reflects the buffering and delays inherent to the input buffers), but the real issue (if so) may be to accurately determine how many buffers we need for good quality audio.
Mass change P4->P5 to align with new Mozilla triage process.
Priority: P4 → P5
No longer blocks: webrtc_upstream_bugs
You need to log in before you can comment on or make changes to this bug.