908834 - Audio realtime input clock mismatch (drift) causes local delay in MediaStreamGraph/media elements

Status: RESOLVED INVALID

(Core :: WebRTC: Audio/Video, defect, P3)

Description

[Randell Jesup [:jesup] (needinfo me)]

+++ This bug was initially created as a clone of Bug #884365 +++

[ This is "Part 2" of bug 884365, in particular resampling the data to track MSG's clock and so avoid any data buildup or underflows in MSG ]

We have a huge problem with audio latency building up over time in MSG (or, conversely, with it having to insert silence on underflows) when the microphone input and MSG don't share the same clock (which is common, as different hardware devices often will have different clocks)

MSG is currently clocked off the system clock, delivering 10ms of audio data for each 10ms of system clock time.  This will be changed to clock off the output clock of cubeb.  Neither is guaranteed to be the same as any given microphone, even if both are part of the same headset.  For added fun, hardware clockrates themselves can (slowly) drift.

Resampling the input to the system clock rate will work, but can cause problems with the AEC, especially if the resampling rate is not *extremely* consistent.

Comment 1

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to Randell Jesup [:jesup] from comment #0)
> Resampling the input to the system clock rate will work, but can cause
> problems with the AEC, especially if the resampling rate is not *extremely*
> consistent.

This confuses me. AFAIK the inputs to AEC are a) audio being produced by Firefox or the entire system and b) microphone input, and AEC occurs before any resampling of the microphone input we would do. So how does resampling the post-AEC input affect the AEC? I assume it could only affect a), but surely mixing all application or system audio output requires resampling anyway?

Comment 2

[Timothy B. Terriberry (:derf)]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #1)
> any resampling of the microphone input we would do. So how does resampling
> the post-AEC input affect the AEC? I assume it could only affect a), but

The answer is that we are currently doing AEC in the wrong place. That is, we're not doing it in gUM, but are instead doing it on the input to the PC. This was supposed to be a temporary hack (bug 818670), but it has stuck around a long time now.

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

See bug 694814 (move the AEC to PeerConnection)

I've been discussing with padenot some of our options for getting a clean (no frequent delay changes) output stream for the entire browser from the backend (cubeb) to feed over to getUserMedia.  Until we have that, we can't easily move the AEC.

Moving the AEC is currently slated for FF28 timeframe, but I've been thinking of pushing that to the FF27 timeframe and re-juggling, depending on the result of discussions with padenot.

Comment 4

[Robert O'Callahan (:roc) (email my personal email if necessary)]

OK. I don't think we should be trying to work around that, especially if the workaround is difficult.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

Note that moving the AEC won't change that we need to resample at the gUM->MediaStreamGraph interface; at most moving it will mean that the resampling can change rates faster - but true hardware clockrate mismatches aren't likely to change very quickly at all.  The only thing that might change them more quickly would be something like a an MSG underrun or stall.

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

Definitely not fixing for 24, and maybe not for 25

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

A patch at this point for 25 would be too much risk for benefit.  Will try to target 26 (which aligns with b2g/recording api)

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

Correction: per the target milestone, this targets 27, not 26.  It's not needed for 26

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

Rebalancing this to padenot

Comment 10

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.

Comment 11

[Z@]

Hi,
Could you please clarify what is with this bug? Is it fixed or not? The status "resolved invalid" is very confusing.