Audio realtime input clock mismatch (drift) causes local delay in MediaStreamGraph/media elements

NEW
Assigned to

Status

()

Core
WebRTC: Audio/Video
P2
major
Rank:
23
4 years ago
2 years ago

People

(Reporter: jesup, Assigned: jesup)

Tracking

(Blocks: 1 bug)

22 Branch
mozilla34
Points:
5
Dependency tree / graph

Firefox Tracking Flags

(firefox22 wontfix, firefox23- wontfix, firefox24- wontfix, firefox25- wontfix, firefox26- wontfix)

Details

(Whiteboard: [getUserMedia] [blocking-gum-] )

(Assignee)

Description

4 years ago
+++ This bug was initially created as a clone of Bug #884365 +++

[ This is "Part 2" of bug 884365, in particular resampling the data to track MSG's clock and so avoid any data buildup or underflows in MSG ]

We have a huge problem with audio latency building up over time in MSG (or, conversely, with it having to insert silence on underflows) when the microphone input and MSG don't share the same clock (which is common, as different hardware devices often will have different clocks)

MSG is currently clocked off the system clock, delivering 10ms of audio data for each 10ms of system clock time.  This will be changed to clock off the output clock of cubeb.  Neither is guaranteed to be the same as any given microphone, even if both are part of the same headset.  For added fun, hardware clockrates themselves can (slowly) drift.

Resampling the input to the system clock rate will work, but can cause problems with the AEC, especially if the resampling rate is not *extremely* consistent.
Target Milestone: --- → mozilla27
(In reply to Randell Jesup [:jesup] from comment #0)
> Resampling the input to the system clock rate will work, but can cause
> problems with the AEC, especially if the resampling rate is not *extremely*
> consistent.

This confuses me. AFAIK the inputs to AEC are a) audio being produced by Firefox or the entire system and b) microphone input, and AEC occurs before any resampling of the microphone input we would do. So how does resampling the post-AEC input affect the AEC? I assume it could only affect a), but surely mixing all application or system audio output requires resampling anyway?
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #1)
> any resampling of the microphone input we would do. So how does resampling
> the post-AEC input affect the AEC? I assume it could only affect a), but

The answer is that we are currently doing AEC in the wrong place. That is, we're not doing it in gUM, but are instead doing it on the input to the PC. This was supposed to be a temporary hack (bug 818670), but it has stuck around a long time now.
(Assignee)

Comment 3

4 years ago
See bug 694814 (move the AEC to PeerConnection)

I've been discussing with padenot some of our options for getting a clean (no frequent delay changes) output stream for the entire browser from the backend (cubeb) to feed over to getUserMedia.  Until we have that, we can't easily move the AEC.

Moving the AEC is currently slated for FF28 timeframe, but I've been thinking of pushing that to the FF27 timeframe and re-juggling, depending on the result of discussions with padenot.
OK. I don't think we should be trying to work around that, especially if the workaround is difficult.
(Assignee)

Comment 5

4 years ago
Note that moving the AEC won't change that we need to resample at the gUM->MediaStreamGraph interface; at most moving it will mean that the resampling can change rates faster - but true hardware clockrate mismatches aren't likely to change very quickly at all.  The only thing that might change them more quickly would be something like a an MSG underrun or stall.
(Assignee)

Comment 6

4 years ago
Definitely not fixing for 24, and maybe not for 25
status-firefox24: affected → wontfix
status-firefox26: --- → affected
tracking-firefox24: + → ?

Updated

4 years ago
tracking-firefox24: ? → -
(Assignee)

Comment 7

4 years ago
A patch at this point for 25 would be too much risk for benefit.  Will try to target 26 (which aligns with b2g/recording api)
status-firefox25: affected → wontfix
tracking-firefox26: --- → ?
(Assignee)

Updated

4 years ago
tracking-firefox25: + → ?
(Assignee)

Comment 8

4 years ago
Correction: per the target milestone, this targets 27, not 26.  It's not needed for 26
status-firefox26: affected → wontfix

Updated

4 years ago
tracking-firefox25: ? → -
tracking-firefox26: ? → -
(Assignee)

Comment 9

4 years ago
Rebalancing this to padenot
Assignee: rjesup → paul
(Assignee)

Updated

4 years ago
Blocks: 958090
Depends on: 848954
Blocks: 970426
No longer blocks: 970426
(Assignee)

Updated

4 years ago
Depends on: 970715
(Assignee)

Updated

4 years ago
Blocks: 970426
Assignee: paul → rjesup

Updated

3 years ago
Whiteboard: [getUserMedia] [blocking-gum-] → [getUserMedia] [blocking-gum-][p=5, priority, ft:webrtc]
Target Milestone: mozilla27 → mozilla32
No longer blocks: 970426
Severity: critical → major
Target Milestone: mozilla32 → mozilla33
Priority: P1 → P2
Target Milestone: mozilla33 → mozilla34

Updated

2 years ago
backlog: --- → webRTC+
Points: --- → 5
Rank: 23
Whiteboard: [getUserMedia] [blocking-gum-][p=5, priority, ft:webrtc] → [getUserMedia] [blocking-gum-]
You need to log in before you can comment on or make changes to this bug.