Poor quality in WebRTC audio capture on Linux
Categories
(Core :: WebRTC: Audio/Video, defect, P3)
Tracking
()
Tracking | Status | |
---|---|---|
firefox120 | --- | fixed |
People
(Reporter: jim3692, Assigned: pehrsons)
References
(Blocks 1 open bug)
Details
Attachments
(49 files)
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review |
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0
Steps to reproduce:
We are developing this extension, that allows screensharing with audio on Linux: https://github.com/IceDBorn/pipewire-screenaudio
Actual results:
When getUserMedia creates a new Pipewire Node, it's locked to F32LE format. We suspect that this causes audio pitching in calls, mainly when music is present. When the audio transmitted only contains voices, or sound effects, the quality is perfect. We checked the format on Chromium and it is set to S16LE. They also have different latency values. Firefox has 1200/48000 whereas Chromium has 480/48000.
Expected results:
Both browsers should have transmitted audio with similar quality, while Chromium has crystal clear audio, even with music, Firefox has pitching issues.
The only place I have seen a discussion related to this topic is this issue on Github: https://github.com/edisionnano/Screenshare-with-audio-on-Discord-with-Linux/issues/12
Comment 2•2 years ago
|
||
The Bugbug bot thinks this bug should belong to the 'Core::WebRTC: Audio/Video' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.
Comment 3•2 years ago
|
||
We are developing this extension, that allows screensharing with audio on Linux ...
When getUserMedia creates a new Pipewire Node, it's locked to F32LE format. We suspect that this causes audio pitching in calls, mainly when music is present.
Do you mean getDisplayMedia? getUserMedia is for camera and microphone.
Comment 4•2 years ago
|
||
He meant getDisplayMedia
, I'm another developer of the project and I can confirm by looking at the source code.
Updated•2 years ago
|
Updated•2 years ago
|
(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #3)
We are developing this extension, that allows screensharing with audio on Linux ...
When getUserMedia creates a new Pipewire Node, it's locked to F32LE format. We suspect that this causes audio pitching in calls, mainly when music is present.
Do you mean getDisplayMedia? getUserMedia is for camera and microphone.
Sorry for the confusion. We override the getDisplayMedia
method in order to inject the audio from the pipewire-screenaudio
node. To do that, we use getUserMedia
to find that node.
Updated•2 years ago
|
Comment 6•2 years ago
|
||
Can someone get us ways to reproduce or audio examples of this happening? Should we install the extension or something?
(In reply to Paul Adenot (:padenot) from comment #6)
Can someone get us ways to reproduce or audio examples of this happening? Should we install the extension or something?
I have samples from when the bug was discovered (plus samples of the original music for comparison). https://mega.nz/file/dK4mXQqL#quxhbB_sS6ukf5Qqyi6iGHobcCepKg2ycBYxV7RDthE
(In reply to Paul Adenot (:padenot) from comment #6)
Can someone get us ways to reproduce or audio examples of this happening? Should we install the extension or something?
You will need to install the extension from Addons, and optimally give it access to all websites, in case the call is in an iframe: https://addons.mozilla.org/en-US/firefox/addon/pipewire-screenaudio/
For its native part, which controls Pipewire you can either:
-
Install our AUR package on Arch: https://aur.archlinux.org/packages/pipewire-screenaudio
And choose the audio source to stream from within the extension -
Or you can manually create the capturing node with this command:
pw-cli create-node adapter "{ factory.name=support.null-audio-sink node.name=pipewire-screenaudio media.class=Audio/Source/Virtual object.linger=1 audio.position=[FL,FR] }"
And connect the inputs of the " Pipewire Screenaudio" node to a media player using Helvum
After selecting or connecting your media, you can start your screen share.
You can try using Spotify for the test. We saw that setting its quality to High, made the pitching much more noticeable than when it was set to Normal. But you can also try YouTube videos with and without music.
Comment 9•2 years ago
|
||
This is a bug in our drift compensator. An easier way to repro:
Run Helvum, make a new device like instructed:
pw-cli create-node adapter "{ factory.name=support.null-audio-sink node.name=pipewire-screenaudio media.class=Audio/Source/Virtual object.linger=1 audio.position=[FL,FR] }"
Play something in a media player, connect it to this new device using Helvum.
Open:
https://paul.cx/public/testing-multi-gum.html
Open a regular mic, and then open the new device, that should play what the media player plays.
Essentially what happens is that the device buffer size isn't the same as the graph, and everything breaks.
Comment 10•2 years ago
|
||
This is not a functional change, only a small refactoring.
Updated•2 years ago
|
Comment 11•2 years ago
|
||
Is there a workaround we can use on our extension until this bug gets fixed?
Comment 12•2 years ago
|
||
(In reply to github.envenomed from comment #11)
Is there a workaround we can use on our extension until this bug gets fixed?
I'm not sure. Opening the mic in second instead of first should do the trick, but maybe it's not practical. We'll get this fixed.
![]() |
||
Updated•2 years ago
|
Comment 13•2 years ago
|
||
Opening the mic in second instead of first should do the trick
What do you mean by that?
Comment 14•2 years ago
|
||
Do the capture first, then the microphone.
But we have new code for this that is getting ready, there should be something to try in a few days.
Comment 15•2 years ago
|
||
Do the capture first, then the microphone.
But we have new code for this that is getting ready, there should be something to try in a few days.
Comment 16•1 years ago
|
||
Watching a stream using Firefox 117.0b3
seems to fix the issue. We tried watching the stream with Firefox ESR and the audio drift bug was present. Did you fix the bug in 117
?
Assignee | ||
Comment 17•1 years ago
|
||
No. Maybe you end up with different latencies on 117?
This try build is of my WIP work. You could try the linux x64 shippable build to see how it fares.
Comment 19•1 years ago
|
||
There is an r+ patch which didn't land and no activity in this bug for 2 weeks.
:pehrsons, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit BugBot documentation.
Assignee | ||
Updated•1 years ago
|
Assignee | ||
Comment 20•1 year ago
|
||
Assignee | ||
Comment 21•1 year ago
|
||
Assignee | ||
Comment 22•1 year ago
|
||
Assignee | ||
Comment 23•1 year ago
|
||
Enables differentiating audio callback sources when tracing a two-mic use case
together with https://github.com/padenot/fx-profiler-audio-cb/pull/3.
Assignee | ||
Comment 24•1 year ago
|
||
Assignee | ||
Comment 25•1 year ago
|
||
This will help keep things together when in a later patch we add a plotting
script and instructions for using it.
Assignee | ||
Comment 26•1 year ago
|
||
Assignee | ||
Comment 27•1 year ago
|
||
Assignee | ||
Comment 28•1 year ago
|
||
Assignee | ||
Comment 29•1 year ago
|
||
The pre-buffer has the same size as the target buffer size we aim towards
keeping constant. If the first input segment is large the total amount buffered
is significantly higher than the target buffer size, which leads to us applying
the minimum drift correction (0.9) to shrink it back to the target buffer size
eventually. This can be avoided by accounting for the first input segment when
adding pre-buffering silence.
Assignee | ||
Comment 30•1 year ago
|
||
Assignee | ||
Comment 31•1 year ago
|
||
In order to do the correction calculations on fresh data.
Assignee | ||
Comment 32•1 year ago
|
||
This guarantees that our measurement of the current buffering level is fairly
accurate, since the reading of the input buffering level is recent.
This is especially important in situations where the input stream latency
(callback buffer size) is significantly higher than the output stream latency
which drives the drift correction.
Assignee | ||
Comment 33•1 year ago
|
||
These help verify the functionality of the drift correction. Especially useful
together with the data logging and plotting, see comments on the test cases.
Assignee | ||
Comment 34•1 year ago
|
||
Assignee | ||
Comment 35•1 year ago
|
||
Assignee | ||
Comment 36•1 year ago
|
||
Assignee | ||
Comment 37•1 year ago
|
||
PID-controllers are a proven way to control a system by only measuring an
indirect output of them.
This PID-controller is tuned by hand by looking at step responses
(see TestDriftController.cpp) and applying, in order, the P, D and I terms.
The tuning was done on top of later patches in this bug, so may not be ideal
when looking at this patch in isolation.
Assignee | ||
Comment 38•1 year ago
|
||
This can be handy when AudioDriftCorrection resamples the first time.
In order to calculate how many frames to prepend in order to get the buffer to
contain exactly the number of desired pre-buffer frames after resampling, so
the DriftController starts out with an error = 0.
Assignee | ||
Comment 39•1 year ago
|
||
This allows the drift correction code to start out with an error of 0, avoiding
starting out with a significant correction.
Assignee | ||
Comment 40•1 year ago
|
||
This lets AudioDriftCorrection handle situations where it's about to overflow
the buffer. Be it because of unexpectedly high drift, very high input latency or
an increased target buffering level.
Assignee | ||
Comment 41•1 year ago
|
||
Assignee | ||
Comment 42•1 year ago
|
||
Assignee | ||
Comment 43•1 year ago
|
||
Assignee | ||
Comment 44•1 year ago
|
||
This patch allows setting the desired buffering level on AudioDriftCorrection
dynamically. If the AudioResampler's pre-buffer has not yet been set, it will
be populated to match the new desired buffering level on the next Resample().
Assignee | ||
Comment 45•1 year ago
|
||
Note that this patch forwards the latency from CubebInputStream which can be
queried prior to starting, and we are therefore guaranteed to be able to set it
on the drift corrector prior to the first input callback.
CrossGraphPort is more complex with an entire MediaTrackGraph and its
GraphDriver behind it, and is left out at this time.
Assignee | ||
Comment 46•1 year ago
|
||
This patch handles an underrun to the drift correction input buffer by:
- Resetting the pre-buffer in the AudioResampler, so on the next Resample(),
it will prepend silence to match the desired buffering level. - Doubling the desired buffering level, since apparently it was not enough.
Overall this will make the perceived discontinuity longer than if we just output
the underrun duration worth of silence. There should also be fewer
discontinuities because of this, likely just one.
Assignee | ||
Comment 47•1 year ago
|
||
This is so that the integral term does not grow unboundedly while the control
signal is hitting its cap, as it would lead to increasing overshoot and
instability.
Assignee | ||
Comment 48•1 year ago
|
||
Having a hysteresis threshold on the error signal significantly reduces the
frequency at which we reconfigure the AudioResampler as a response to changes
in the control signal.
Assignee | ||
Comment 49•1 year ago
|
||
Assignee | ||
Comment 50•1 year ago
|
||
For allowing rounding upwards.
Assignee | ||
Comment 51•1 year ago
|
||
For safer conversions between time bases, of which drift control has plenty.
Assignee | ||
Comment 52•1 year ago
|
||
Assignee | ||
Comment 53•1 year ago
|
||
This prevents iterating MediaTrackGraph 0 frames.
Assignee | ||
Comment 54•1 year ago
|
||
CrossGraphTransmitter does not have an mSegment so its duration is meaningless.
Assignee | ||
Comment 55•1 year ago
|
||
There are not many statements identifying fallback wrappers in MediaTrackGraph:5
logs. Log the owning AudioCallbackDriver pointer instead.
Assignee | ||
Comment 56•1 year ago
|
||
Manual mode means it is up to the test to manually call the DataCallback.
With manual mode there will be no background thread spawned to run callbacks.
This can help make unittests that use MockCubeb more deterministic.
Assignee | ||
Comment 57•1 year ago
|
||
When a sudden (one high sample to a zero sample) discontinuity is fed to the
speex resampler, this will typically be smoothed out over multiple samples in
the output. Before this patch AudioVerifier only cares about the difference
between two adjacent samples when counting discontinuity.
This patch makes AudioVerifier account for several discontinuous samples in a
single discontinuity. I.e., once a discontinuity is detected, AudioVerifier
waits for a handful of samples to pass or for the discontinuity to go away,
whichever comes first, before being eligible for detecting another
discontinuity.
Assignee | ||
Comment 58•1 year ago
|
||
Assignee | ||
Comment 59•1 year ago
|
||
Assignee | ||
Comment 60•1 year ago
|
||
Assignee | ||
Comment 61•1 year ago
|
||
Assignee | ||
Comment 62•1 year ago
|
||
Assignee | ||
Comment 63•1 year ago
|
||
This also makes testMonoToStereoInput more similar to testAudioCorrection, since
they are essentially two variants of the same test, and fixes a test expectation.
Assignee | ||
Comment 64•1 year ago
|
||
It must not store more data than was consumed by the resampler in the tail
buffer.
Assignee | ||
Comment 65•1 year ago
|
||
Assignee | ||
Comment 66•1 year ago
|
||
Assignee | ||
Comment 67•1 year ago
|
||
This is ready, except for a data race found by tsan (roughly 20% failure rate) that we don't yet understand.
Assignee | ||
Comment 68•1 year ago
|
||
Without this, the necessary synchronization must be provided externally.
This fixes the memory order in the following case of changing producer thread:
- Thread A does SPSCQueue::Enqueue
- non-atomic write into the ring buffer, at memory location X
- mWriteIndex.load(relaxed)
- mWriteIndex.store(release)
- Producer thread is switched to B, no external memory order synchronization is
provided, but thread B is guaranteed to run after thread A has finished its
Enqueue task. - Thread B does SPSCQueue::Enqueue
- mWriteIndex.load(relaxed)
- mWriteIndex.store(release)
- Thread C does SPSCQueue::Dequeue
- mWriteIndex.load(acquire)
- non-atomic read from the ring buffer, at memory location X
In this scenario, there is no memory synchronization between threads A and B,
and therefore the non-atomic read on C is a data race, and flagged as such by
TSAN.
A similar scenario can be applied to changing the consumer thread, if first A
enqueues, then B dequeues, then C dequeues. However, since Dequeue doesn't
necessarily (MoveOrCopy) do non-atomic writes to the ring buffer, and more
importantly, since Enqueue doesn't do non-atomic reads from the ring buffer,
this is less of a problem.
Comment 69•1 year ago
|
||
Comment 70•1 year ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/f3d52a166497
https://hg.mozilla.org/mozilla-central/rev/07f593daa504
https://hg.mozilla.org/mozilla-central/rev/b3ab4f7079a6
https://hg.mozilla.org/mozilla-central/rev/3995617f816a
https://hg.mozilla.org/mozilla-central/rev/c1ab79a9ac72
https://hg.mozilla.org/mozilla-central/rev/30f59a0c95f2
https://hg.mozilla.org/mozilla-central/rev/0bce9535589c
https://hg.mozilla.org/mozilla-central/rev/6f25e0f2a000
https://hg.mozilla.org/mozilla-central/rev/dfc424bf89e4
https://hg.mozilla.org/mozilla-central/rev/d5b79e973899
https://hg.mozilla.org/mozilla-central/rev/42f435429f25
https://hg.mozilla.org/mozilla-central/rev/6197b1172cdc
https://hg.mozilla.org/mozilla-central/rev/1538a5bea488
https://hg.mozilla.org/mozilla-central/rev/60880eedb9de
https://hg.mozilla.org/mozilla-central/rev/39e5ea5b7c9a
https://hg.mozilla.org/mozilla-central/rev/ed07e8de5754
https://hg.mozilla.org/mozilla-central/rev/541cfe3da283
https://hg.mozilla.org/mozilla-central/rev/907c917b52fe
https://hg.mozilla.org/mozilla-central/rev/c4f57f87e34d
https://hg.mozilla.org/mozilla-central/rev/15af59e306c6
https://hg.mozilla.org/mozilla-central/rev/68d05ed50d8a
https://hg.mozilla.org/mozilla-central/rev/d448783c33e4
https://hg.mozilla.org/mozilla-central/rev/5cdcf42ebc69
https://hg.mozilla.org/mozilla-central/rev/89c0108a7c3a
https://hg.mozilla.org/mozilla-central/rev/2e257850dc13
https://hg.mozilla.org/mozilla-central/rev/1e1f3831e261
https://hg.mozilla.org/mozilla-central/rev/d0497551c605
https://hg.mozilla.org/mozilla-central/rev/f4195ac81452
https://hg.mozilla.org/mozilla-central/rev/8484b3a81291
https://hg.mozilla.org/mozilla-central/rev/b06a6636ac90
https://hg.mozilla.org/mozilla-central/rev/7b797b4dd354
https://hg.mozilla.org/mozilla-central/rev/fbbfe1de3c65
https://hg.mozilla.org/mozilla-central/rev/25757a9dd4ca
https://hg.mozilla.org/mozilla-central/rev/1aa8fb4ee69c
https://hg.mozilla.org/mozilla-central/rev/706c4902fd36
https://hg.mozilla.org/mozilla-central/rev/5c577d284800
https://hg.mozilla.org/mozilla-central/rev/e0ab2775ca26
https://hg.mozilla.org/mozilla-central/rev/333b7208d6d7
https://hg.mozilla.org/mozilla-central/rev/e2e5c96f32e4
https://hg.mozilla.org/mozilla-central/rev/630a12c1de0e
https://hg.mozilla.org/mozilla-central/rev/18c8184118db
https://hg.mozilla.org/mozilla-central/rev/da112e1234e5
https://hg.mozilla.org/mozilla-central/rev/dd7b8d38431e
https://hg.mozilla.org/mozilla-central/rev/f103c687dfe1
https://hg.mozilla.org/mozilla-central/rev/61d2120b3d48
https://hg.mozilla.org/mozilla-central/rev/c86d8d07927a
https://hg.mozilla.org/mozilla-central/rev/e7b747380a92
https://hg.mozilla.org/mozilla-central/rev/c406e2e8f277
Comment 71•1 year ago
|
||
Reporter, do you mind testing your add-on again in a Firefox Nightly? Please make sure that it's up-to-date since this just got merged, it's only on the very last version.
This is set to release to the general public in December 5th.
Reporter | ||
Comment 72•1 year ago
|
||
Hello, we have tested with Firefox Nightly and it looks like it is fixed.
Thank you very much for your time and support.
Description
•