1879180 - echo cancellation not working with audio output to multiple devices

Status: NEW

(Core :: WebRTC: Audio/Video, defect)

Description

[paul]

Steps to reproduce:

Setup

• Laptop A is running Brave
• Laptop B is running Firefox
  • On this laptop, a headset is the default audio in/out device, per my Mac system settings

Repro steps

• Start a WebRTC call between laptops A and B, keeping them in separate rooms of the house for noise isolation
• On laptop B (the one running Firefox)…
  • Wire up the audio output to the laptop speakers (i.e. set the output audio element's sinkId appropriately)
  • Wire up the audio input to the laptop mic (i.e. send a track obtained from the laptop mic via getUserMedia)
• Speak into laptop A

Actual results:

I heard myself echoed back through laptop A’s speakers.

Expected results:

I should hear nothing from laptop A.

Comment 1

[paul]

Additional notes:

• I believe this might be the same—or at least a very similar—issue as https://bugzilla.mozilla.org/show_bug.cgi?id=1849108. I was unable to comment directly on that bug since it was already closed.
• The above bug was marked as fixed on Firefox 123, but I was able to reproduce this issue on:
  • Firefox 123 beta (123.0b7)
  • Firefox 124 nightly (124.0a1)
• I tried explicitly adding echoCancellation to the getUserMedia constraints, but it didn't seem to make a difference.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::WebRTC: Audio/Video' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[Negritas Sergiu, Desktop QA]

Hello! thank you for submitting this issue as per the comment 1 I will update the flags of this issue and mark it as NEW in order to get our developers involved and provide a solution for this issue.
Unfortunately I wasn't able to reproduce the issue on my end on Ubuntu 22.04 and MacOS 12.6

Have a nice day!

Comment 4

[Dan Baker[:dbaker]]

Thank you for this information. Just a couple of questions to help improve us reproducing the issue:

• Are you able to reproduce this issue regardless of service (Meet, Jitsi, sample app, etc.)?
  • If this only reproduces in specific services would you share what ones work and what ones don't?
• Does this reproduce only when using Brave and Firefox or can you reproduce this is both machines are running Firefox Nightly?
• Does this reproduce for you if you switch which laptop is using the non default system device?
• Would you be willing to gather a Media Profiler of the issue and share it?

Comment 5

[paul]

(In reply to Dan Baker[:dbaker] from comment #4)

Thank you for this information. Just a couple of questions to help improve us reproducing the issue:

Sure thing, thanks so much for looking into this!

• Are you able to reproduce this issue regardless of service (Meet, Jitsi, sample app, etc.)?
  • If this only reproduces in specific services would you share what ones work and what ones don't?

I had originally tested using Daily (I work there; I used an internal version of the library that allows switching to non-default output devices in Firefox, which is currently disabled in publicly-available versions due to this very issue).

I was able to reproduce this issue using Jitsi Meet as well.

Sadly, I couldn't test this issue in Google Meet since they seem to be having issues even switching to the non-default audio output device in Firefox 🤔.

• Does this reproduce only when using Brave and Firefox or can you reproduce this is both machines are running Firefox Nightly?

I was also able to reproduce the issue in a call between Safari and Firefox, and between two instances of FIrefox Nightly.

• Does this reproduce for you if you switch which laptop is using the non default system device?

If laptop B (the Firefox one) is using the system default device (i.e. when the system default devices are the built-in Macbook mic & speaker), the issue goes away. It doesn’t seem to matter if laptop A is using their system default device or a different one.

• Would you be willing to gather a Media Profiler of the issue and share it?

Sure! https://share.firefox.dev/3HXPqoA. This profile is captured on Laptop B. I joined a Jitsi Meet room, switched from headphones (default) to Macbook mic & speaker. Then I joined the same Jitsi room from a laptop in another room, spoke into the mic on that laptop, and heard myself echoed back.

Some more notes on the issue:

• It's sometimes (but not always) pretty subtle.
• It seems a bit nondeterministic, with the echo being more pronounced at times and less at others. I can't exactly figure out what it's correlated with.
• I might be more pronounced if you whisper into the mic instead of speak normally into it, but I wouldn't swear to that.

Thanks again!

Comment 6

[paul]

Wanted to share a bit more info about this issue, in case it’s helpful.

As stated above, the echo can be pretty faint/subtle, and it comes and goes. But I was noticing that in my dev environment, in our product specifically, it was much louder, and much more consistent. I recently figured out the culprit—hopefully it helps with reproducing the issue, and in determining priority.

It looks like we may have been unintentionally wiring up playback of local audio (which contained the faint echo) through the local audio output device. There was an AudioContext in our codebase that we were using just for measuring local audio that we were accidentally wiring up to the audioContext.destination at full volume (without first turning down the gain).

For some reason this wasn't a problem in almost every case, and didn't result in you hearing yourself talking across the board. But in this very specific case—in Firefox only, and only when the non-system-default laptop mic and speaker are selected for the call—it seems to pick up the faint echo and literally turn up the gain on it, making this echo cancellation bug quite noticeable.

Comment 7

[Karl Tomlinson (:karlt)]

Thank you for these details.

(In reply to paul from comment #6)

It looks like we may have been unintentionally wiring up playback of local audio (which contained the faint echo) through the local audio output device. There was an AudioContext in our codebase that we were using just for measuring local audio that we were accidentally wiring up to the audioContext.destination at full volume (without first turning down the gain).

Yes, that would explain what you are experiencing.

Firefox uses the audio output stream to only one output device in the echo cancellation algorithm. When the remote audio is played through a non-default device and the local audio through the default device, Firefox picks one for the echo cancellation. Ideally both would be used. The delays can be different, so echo cancellation would need to be done twice to cancel both. Chrome also picks only one output device stream, but its heurisitics for this are different. I would expect local echo if the remote stream is used for AEC and remote echo if the local stream is used for AEC, so I don't have a good explanation for why there would be no echo in Chrome. I don't know what Safari does.

The AnalyserNode shouldn't need to be connected to the AudioDestinationNode at all, but earlier versions at least of Chrome and maybe Safari failed to passed audio to the AnalyserNode unless it was wired up to a destination. I thought Chrome had resolved this though I'm not certain and I don't know the situation in Safari. A gain of zero should behave as intended everywhere. Firefox may still choose this output for echo cancellation if the gain is small but non-zero.

Comment 8

[Karl Tomlinson (:karlt)]

(In reply to paul from comment #5)

(In reply to Dan Baker[:dbaker] from comment #4)
I was able to reproduce this issue using Jitsi Meet as well.

Sure! https://share.firefox.dev/3HXPqoA. This profile is captured on Laptop B. I joined a Jitsi Meet room, switched from headphones (default) to Macbook mic & speaker. Then I joined the same Jitsi room from a laptop in another room, spoke into the mic on that laptop, and heard myself echoed back.

That profile is captured on the right Laptop, thanks, but I think it's missing the "Media" option, and so doesn't have much on what's happening with audio. If you'd like to repeat with that option, I'd be keen to take another look.

I tried to reproduce myself with Jitsi. One time I did hear a couple of echoes and then no more. Is that the kind of thing you heard with Jitsi or was it repeatable throughout the call?

I've observed that elsewhere too and assumed it is due to the AEC algorithm taking two or three seconds to tune in to get the right delay for cancellation. If that is happening more in Firefox than elsewhere, then I guess we should be able to do better.
(The echo you heard in the internal Daily setup would have been repeatable throughout the call, when it occurred for that call, I assume.)

Comment 9

[paul]

(In reply to Karl Tomlinson (:karlt) from comment #8)

(In reply to paul from comment #5)

(In reply to Dan Baker[:dbaker] from comment #4)
I was able to reproduce this issue using Jitsi Meet as well.

Sure! https://share.firefox.dev/3HXPqoA. This profile is captured on Laptop B. I joined a Jitsi Meet room, switched from headphones (default) to Macbook mic & speaker. Then I joined the same Jitsi room from a laptop in another room, spoke into the mic on that laptop, and heard myself echoed back.

That profile is captured on the right Laptop, thanks, but I think it's missing the "Media" option, and so doesn't have much on what's happening with audio. If you'd like to repeat with that option, I'd be keen to take another look.

I tried to reproduce myself with Jitsi. One time I did hear a couple of echoes and then no more. Is that the kind of thing you heard with Jitsi or was it repeatable throughout the call?

I have a vague memory of hearing echoes faintly later in the call, too (but I could be misremembering). But yes, by and large it tended to happen at the start of the call, just as you describe.

I've observed that elsewhere too and assumed it is due to the AEC algorithm taking two or three seconds to tune in to get the right delay for cancellation. If that is happening more in Firefox than elsewhere, then I guess we should be able to do better.
(The echo you heard in the internal Daily setup would have been repeatable throughout the call, when it occurred for that call, I assume.)

That's correct, in my janky internal setup the echo was quite loud and clear throughout the call. But after addressing the AudioContext-outputting-to-its-destination-at-full-volume issue, the issue turned into the much subtler one you describe above.