1674892 - Enable connecting getUserMedia and MediaStreamAudioSourceNode running at different rates

Status: NEW

(Core :: WebRTC: Audio/Video, enhancement, P2)

Description

[Mark Hills]

Attachment: test-firefox-sample-rates.html

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0

Steps to reproduce:

See attached minimal test case, which does:

1. new AudioContext({sampleRate: 48000})
2. getUserMedia

Actual results:

"Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported."

Expected results:

This limitation is self-explanatory, of course.

So I file this bug as a request for the 'enhancement', as I could find no pre-existing bugzilla entry?

But it renders the sampleRate setting of AudioContext redundant because there's no way to match an incoming stream -- there's a documented [1] constraint for the getUserMedia sample rate, but this is not implemented.

Does this mean there is even a guarantee that the default getUserMedia sample rate will match the AudioContext?

Typically this difference is going to be 44100 vs 48000Hz.

At least with getUserMedia(sampleRate) this is not a hard limitation and developers with specific requirements can work around it (and avoid resamplers)

But I note that Chromium is not limited in either so inevitibly there will be pages in the wild using this.

Comment 1

[Timea Cernea [:tbabos][inactive]]

Thank you for the enhancement request, moving this to its component for the development team to check it out.

Comment 2

[Mark Hills]

I think it's probably a mistake to assume this is 'enhancement'.

Because it's really a bug that means if one sets the sameRate of AudioContext then the result is unusable.

Comment 3

[Byron Campen [:bwc]]

This look like a P3 to you jib?

Comment 4

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Depends. We have both P2 and P3 bugs in this area. Paul, would this be resolved by Bug 1238038 do you think, or would more work be needed?

Comment 5

[Paul Adenot (:padenot)]

This is expected for now, and will be solved by fixing bug 1238038, yes. This is well underway.

Comment 6

[Tasnim Khan]

Just wanted to ask how far we are in fixing this? I assume this is similar to bug 1725336 which should be fixed as well as part of fixing bug 1238038?

Comment 7

[Paul Adenot (:padenot)]

Tasnim, my colleague Chun-Min is quite far along already, I think it's more or less working, and I expect that we'll be able to start reviewing the patch set soon. I think there's going to be a bit of plumbing left to do to solve bug 1725336 after bug 1238038 is done, but most of the complexity is in bug 1238038, and it shouldn't be that hard/long.

Comment 8

[Tasnim Khan]

Thanks for the quick update Paul!

Comment 9

[Tasnim Khan]

Just wanted to get a sense on ETA around this. A lot of work already seems to be done on bug 1238038 but just wanted to check what's ETA on that to be done now? Afterwards, how far we are with getting this bug (or bug 1725336) to be fixed?

Comment 10

[Nils Ohlmeier [:drno]]

I'm hitting this problem as well.

We are calling gUM with 'sampleRate: 48000', which currently gets ignored by Firefox because of bug 1388586. And when we connect the audio track from the mic to our AudioContext we hit error mentioned in the initial report.

Besides all of this working fine in Chrome, what makes this an annoying bug is that Firefox also doesn't report the sample rate of a track from the mic.

I guess one way to find out the sampleRate could be to connect the microphone track to an AudioContext and then look at the sampleRate. That way we could avoid hitting the above mentioned error, because we would no longer blindly connect an unknown sampleRate to an AudioContext with a fixed sampleRate.

Comment 11

[belojar]

+1

Comment 13

[C.M.Chang[:chunmin]]

If the cause is the same as Bug 1664223, then the error likely originates from here. In this case, the problem would be: Why do AudioContext and gUM have different MediaTrackGraphs? I would expect them to share the same graph if they're created in the same window and using the same default output device. However, if one of them originates from an iframe, they might be assigned to different graphs.

If they do end up in different MediaTrackGraphs, CrossGraphTransmitter and CrossGraphPort could be used to transfer (resampled) audio data from one graph to another, which should allow for proper synchronization.

Hi Karl, do you have a sense of how much effort would be involved in implementing this?

Comment 14

[Karl Tomlinson (:karlt)]

(In reply to C.M.Chang[:chunmin] from comment #13)

Why do AudioContext and gUM have different MediaTrackGraphs?

The AudioContext is created with a specific sample rate. The microphone gets the default sample rate. At least if these rates differ, two graphs are required.

Plausibly, the microphone rate could be chosen on connection to a consumer or sink, but it might be connected to two sinks of different rates, so the generic solution is to handle different rates through the cross graph mechanisms you identify. That would also be easier to implement.

Resolving for the testcase here is about hooking up MediaStreamAudioSourceNode to the cross graph machinery, as you indicate. AFAIK the parts would be a good fit.

Some of the finer details may be more tricky to sort out, such as suspending tracks in the other graph while the AudioContext is suspended or the microphone is muted. In a minimum viable product, the key would be to ensure than nothing is accumulating unused frames in a buffer. Beyond that suspending streams is an optimization.

There are similar situations elsewhere such as in HTMLMediaElement, but they can be addressed separately.

Comment 15

[C.M.Chang[:chunmin]]

Attachment: Bug 1674892 - Add a WPT verifying MediaStreamAudioSource from AudioDestination under different rates

Comment 16

[C.M.Chang[:chunmin]]

Attachment: Bug 1674892 - Add a ref-counted version of SimpleMap

Comment 17

[C.M.Chang[:chunmin]]

Attachment: Bug 1674892 - Enable connecting AudioStreamTrack and MediaStreamAudioSourceNode in different graphs

Comment 18

[C.M.Chang[:chunmin]]

Attachment: Bug 1674892 - Add a crashtest

This patch adds a crashtest that verifies it's safe to close a page while a
running MediaStreamAudioSourceNode is sourcing from an active MediaStream
created via getUserMedia. The test ensures that dropping the underlying
AudioStreamTrack that has active input cleanly tears down any cross-graph
ports, with no crashes or assertion failures during document shutdown.

Comment 19

[C.M.Chang[:chunmin]]

Attachment: Bug 1674892 - Update mochitests to validate cross-rate AudioContext connections

Now that connecting two AudioContexts running at different sample rate is
supported, this patch updates the mochitests accordingly:

• Remove test_getUserMedia_nonDefaultRate.html
• Validate cross-rate connection in test_audioContextParams_sampleRate.html
• Validate cross-rate connection in test_peerConnection_nonDefaultRate.html

Comment 20

[C.M.Chang[:chunmin]]

Manual test page here