Closed Bug 1869043 Opened 2 years ago Closed 2 years ago

Move CrossGraphReceiver usage for audio outputs from AudioStreamTrack to MediaTrackGraph

Categories

(Core :: Audio/Video: MediaStreamGraph, task)

Product:
Core
Component:
Audio/Video: MediaStreamGraph
Type:
task

Tracking

()

Status:
RESOLVED FIXED
Milestone:
123 Branch
Tracking Flags:
Tracking Status
firefox123 --- fixed

People

(Reporter: karlt, Assigned: karlt)

References

Details

Attachments

(14 external files)

Outputs move from AudioStreamTrack to the MediaTracks and all outputs from the same graph to the same non-primary output device will be mixed before passing to a CrossGraphReceiver managed by the MediaTrackGraph.

This will support solutions to bug 1849108 such as choosing a non-default primary output device
and/or passing the output for a secondary output device to AudioProcessing for AEC.

PlayAudio will mix output in planar layout, suitable for CrossGraphReceiver(). Interleaving output for the AudioCallbackDriver for the primary output device will be done after, rather than before, mixing.

https://treeherder.mozilla.org/jobs?repo=try&revision=1e53183f8a7bc2a5a38f0a462052bf483d3615ee

Blocks: 1872519
Depends on: 1873806

This will enable PlayAudio() to be used in a subsequent patch to mix audio
output for CrossGraphReceiver::EnqueueAudio().

An AudioChunk is used to clarify the planar storage and simplify use of
InterleaveAndConvertBuffer().

AudioMixer is also changed to initialize to zero only as many samples in its
buffer as will be used. Resetting to zero is no longer performed in
FinishMixing() because that is done on first call to Mix().

AudioCallbackDriver is the only client to benefit from a callback.

StartMixing() is now required if reusing the AudioMixer because getting the
mixed data after mixing is no longer collapses the duration to zero.

Depends on D198220

This clarifies that it is constant for all outputs mixed together and will
facilitate using PlayAudio() for secondary output devices.

Depends on D198221

This is performed in CheckDriver() so unnecessary in RegisterAudioOutput().

Depends on D198222

mAudioOutputParams will also be used in a subsequent patch to support
secondary output devices in the MediaTrackGraph because a CrossGraphReceiver
will be created and destroyed on the main thread when it knows whether or not
a secondary device will be in use.

Volumes of different audio outputs (with different keys) on the same track are
coalesced on the main thread, so that the graph thread need not care about
output keys, nor mix the same stream multiple times for different keys.

Depends on D198224

A subsequent patch will add secondary output devices for the same graph.
The primary output device will still be the device used with an
AudioCallbackDriver.

Depends on D198226

Output to secondary devices is supported through CrossGraphReceivers on other
graphs created when required.

Depends on D198228

The MediaTrackGraphImpl is used directly rather than obtained from the
MediaTrack because, with subsequent patches, the MediaTrack may be from
another graph.

Depends on D198229

The primary motivation for MediaStreamRenderer keeping track of and settling
incomplete promises is that, after changes in subsequent patches,
AudioStreamTrack outputs will share CrossGraphReceivers and so dedicated
CrossGraphReceivers will no longer be available to reject incomplete promises
when CrossGraphRecievers are Destroy()ed when an output is removed.

This also reliably keeps promise resolution in order wrt the synchronous
resolution from a setSinkId() call while playback is paused.

When a promise is settled because a subsequent pause or setSinkId() makes
the device change unnecessary, the promise is now resolved instead of
rejected. The new behavior is consistent with the resolution of a
promise created while playback is paused and with AudioSinkWrapper. Promise
resolution may be less likely to surprise content script than promise
rejection.

The situation with multiple tracks is somewhat arbitrary. Settling of the
promise depends on which tracks were present when setSinkId() was called.
GenericPromise::All() in MediaStreamRenderer::SetAudioOutputDevice() would
reject when the first track that existed at setSinkId() was removed or ended.
This patch switches to AllSettled() and resolves when all tracks that existed
at setSinkId() have ended. When AudioStreamTrack outputs no longer have
dedicated CrossGraphRecievers, removal of tracks will no longer cause the
promise to be settled until no tracks require the device.

Depends on D198230

Multiple outputs to the same device are now mixed to a single
CrossGraphReceiver and device output stream.

Depends on D198232

Keywords: leave-open
Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/5fed273e4f3d interleave audio output after AudioMixer mixing r=padenot
Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/1d2ccbe0bb6d pull mixed audio from AudioMixer instead of using a callback r=padenot
Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4e4d53fe90bd determine the number of output channels outside of PlayAudio() r=padenot https://hg.mozilla.org/integration/autoland/rev/d42cf476e404 remove driver switching from RegisterAudioOutput() r=padenot https://hg.mozilla.org/integration/autoland/rev/1dc9f987063c remove declaration of missing CreateOrDestroyAudioTracks r=padenot
Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/ae005b3084dc add a main thread record of track audio outputs r=padenot
Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/3f91b7b85223 assert that graph set access is main thread only r=padenot https://hg.mozilla.org/integration/autoland/rev/1fe2a5b597c7 introduce the concept of a primary output device for a MediaTrackGraph r=padenot https://hg.mozilla.org/integration/autoland/rev/734e2e027196 rename MockCubebStream::InputSampleRate() SampleRate() r=pehrsons
Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/df2d75db2354 allow a device to be specified with MediaTrack::AddAudioOutput() r=padenot
Regressions: 1874484
Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/58cbfdc760aa replace GraphStartedNotificationControlMessage with a lambda r=pehrsons
Attachment #9372166 - Attachment description: Bug 1869043 manage AudioStreamTrack outputs to non-default devices in MediaTrackGraph r?pehrsons → Bug 1869043 manage AudioStreamTrack outputs to non-default devices in MediaTrackGraph r=pehrsons
Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4ff0931a06b4 track and resolve device changed/running promises in MediaStreamRenderer r=pehrsons
Keywords: leave-open
Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/89f6c09ff6f0 allow a device to be specified with MediaTrackGraph::NotifyWhenDeviceStarted() r=pehrsons
Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/fd9c640cef86 manage AudioStreamTrack outputs to non-default devices in MediaTrackGraph r=pehrsons

https://hg.mozilla.org/mozilla-central/rev/89f6c09ff6f0

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox123: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 123 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: