1287672 - Add telemetry for 5.1 audio

Status: NEW

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[Blake Wu [:bwu][:blakewu]]

Per discussing with Anthony yesterday, we should add telemetry for 5.1 audio to get a sense of how many people use it.

Comment 1

[C.M.Chang[:chunmin]]

Attachment: [WIP] Add telemetry for channel count of the decoded audio data

Comment 2

[C.M.Chang[:chunmin]]

I think the best data we could have is the channel layout of raw decoded audio, which indicates not only channel counts but also channel order. The reason why I choose the "raw" data is because the channel count and the layout can be changed. We have two preferences, MediaPrefs::MonoAudio() and MediaPrefs::AudioSinkForceStereo(), which will downmix or upmix the channel before playing audio in cubeb. Thus, it's better to use the "raw" information to know how many times that users try playing audio 5.1.

However, the raw audio data doesn't provide information about the channel layout. In fact, we could collect the channel bitmask to indicate the used channels instead, but we have no way to know the channel's order from these data.

kinetik, could you give me some suggestions?

Comment 3

[Matthew Gregan [:kinetik]]

The original channel layout has been forgotten by the time the audio data reaches DecodedAudioDataSink since the data is being passed to AudioConverter for conversion to SMPTE layout as it's decoded, so you'd either need to plumb the original channel layout through from the demuxer/decoder it was read from or report the telemetry from those places directly.

It'd probably also be useful to know the common channel layouts and speaker/channel counts of the target output audio devices as well, since we don't know how common non-stereo presentation is in the wild.

Comment 4

[C.M.Chang[:chunmin]]

(In reply to Matthew Gregan [:kinetik] from comment #3)
Thanks for sharing your opinions!

> The original channel layout has been forgotten by the time the audio data
> reaches DecodedAudioDataSink since the data is being passed to
> AudioConverter for conversion to SMPTE layout as it's decoded, so you'd
> either need to plumb the original channel layout through from the
> demuxer/decoder it was read from or report the telemetry from those places
> directly.
I think I need to add the layout information manually if I'd like to get layout information when finishing decoding metadata. AFAIK, we can't read the layout information from metadata now. We can add them in [0] based on media source's format like [1](and other format besides Vorbis). 

Therefore, we're able to get the layout information when DecodedAudioDataSink is created[2] and use this information to initialize the AudioStream when AudioQueue is empty. It's a better solution than asking the preferred layout of the user's audio device[4]. 

> It'd probably also be useful to know the common channel layouts and
> speaker/channel counts of the target output audio devices as well, since we
> don't know how common non-stereo presentation is in the wild.
If we'd like to collect data of the users' audio device, we could use |cubeb_get_preferred_channel_layout|, which should return the information of the default device setting. However, we only support SMPTE layout for now. Thus, if the default layout is non-SMPTE, it will return |CUBEB_LAYOUT_UNDEFINED| and we have no idea about what it is. If we could collect the data of channel counts by |cubeb_get_max_channel_count| and associate them with layout data, it would help us to guess what the possible layout they are.


[0] http://searchfox.org/mozilla-central/rev/672c83ed65da286b68be1d02799c35fdd14d0134/dom/media/ogg/OggCodecState.cpp#782
[1] http://searchfox.org/mozilla-central/rev/672c83ed65da286b68be1d02799c35fdd14d0134/dom/media/platforms/agnostic/OpusDecoder.cpp#128
[2] http://searchfox.org/mozilla-central/rev/b1aadb3572eaf7d2c70e19a2ba5413809d9ac698/dom/media/mediasink/DecodedAudioDataSink.cpp#43
[3] http://searchfox.org/mozilla-central/rev/672c83ed65da286b68be1d02799c35fdd14d0134/dom/media/mediasink/DecodedAudioDataSink.cpp#204
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1321502#c18

Comment 5

[C.M.Chang[:chunmin]]

> If we only support downmixing for audio 5.1, I think it would be better to expose the downmix_3f2 and use it directly rather than hiding the inconsistent downmixing mechanism behind cubeb_downmix_*.

repost from: https://github.com/kinetiknz/cubeb/pull/234#discussion_r105833040