1734353 - Muting tab pauses/mutes AudioContext AnalyserNode

Status: NEW

(Core :: Audio/Video: Playback, defect, P3)

Description

[alexanderhausmann]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0

Steps to reproduce:

This happens on Firefox Dev and on the normal Firefox

• create AudioElement and load a blob (e.g. via file input)
• audioElement.captureStream()
• put that into audioContext.createMediaStreamSource()
• connect that StreamSource to an AudioGainNode
• connect that GainNode to an AnalyserNode
• now play the AudioElement
• now mute the tab
• analyserNode.getByteFrequencyData() returns 0 for every frequency

I have a personal just-for-fun project if you want to reproduce this. It's a huge codebase for that simple reproduction, but i don't have time unfortunately. https://gitlab.com/j-avis/j-avis/-/commit/9994568300334c3bee9b3fa00726e94330445d21

• Open the index.html file
• click on the music file icon on the bottom left
• load any sound file

Actual results:

analyserNode.getByteFrequencyData() returns 0 for every frequency

but the audioElement continues to play/cue forward

Expected results:

analyserNode.getByteFrequencyData() continues to give valid return data, even though the tab is muted.

This works perfectly fine in Chrome 94.0.4606.71

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Web Audio' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Karl Tomlinson (:karlt)]

Probably related to MUTED_BY_AUDIO_CHANNEL.

Comment 3

[Alastor Wu [:alwu]]

I don't know much about captureStream() so if the audio is muted, should the captured stream be supposed to still receive audio data? Or it should receive silence?

Comment 4

[Karl Tomlinson (:karlt)]

This certainly is a confusing story.
The intention was that HTMLMediaElement.volume and HTMLMediaElement.muted should affect the audio in streams from mozCaptureStream(), but the spec for captureStream() says otherwise:

Whether a media element is actively rendering content (e.g., to a screen or audio device) has no effect on the content of captured streams. Muting the audio on a media element does not cause the capture to produce silence, nor does hiding a media element cause captured video to stop. Similarly, the audio level or volume of the media element does not affect the volume of captured audio.

So captureStream() should not affect the normal video and audio rendering of the HTMLMediaElement (bug 1178751), and muted and volume only affect this output to speakers without affecting the audio of a MediaStreamTrack.

Gecko doesn't implement captureStream() and so is free to choose how mozCaptureStream() behaves. Keeping the historical behavior wrt volume and muted may be best for site compat.

MediaElementAudioSourceNode is different from captureStream()

the rendered audio will no longer be heard directly, but instead will be heard as a consequence of the MediaElementAudioSourceNode being connected through the routing graph. Thus pausing, seeking, volume, src attribute changes, and other aspects of the HTMLMediaElement MUST behave as they normally would if not used with a MediaElementAudioSourceNode..

With MediaElementAudioSourceNode the HTMLMediaElement no longer outputs directly to speakers, but the normal rendering, including the effects of muted and volume, is redirected to the MediaElementAudioSourceNode.

That's all for HTMLMediaElement.volume and HTMLMediaElement.muted.
I assume the reporter is using mozCaptureStream() rather than captureStream().

However, probably the key part of this report is that HTMLMediaElement.volume and HTMLMediaElement.muted are not changed. What is changed is the mute on the tab.

For the tab mute, the expectation is that this affects audio rendered to speakers rather than audio captured to a MediaStreamTrack.

Comment 5

[Alastor Wu [:alwu]]

probably the key part of this report is that HTMLMediaElement.volume and HTMLMediaElement.muted are not changed. What is changed is the mute on the tab.

Muting a tab is not only affecting the audio element source, it would also affect the destination node as well. Does that (changing that output volume) affect the analyser node?

Comment 6

[guest271314]

Attachment: file_input_firefox.html

Cannot reproduce on Firefox 92.

Comment 7

[guest271314]

• now mute the tab

Does this mean mute the tab using keyboard mute button or HTMLMediaElement.muted attribute or control?

Comment 8

[guest271314]

Attachment: file_input_firefox.html

Firefox 93 and Chromium 97 appear to have the same behaviour.

Comment 9

[Karl Tomlinson (:karlt)]

Thanks for the test case.
The default value of smoothingTimeConstant means the output takes multiple samples to decay (values other than zero are not so useful), but this does reproduce with 92.0b8.

(In reply to Alastor Wu [:alwu] from comment #5)

Muting a tab is not only affecting the audio element source, it would also affect the destination node as well. Does that (changing that output volume) affect the analyser node?

No, that only affects data sent to audio output devices. AnalyserNode is upstream.

Comment 10

[Alastor Wu [:alwu]]

However, probably the key part of this report is that HTMLMediaElement.volume and HTMLMediaElement.muted are not changed. What is changed is the mute on the tab. For the tab mute, the expectation is that this affects audio rendered to speakers rather than audio captured to a MediaStreamTrack.

For all different kind of muting, things are all the same, what we will do is to set the volume to the decoder or the media stream renderer. In the capturing case, this seems setting the volume on output device only?

Comment 11

[Karl Tomlinson (:karlt)]

(In reply to Alastor Wu [:alwu] from comment #10)

For all different kind of muting, things are all the same,

Yes, but the expectation is that tab muting will sometimes behave differently to setting HTMLMediaElement.muted.

what we will do is to set the volume to the decoder or the media stream renderer. In the capturing case, this seems setting the volume on output device only?

mMediaStreamRenderer is used only for source streams and so not involved in this bug.

I assume the MediaDecoder volume is forwarded to the DecodedStream, which would then be applied to the AudioDecoderInputTrack (rather than any associated audio output device).

Comment 12

[Alastor Wu [:alwu]]

Ah, it sounds reasonable. It seems that we need to apply volume to this output track?

Comment 13

[Karl Tomlinson (:karlt)]

I'm not familiar with the reason for having two tracks mAudioTrack and mAudioOutputTrack nor the specific purpose of each.
I would expect that volume effects from HTMLMediaElement.volume and HTMLMediaElement.muted should be applied to one of those, but tab muting should apply to neither.

With audio capture, mAudioOutputTrack would not represent an audio output device, but just the audio stream, which may be connected to an AnalyserNode. If it does get connected to a track having an audio output device, then the tab muting of AudioDestinationNode and/or mMediaStreamRenderer would already ensure than no audio is heard.