2010427 - Changing original media element volume and mute should not affect audio captured through captureStream()

Status: RESOLVED FIXED

(Core :: Audio/Video: MediaStreamGraph, defect, P2)

Description

[Alastor Wu [:alwu]]

STR.

1. go to https://jsfiddle.net/drqLhbx8/1/ (it needs bug 2007596 and pref media.captureStream.enabled=true)
2. start video
3. press start captureStream button
4. mute the original video

Expected.
5. the new video is still audible

Actual.
5. the new video becomes muted as well

Comment 1

[Alastor Wu [:alwu]]

This is defined in the spec.

Comment 2

[Andreas Pehrson [:pehrsons]]

Note that MediaElementAudioSourceNode should apply volume, so if used in combination with captureStream you should end up with some tracks with applied volume and some without.

Comment 3

[Alastor Wu [:alwu]]

Attachment: Bug 2010427 - part1 : monitor media element's effective volume in order to apply volume change on the media element source node properly.

Comment 4

[Alastor Wu [:alwu]]

Attachment: Bug 2010427 - part2 : no longer apply volume on AudioDecoderInputTrack.

Comment 5

[Alastor Wu [:alwu]]

Attachment: Bug 2010427 - part2 : add a WPT to verify MediaElementAudioSourceNode output reflects HTMLMediaElement effective volume change.

Comment 6

[Alastor Wu [:alwu]]

Attachment: Bug 2010427 - part3 : add a WPT to verify captured audio from captureStream is unaffected by its source element's effective volume change.

Comment 7

[Karl Tomlinson (:karlt)]

Attachment: MediaStream srcObject testcase

1. Play "MediaElementAudioSourceNode" element.
2. Mute "MediaElementAudioSourceNode" element.

Expect: no audio output.
Actual: audio output.

https://phabricator.services.mozilla.com/D279751?id=1194910 fixes this.

Comment 8

[Andreas Pehrson [:pehrsons]]

Would be good to capture these behavior changes in release notes for developers.
To summarize:

• MediaElementAudioSourceNode will now capture audio at the media element's volume also when the media element is playing a MediaStream, per spec.
• mozCaptureStream (and soon captureStream) will capture audio without applying the media element's volume, regardless of the type of source the media element is playing, per spec.

Comment 9

[Alastor Wu [:alwu]]

Attachment: Bug 2010427 - part4 : switch RMS measurement to ScriptProcessorNode and add volume tests.

Comment 10

[Alastor Wu [:alwu]]

Attachment: Bug 2010427 - part5 : add a loopback test for MediaElementAudioSourceNode.

Comment 11

[Alastor Wu [:alwu]]

Attachment: Bug 2010427 - part6 : update test_getUserMedia_audioCapture.

Window audio capture now captures all audio sources at full volume,
regardless of the mute state of individual elements. We will address
this problem further in bug 1864067.

Update the test to expect 5 frequency peaks instead of 3 (including
the muted elements)

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e0f4d3e044ca
https://hg.mozilla.org/integration/autoland/rev/8cd32254da8d
part1 : monitor media element's effective volume in order to apply volume change on the media element source node properly. r=media-playback-reviewers,pehrsons,karlt
https://github.com/mozilla-firefox/firefox/commit/15f528f3afeb
https://hg.mozilla.org/integration/autoland/rev/c23e57301169
part2 : add a WPT to verify MediaElementAudioSourceNode output reflects HTMLMediaElement effective volume change. r=pehrsons,karlt
https://github.com/mozilla-firefox/firefox/commit/935e4940963e
https://hg.mozilla.org/integration/autoland/rev/4adc67f8f311
part3 : add a WPT to verify captured audio from captureStream is unaffected by its source element's effective volume change. r=pehrsons
https://github.com/mozilla-firefox/firefox/commit/ec3af719c70a
https://hg.mozilla.org/integration/autoland/rev/4ae77e6a4810
part4 : switch RMS measurement to ScriptProcessorNode and add volume tests. r=pehrsons
https://github.com/mozilla-firefox/firefox/commit/afdb83876aa4
https://hg.mozilla.org/integration/autoland/rev/cd257972a761
part5 : add a loopback test for MediaElementAudioSourceNode. r=pehrsons
https://github.com/mozilla-firefox/firefox/commit/894cf1a51c74
https://hg.mozilla.org/integration/autoland/rev/0455c603ae10
part6 : update test_getUserMedia_audioCapture. r=webrtc-reviewers,pehrsons

Comment 13

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/57700 for changes under testing/web-platform/tests

Comment 14

[amarc]

https://hg.mozilla.org/mozilla-central/rev/8cd32254da8d
https://hg.mozilla.org/mozilla-central/rev/c23e57301169
https://hg.mozilla.org/mozilla-central/rev/4adc67f8f311
https://hg.mozilla.org/mozilla-central/rev/4ae77e6a4810
https://hg.mozilla.org/mozilla-central/rev/cd257972a761
https://hg.mozilla.org/mozilla-central/rev/0455c603ae10

Comment 15

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 16

[Hamish Willee]

FF149 MDN work for this can be tracked in https://github.com/mdn/content/issues/43215 (was https://github.com/mdn/content/issues/43213)

What documentation did you think should be done?
Support for non-prefixed HTMLMediaElement.captureStream() only came in the same release. So to me this looks like (part of the) "Implementation of that method as you would expect from the specification". So not planning on capturing this.

But if there is specific developer-facing information that needs to be conveyed, please let me know!

Comment 17

[Andreas Pehrson [:pehrsons]]

See comment 8. So mainly developer release notes and that this affects MediaElementAudioSourceNode as well.

Comment 18

[Hamish Willee]

Thank you. This will be captured in https://github.com/mdn/content/pull/43349