1537986 - Video MediaStreamTrack.getSettings returns empty object with media.navigator.streams.fake = true

Status: RESOLVED FIXED

(Core :: WebRTC, defect, P3)

Description

[devin.wilson]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36

Steps to reproduce:

1. Open Firefox 66
2. Navigate to about:config and set the "media.navigator.streams.fake" option to true
3. Get a video track using getUserMedia
   const stream = await navigator.mediaDevices.getUserMedia({ video: true });
4. Get the track settings
   `const settings = stream.getTracks()[0].getSettings();

Actual results:

settings is an empty object:
{}

This makes it harder to have automated tests validate the resolution of a stream without a video element.

Expected results:

settings should be an object with resolution and frame rate:
{
frameRate: 30,
height: 480,
width: 640
}

Comment 1

[Michael Froman [:mjf]]

Jan-Ivar, do you know who should look at this?

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Unfortunately, we don't support constraints in our fake devices. This is a long standing issue. See bug 1088621.

Comment 5

[devin.wilson]

Would it be possible to at least expose the actual dimensions of the stream even if you can't use constraints to request a specific resolution? Is it difficult to add constraints and/or settings support to the fake devices or is it more a matter of priority?

Comment 6

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Make MediaSourceEngine::GetSettings mandatory. r?jib

Comment 7

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Implement GetSettings for fake devices. r?jib

This will help our WPT stats.

Depends on D32960

Comment 8

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Implement GetSettings for audio capture. r?padenot

Depends on D32961

Comment 9

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Implement GetSettings for the tab source. r?jib

Depends on D32962

Comment 10

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Add deviceId and groupId to all camera and microphone settings. r?jib!, r?smaug!

Depends on D32963

Comment 11

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Add facingMode to MediaEngineRemoteVideoSource's settings. r?jib

Depends on D32964

Comment 12

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Update WPT expectations. r?jib

Depends on D32965

Comment 13

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Make WPT for opening the same video device at different resolutions spec compliant. r?jib

This is not mandated behavior by the spec.

Depends on D32966

Comment 14

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Remove dom:: prefixes in MediaManager. r?jib

Depends on D32965

Comment 15

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Async/await-ify remaining test cases in MST-getSettings.https.html. r?jib

Depends on D32967

Comment 16

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Merge createAudioStreamAndGetSettings and createVideoStreamAndGetSettings. r?jib

Comment 17

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Make deviceId constraints in MST-getSettings.https.html use `exact`. r?jib

Depends on D33470

Comment 18

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1537986 - Check that groupId test works on non-empty list of input devices. r?jib

Depends on D33472

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f1b4c01cf3b9
Make MediaSourceEngine::GetSettings mandatory. r=jib
https://hg.mozilla.org/integration/autoland/rev/ec41532452ed
Implement GetSettings for fake devices. r=jib
https://hg.mozilla.org/integration/autoland/rev/257a8501db35
Implement GetSettings for audio capture. r=jib
https://hg.mozilla.org/integration/autoland/rev/4f3d4d29aaa3
Implement GetSettings for the tab source. r=jib
https://hg.mozilla.org/integration/autoland/rev/e046f15e4dbd
Add deviceId and groupId to all camera and microphone settings. r=jib,smaug
https://hg.mozilla.org/integration/autoland/rev/1e0ef2e38659
Add facingMode to MediaEngineRemoteVideoSource's settings. r=jib
https://hg.mozilla.org/integration/autoland/rev/3a46c2b53953
Remove dom:: prefixes in MediaManager. r=jib
https://hg.mozilla.org/integration/autoland/rev/c0f5518234b9
Update WPT expectations. r=jib
https://hg.mozilla.org/integration/autoland/rev/0620e8b80b92
Make WPT for opening the same video device at different resolutions spec compliant. r=jib
https://hg.mozilla.org/integration/autoland/rev/04b0f6e76803
Async/await-ify remaining test cases in MST-getSettings.https.html. r=jib
https://hg.mozilla.org/integration/autoland/rev/4be44814ed94
Merge createAudioStreamAndGetSettings and createVideoStreamAndGetSettings. r=jib
https://hg.mozilla.org/integration/autoland/rev/a1d860f55aab
Make deviceId constraints in MST-getSettings.https.html use `exact`. r=jib
https://hg.mozilla.org/integration/autoland/rev/fd8eaee557a5
Check that groupId test works on non-empty list of input devices. r=jib

Comment 20

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/f1b4c01cf3b9
https://hg.mozilla.org/mozilla-central/rev/ec41532452ed
https://hg.mozilla.org/mozilla-central/rev/257a8501db35
https://hg.mozilla.org/mozilla-central/rev/4f3d4d29aaa3
https://hg.mozilla.org/mozilla-central/rev/e046f15e4dbd
https://hg.mozilla.org/mozilla-central/rev/1e0ef2e38659
https://hg.mozilla.org/mozilla-central/rev/3a46c2b53953
https://hg.mozilla.org/mozilla-central/rev/c0f5518234b9
https://hg.mozilla.org/mozilla-central/rev/0620e8b80b92
https://hg.mozilla.org/mozilla-central/rev/04b0f6e76803
https://hg.mozilla.org/mozilla-central/rev/4be44814ed94
https://hg.mozilla.org/mozilla-central/rev/a1d860f55aab
https://hg.mozilla.org/mozilla-central/rev/fd8eaee557a5

Comment 21

[guest271314]

Firefox 69 getSettings() called on remote WebRTC MediaStreamTrack and MediaStreamTrack from HTMLMediaElement.captureStream() return an empty object. getConstraints() returns Object { mediaSource: "camera" }. Is that the expected behaviour?

Comment 22

[Andreas Pehrson [:pehrsons]]

(In reply to guest271314 from comment #21)

Firefox 69 getSettings() called on remote WebRTC MediaStreamTrack and MediaStreamTrack from HTMLMediaElement.captureStream() return an empty object.

Expected.

getConstraints() returns Object { mediaSource: "camera" }. Is that the expected behaviour?

Not expected. I see this in release too. Filed bug 1560907.

Comment 23

[guest271314]

@(In reply to Andreas Pehrson [:pehrsons] from comment #22)

(In reply to guest271314 from comment #21)

Firefox 69 getSettings() called on remote WebRTC MediaStreamTrack and MediaStreamTrack from HTMLMediaElement.captureStream() return an empty object.

Expected.

getConstraints() returns Object { mediaSource: "camera" }. Is that the expected behaviour?

Not expected. I see this in release too. Filed bug 1560907.

Why does getSettings() for a MediaStreamTrack from <video>.captureStream() not provide at least the width and height of the underlying image (frame)? Why is an empty object expected?

Comment 24

[guest271314]

Chromium 75-77 does provide details of the MediaStreamTrack when getSettings() is executed.

Comment 25

[Andreas Pehrson [:pehrsons]]

We don't implement getSettings() for other tracks than gUM tracks. The method is exposed but internally it's empty. If you want a workaround you can look at canvas.width and canvas.height instead.

Comment 27

[guest271314]

(In reply to Andreas Pehrson [:pehrsons] from comment #25)

We don't implement getSettings() for other tracks than gUM tracks.

Why? Is Mozilla in accord with the specifications relevant to not implementing getSettings() for other than gUM tracks?

Comment 28

[Andreas Pehrson [:pehrsons]]

We don't implement it right now. Patches welcome!

Comment 29

[guest271314]

have not yet learned C++ or Rust, else would have already gotten rid of mozCaptureStream(), removed restriction on MediaRecorder stopping when <video> src changed, and ported mkvmerge to JavaScript, just for the utility. Perhaps it is now time to finally dive in to a language other than JavaScript.

Comment 30

[guest271314]

(In reply to Andreas Pehrson [:pehrsons] from comment #28)

We don't implement it right now. Patches welcome!

Recommended reading for contributing to Mozilla source code (to write patches)?

Comment 31

[Andreas Pehrson [:pehrsons]]

https://wiki.mozilla.org/Contribute

Comment 32

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/18207 for changes under testing/web-platform/tests

Comment 33

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/18207
* Taskcluster (pull_request) (https://tools.taskcluster.net/task-group-inspector/#/XSHpD9TvQiafIJw1hhe9pQ)

Comment 34

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged

Comment 35

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to guest271314 from comment #27)

(In reply to Andreas Pehrson [:pehrsons] from comment #25)

We don't implement getSettings() for other tracks than gUM tracks.

Why? Is Mozilla in accord with the specifications relevant to not implementing getSettings() for other than gUM tracks?

Note the mediacapture-main spec says: "The constraints specified in this specification apply only to instances of MediaStreamTrack generated by getUserMedia(), unless stated otherwise in other specifications."

Comment 36

[guest271314]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #35)

(In reply to guest271314 from comment #27)

(In reply to Andreas Pehrson [:pehrsons] from comment #25)

We don't implement getSettings() for other tracks than gUM tracks.

Why? Is Mozilla in accord with the specifications relevant to not implementing getSettings() for other than gUM tracks?

Note the mediacapture-main spec says: "The constraints specified in this specification apply only to instances of MediaStreamTrack generated by getUserMedia(), unless stated otherwise in other specifications."

Note that MediaStreamTrack at Media Stream Recording specification https://www.w3.org/TR/mediastream-recording/#overview

1. Overview
   This API attempts to make basic recording very simple, while still allowing for more complex use cases. In the simplest case, the application instantiates a MediaRecorder object, calls start() and then calls stop() or waits for the MediaStreamTrack(s) to be ended.

and Media Capture from DOM Elements https://www.w3.org/TR/mediacapture-fromelement/#dfn-mediastreamtrack

3. HTML Media Element Media Capture Extensions
   Method captureStream()is defined on HTML [HTML5] media elements. Methods for capture are added to both HTMLMediaElement and HTMLCanvasElement.

Both MediaStream and HTMLMediaElement expose the concept of a track. Since there is no common type used for HTMLMediaElement , this document uses the term track to refer to either VideoTrack or AudioTrack. MediaStreamTrack is used to identify the media in a MediaStream.

both links to Media Capture and Stream specification https://www.w3.org/TR/mediacapture-streams/#mediastreamtrack which means that "unless otherwise stated" at Media Stream Recording directly incorporates by reference MediaStreamTrack at Media Capture and Streams. thus getSettings() should return the relevant constraints of a MediaStreamTrack when captureStream() is executed to get the MediaStreamTrack.

Comment 37

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to guest271314 from comment #36)

both links to Media Capture and Stream specification https://www.w3.org/TR/mediacapture-streams/#mediastreamtrack which means that "unless otherwise stated" at Media Stream Recording directly incorporates by reference MediaStreamTrack at Media Capture and Streams. thus getSettings() should return the relevant constraints of a MediaStreamTrack when captureStream() is executed to get the MediaStreamTrack.

"unless otherwise stated" is not a quote from anywhere I can find. Yes, those specifications normatively say the getSettings() method must be implemented for tracks of those sources, but do not say to return anything, or to implement any constrainable properties for those sources.

This used to be under-specified, but got the stronger mediacapture-main language recently, because implementers need specific instructions on how to implement constraints for a specific source, since it's not always obvious how to do so. That's why mediacapture-main now admonishes each spec to spell out not only the constraints an implementer is required to support, but also how they must work.

There are many examples of constraints unique to a source (e.g. facingMode), and even where there's overlap (e.g. width) there can be subtle differences (screen capture tracks, for instance, must not crop because that might lose important info and/or window borders and look bad, whereas camera tracks were (and still are) allowed to crop to create mock 16:9 out of 4:3 etc. Also resizeMode isn't "none" on retina displays.)

Another example is remote peer connection tracks, where there's questions around whether lowering width should impact the network congestion algorithm, or whether resizeMode = "none" would guarantee the dimensions sent by the other peer (probably not, because that's quite literally action at a distance). So right now constraints on remote tracks might be left to an extension spec, even though Chrome has some support.

Comment 38

[guest271314]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #37)

(In reply to guest271314 from comment #36)

both links to Media Capture and Stream specification https://www.w3.org/TR/mediacapture-streams/#mediastreamtrack which means that "unless otherwise stated" at Media Stream Recording directly incorporates by reference MediaStreamTrack at Media Capture and Streams. thus getSettings() should return the relevant constraints of a MediaStreamTrack when captureStream() is executed to get the MediaStreamTrack.

"unless otherwise stated" is not a quote from anywhere I can find. Yes, those specifications normatively say the getSettings() method must be implemented for tracks of those sources, but do not say to return anything, or to implement any constrainable properties for those sources.

This used to be under-specified, but got the stronger mediacapture-main language recently, because implementers need specific instructions on how to implement constraints for a specific source, since it's not always obvious how to do so. That's why mediacapture-main now admonishes each spec to spell out not only the constraints an implementer is required to support, but also how they must work.

There are many examples of constraints unique to a source (e.g. facingMode), and even where there's overlap (e.g. width) there can be subtle differences (screen capture tracks, for instance, must not crop because that might lose important info and/or window borders and look bad, whereas camera tracks were (and still are) allowed to crop to create mock 16:9 out of 4:3 etc. Also resizeMode isn't "none" on retina displays.)

Another example is remote peer connection tracks, where there's questions around whether lowering width should impact the network congestion algorithm, or whether resizeMode = "none" would guarantee the dimensions sent by the other peer (probably not, because that's quite literally action at a distance). So right now constraints on remote tracks might be left to an extension spec, even though Chrome has some support.

Informative reply. Am particularly interested in width and height constraints relevant to all variations of a MediaStreamTrack of kind "video". While Chromium does output width and height for getSettings() after captureStream() from a DOM element, Chromium still has issues both outputting the width and height at <video> element and recording the width and height at MediaRecorder implementation https://bugs.chromium.org/p/chromium/issues/detail?id=972470, https://bugs.chromium.org/p/chromium/issues/detail?id=983777. Additionally, when using WebRTC and replaceTrack() the first video track is set to the default value of the <video> element at https://github.com/guest271314/MediaFragmentRecorder/tree/webrtc-replacetrack and the branched variations. For the MediaRecorder specification re width and height filed this PR https://github.com/w3c/mediacapture-record/pull/172. Your comment appear to indicate accompanying PR's should be filed to specify how width and height are determined relevant to the output WebM file. That is, the output width and height should match the values of getSettings().