Closed Bug 1537986 Opened 5 years ago Closed 4 years ago

Video MediaStreamTrack.getSettings returns empty object with media.navigator.streams.fake = true

Categories

(Core :: WebRTC, defect, P3)

Product:
Core
Component:
WebRTC
Version:
66 Branch
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla69
Tracking Flags:
Tracking Status
firefox69 --- fixed

People

(Reporter: devin.wilson, Assigned: pehrsons)

References

Details

Attachments

(13 files)

Bug 1537986 - Make MediaSourceEngine::GetSettings mandatory. r?jib
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Implement GetSettings for fake devices. r?jib
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Implement GetSettings for audio capture. r?padenot
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Implement GetSettings for the tab source. r?jib
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Add deviceId and groupId to all camera and microphone settings. r?jib!, r?smaug!
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Add facingMode to MediaEngineRemoteVideoSource's settings. r?jib
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Update WPT expectations. r?jib
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Make WPT for opening the same video device at different resolutions spec compliant. r?jib
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Remove dom:: prefixes in MediaManager. r?jib
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Async/await-ify remaining test cases in MST-getSettings.https.html. r?jib
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Merge createAudioStreamAndGetSettings and createVideoStreamAndGetSettings. r?jib
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Make deviceId constraints in MST-getSettings.https.html use `exact`. r?jib
47 bytes, text/x-phabricator-request
Details | Review
Bug 1537986 - Check that groupId test works on non-empty list of input devices. r?jib
47 bytes, text/x-phabricator-request
Details | Review

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36

Steps to reproduce:

  1. Open Firefox 66
  2. Navigate to about:config and set the "media.navigator.streams.fake" option to true
  3. Get a video track using getUserMedia
    const stream = await navigator.mediaDevices.getUserMedia({ video: true });
  4. Get the track settings
    `const settings = stream.getTracks()[0].getSettings();

Actual results:

settings is an empty object:
{}

This makes it harder to have automated tests validate the resolution of a stream without a video element.

Expected results:

settings should be an object with resolution and frame rate:
{
frameRate: 30,
height: 480,
width: 640
}

Component: Untriaged → WebRTC
Product: Firefox → Core
Has STR: --- → yes

Jan-Ivar, do you know who should look at this?

Flags: needinfo?(jib)

Unfortunately, we don't support constraints in our fake devices. This is a long standing issue. See bug 1088621.

Flags: needinfo?(jib)
Status: UNCONFIRMED → NEW
Rank: 22
Ever confirmed: true
Priority: -- → P3

Would it be possible to at least expose the actual dimensions of the stream even if you can't use constraints to request a specific resolution? Is it difficult to add constraints and/or settings support to the fake devices or is it more a matter of priority?

Assignee: nobody → apehrson
Status: NEW → ASSIGNED
Rank: 22

This will help our WPT stats.

Depends on D32960

Depends on D32965

This is not mandated behavior by the spec.

Depends on D32966

Attachment #9068297 - Attachment description: Bug 1537986 - Add deviceId and groupId to all MediaEngineSources' settings. r?jib → Bug 1537986 - Add deviceId and groupId to all camera and microphone settings. r?jib
Attachment #9068300 - Attachment description: Bug 1537986 - Remove WPT for opening the same video device at different resolutions. r?jib → Bug 1537986 - Make WPT for opening the same video device at different resolutions spec compliant. r?jib
Attachment #9068297 - Attachment description: Bug 1537986 - Add deviceId and groupId to all camera and microphone settings. r?jib → Bug 1537986 - Add deviceId and groupId to all camera and microphone settings. r?jib!, r?smaug!
Pushed by pehrsons@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/f1b4c01cf3b9
Make MediaSourceEngine::GetSettings mandatory. r=jib
https://hg.mozilla.org/integration/autoland/rev/ec41532452ed
Implement GetSettings for fake devices. r=jib
https://hg.mozilla.org/integration/autoland/rev/257a8501db35
Implement GetSettings for audio capture. r=jib
https://hg.mozilla.org/integration/autoland/rev/4f3d4d29aaa3
Implement GetSettings for the tab source. r=jib
https://hg.mozilla.org/integration/autoland/rev/e046f15e4dbd
Add deviceId and groupId to all camera and microphone settings. r=jib,smaug
https://hg.mozilla.org/integration/autoland/rev/1e0ef2e38659
Add facingMode to MediaEngineRemoteVideoSource's settings. r=jib
https://hg.mozilla.org/integration/autoland/rev/3a46c2b53953
Remove dom:: prefixes in MediaManager. r=jib
https://hg.mozilla.org/integration/autoland/rev/c0f5518234b9
Update WPT expectations. r=jib
https://hg.mozilla.org/integration/autoland/rev/0620e8b80b92
Make WPT for opening the same video device at different resolutions spec compliant. r=jib
https://hg.mozilla.org/integration/autoland/rev/04b0f6e76803
Async/await-ify remaining test cases in MST-getSettings.https.html. r=jib
https://hg.mozilla.org/integration/autoland/rev/4be44814ed94
Merge createAudioStreamAndGetSettings and createVideoStreamAndGetSettings. r=jib
https://hg.mozilla.org/integration/autoland/rev/a1d860f55aab
Make deviceId constraints in MST-getSettings.https.html use `exact`. r=jib
https://hg.mozilla.org/integration/autoland/rev/fd8eaee557a5
Check that groupId test works on non-empty list of input devices. r=jib
See Also: → 1307808

Firefox 69 getSettings() called on remote WebRTC MediaStreamTrack and MediaStreamTrack from HTMLMediaElement.captureStream() return an empty object. getConstraints() returns Object { mediaSource: "camera" }. Is that the expected behaviour?

(In reply to guest271314 from comment #21)

Firefox 69 getSettings() called on remote WebRTC MediaStreamTrack and MediaStreamTrack from HTMLMediaElement.captureStream() return an empty object.

Expected.

getConstraints() returns Object { mediaSource: "camera" }. Is that the expected behaviour?

Not expected. I see this in release too. Filed bug 1560907.

@(In reply to Andreas Pehrson [:pehrsons] from comment #22)

(In reply to guest271314 from comment #21)

Firefox 69 getSettings() called on remote WebRTC MediaStreamTrack and MediaStreamTrack from HTMLMediaElement.captureStream() return an empty object.

Expected.

getConstraints() returns Object { mediaSource: "camera" }. Is that the expected behaviour?

Not expected. I see this in release too. Filed bug 1560907.

Why does getSettings() for a MediaStreamTrack from <video>.captureStream() not provide at least the width and height of the underlying image (frame)? Why is an empty object expected?

Chromium 75-77 does provide details of the MediaStreamTrack when getSettings() is executed.

We don't implement getSettings() for other tracks than gUM tracks. The method is exposed but internally it's empty. If you want a workaround you can look at canvas.width and canvas.height instead.

(In reply to Andreas Pehrson [:pehrsons] from comment #25)

We don't implement getSettings() for other tracks than gUM tracks.

Why? Is Mozilla in accord with the specifications relevant to not implementing getSettings() for other than gUM tracks?

We don't implement it right now. Patches welcome!

have not yet learned C++ or Rust, else would have already gotten rid of mozCaptureStream(), removed restriction on MediaRecorder stopping when <video> src changed, and ported mkvmerge to JavaScript, just for the utility. Perhaps it is now time to finally dive in to a language other than JavaScript.

(In reply to Andreas Pehrson [:pehrsons] from comment #28)

We don't implement it right now. Patches welcome!

Recommended reading for contributing to Mozilla source code (to write patches)?

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/18207 for changes under testing/web-platform/tests

(In reply to guest271314 from comment #27)

(In reply to Andreas Pehrson [:pehrsons] from comment #25)

We don't implement getSettings() for other tracks than gUM tracks.

Why? Is Mozilla in accord with the specifications relevant to not implementing getSettings() for other than gUM tracks?

Note the mediacapture-main spec says: "The constraints specified in this specification apply only to instances of MediaStreamTrack generated by getUserMedia(), unless stated otherwise in other specifications."

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #35)

(In reply to guest271314 from comment #27)

(In reply to Andreas Pehrson [:pehrsons] from comment #25)

We don't implement getSettings() for other tracks than gUM tracks.

Why? Is Mozilla in accord with the specifications relevant to not implementing getSettings() for other than gUM tracks?

Note the mediacapture-main spec says: "The constraints specified in this specification apply only to instances of MediaStreamTrack generated by getUserMedia(), unless stated otherwise in other specifications."

Note that MediaStreamTrack at Media Stream Recording specification https://www.w3.org/TR/mediastream-recording/#overview

  1. Overview
    This API attempts to make basic recording very simple, while still allowing for more complex use cases. In the simplest case, the application instantiates a MediaRecorder object, calls start() and then calls stop() or waits for the MediaStreamTrack(s) to be ended.

and Media Capture from DOM Elements https://www.w3.org/TR/mediacapture-fromelement/#dfn-mediastreamtrack

  1. HTML Media Element Media Capture Extensions
    Method captureStream()is defined on HTML [HTML5] media elements. Methods for capture are added to both HTMLMediaElement and HTMLCanvasElement.

Both MediaStream and HTMLMediaElement expose the concept of a track. Since there is no common type used for HTMLMediaElement , this document uses the term track to refer to either VideoTrack or AudioTrack. MediaStreamTrack is used to identify the media in a MediaStream.

both links to Media Capture and Stream specification https://www.w3.org/TR/mediacapture-streams/#mediastreamtrack which means that "unless otherwise stated" at Media Stream Recording directly incorporates by reference MediaStreamTrack at Media Capture and Streams. thus getSettings() should return the relevant constraints of a MediaStreamTrack when captureStream() is executed to get the MediaStreamTrack.

(In reply to guest271314 from comment #36)

both links to Media Capture and Stream specification https://www.w3.org/TR/mediacapture-streams/#mediastreamtrack which means that "unless otherwise stated" at Media Stream Recording directly incorporates by reference MediaStreamTrack at Media Capture and Streams. thus getSettings() should return the relevant constraints of a MediaStreamTrack when captureStream() is executed to get the MediaStreamTrack.

"unless otherwise stated" is not a quote from anywhere I can find. Yes, those specifications normatively say the getSettings() method must be implemented for tracks of those sources, but do not say to return anything, or to implement any constrainable properties for those sources.

This used to be under-specified, but got the stronger mediacapture-main language recently, because implementers need specific instructions on how to implement constraints for a specific source, since it's not always obvious how to do so. That's why mediacapture-main now admonishes each spec to spell out not only the constraints an implementer is required to support, but also how they must work.

There are many examples of constraints unique to a source (e.g. facingMode), and even where there's overlap (e.g. width) there can be subtle differences (screen capture tracks, for instance, must not crop because that might lose important info and/or window borders and look bad, whereas camera tracks were (and still are) allowed to crop to create mock 16:9 out of 4:3 etc. Also resizeMode isn't "none" on retina displays.)

Another example is remote peer connection tracks, where there's questions around whether lowering width should impact the network congestion algorithm, or whether resizeMode = "none" would guarantee the dimensions sent by the other peer (probably not, because that's quite literally action at a distance). So right now constraints on remote tracks might be left to an extension spec, even though Chrome has some support.

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #37)

(In reply to guest271314 from comment #36)

both links to Media Capture and Stream specification https://www.w3.org/TR/mediacapture-streams/#mediastreamtrack which means that "unless otherwise stated" at Media Stream Recording directly incorporates by reference MediaStreamTrack at Media Capture and Streams. thus getSettings() should return the relevant constraints of a MediaStreamTrack when captureStream() is executed to get the MediaStreamTrack.

"unless otherwise stated" is not a quote from anywhere I can find. Yes, those specifications normatively say the getSettings() method must be implemented for tracks of those sources, but do not say to return anything, or to implement any constrainable properties for those sources.

This used to be under-specified, but got the stronger mediacapture-main language recently, because implementers need specific instructions on how to implement constraints for a specific source, since it's not always obvious how to do so. That's why mediacapture-main now admonishes each spec to spell out not only the constraints an implementer is required to support, but also how they must work.

There are many examples of constraints unique to a source (e.g. facingMode), and even where there's overlap (e.g. width) there can be subtle differences (screen capture tracks, for instance, must not crop because that might lose important info and/or window borders and look bad, whereas camera tracks were (and still are) allowed to crop to create mock 16:9 out of 4:3 etc. Also resizeMode isn't "none" on retina displays.)

Another example is remote peer connection tracks, where there's questions around whether lowering width should impact the network congestion algorithm, or whether resizeMode = "none" would guarantee the dimensions sent by the other peer (probably not, because that's quite literally action at a distance). So right now constraints on remote tracks might be left to an extension spec, even though Chrome has some support.

Informative reply. Am particularly interested in width and height constraints relevant to all variations of a MediaStreamTrack of kind "video". While Chromium does output width and height for getSettings() after captureStream() from a DOM element, Chromium still has issues both outputting the width and height at <video> element and recording the width and height at MediaRecorder implementation https://bugs.chromium.org/p/chromium/issues/detail?id=972470, https://bugs.chromium.org/p/chromium/issues/detail?id=983777. Additionally, when using WebRTC and replaceTrack() the first video track is set to the default value of the <video> element at https://github.com/guest271314/MediaFragmentRecorder/tree/webrtc-replacetrack and the branched variations. For the MediaRecorder specification re width and height filed this PR https://github.com/w3c/mediacapture-record/pull/172. Your comment appear to indicate accompanying PR's should be filed to specify how width and height are determined relevant to the output WebM file. That is, the output width and height should match the values of getSettings().

You need to log in before you can comment on or make changes to this bug.