1131861 - 'frameRate' constraint for getUserMedia doesn't work on OS X cameras

Status: RESOLVED DUPLICATE of bug 1273734

(Core :: WebRTC: Audio/Video, defect, P3)

Description

[louis]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:35.0) Gecko/20100101 Firefox/35.0
Build ID: 20150122214805

Steps to reproduce:

Using Firefox 35.0.1 on Mac OS X 10.10.1, and built in iSight/FaceTime camera.

This JSFiddle (provided by 'jib' on #media) demonstrates the use of 'frameRate' when using getUserMedia: http://jsfiddle.net/34qxx5w1/


Actual results:

The frame rate doesn't change despite 'frameRate' being included in the constraints object passed into getUserMedia.


Expected results:

The frame rate of the video should change to 2fps.

Apparently this works on Windows according to 'jib', but is problematic on OS X. I pointed out that setting frameRate on Chrome works perfectly fine on OS X. This is demonstrated at the following JSFiddle: http://jsfiddle.net/qpdx8p3q/

Comment 1

[Andreas Pehrson [:pehrsons]]

Haakon will take a look at this as his first bug.

Comment 2

[Andreas Pehrson [:pehrsons]]

jib, do you remember why it was problematic (per comment 0) on OSX?

Comment 3

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

As I recall, the problem is that OSX doesn't expose any "capabilities", so for constraints purposes we hardcode our own, and that table is currently hardcoded to 30 fps only [1].

The solution is either to populate that table with more values, or set frameRate to zero in the capabilities (not the constraints) as I mention in my April 24 comment in [2].
ted
Would love to hear from an OSX expert which approach of these approaches we should pick, i.e. whether it's true the OS tries to convert to any frame rate asked for, or whether it is limited to a few undocumented frame rates (or whether it varies by apple hardware).

[1] http://mxr.mozilla.org/mozilla-central/source/dom/media/webrtc/MediaEngineWebRTCVideo.cpp?rev=f52c18aac7ce#151
[2] https://github.com/w3c/mediacapture-main/issues/118

Comment 4

[Haakon Sporsheim [:haaspors] (Telenor)]

I won't classify myself as an OSX expert, though I have fiddled with Apple AV in a previous life.

Now, the video capture backend for OS X is currently based on QTkit which is deprecated in OS 10.9. Apple has a lot of good docs on how to transition from QTkit to AV foundation. Though AV foundation APIs are available from 10.7. AFAIK firefox for OS X is supported on 10.6 and later.

Anyways, as jib points out, for OS X it is hardcoded to 30 fps, which means that trying to specify constraints with something that doesn't have 30fps in it's range will fail to compute any capabilities and the doorhanger will not even show camera selection. As expected I guess.

QTkit will accommodate as best as possible for what's being specified. How QTkit does this is not clear to me. It seems like some interpolation is done when specifying a low frame rate. Whether this is done in software or hardware I do not know. My gut feeling is that this might also vary by Apple HW as jib also points out.

AVFoundation on the other hand has APIs for getting AVCaptureDeviceFormat which also include AVFrameRateRange in plural.

Comment 5

[Haakon Sporsheim [:haaspors] (Telenor)]

I've tried to set the hardcoded fps to 0, which has the effect that the doorhanger lets you choose all devices. However, - since the new capabilities is not fixed, it [1] will call GetBestFormat() which will choose the best format based on the renderer. which isn't the desired effect at all. It then ends up in default width, height and frame. 352x288@30.

What do we want to happen? Are we waiting for w3 to spec this out? Is it already done? (I haven't been following recent changes in this domain). Should we try to use what's specified by constraints instead?

[1] http://mxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/video_engine/vie_capturer.cc#204

Comment 6

[Haakon Sporsheim [:haaspors] (Telenor)]

Attachment: Enable orthogonal FPS constraint for OS X.

Comment 7

[Haakon Sporsheim [:haaspors] (Telenor)]

Attachment: Make sure to set frameRate for QTkit capture device.

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

The main reason for setting capability fps to 0 is to get past this point [1] in the constraints algorithm. There's implicit support for this in the spec as mentioned here [2], so we should be good with the spec.

Once we're past the algorithm, we want the frame rate closest to the ideal provided, or absent that, some default or whatever is available.

I've run across the "GetBestFormat" and "fixed" vs. something-else stuff in the low-level code before and am not sure why it's still there. I think we inherited that from GIPS and it may very well have been superseded by the constraints code, since it is not compatible with it (width and height never be affected the way you describe, and I've only seen those numbers when something is wrong). One way to avoid this code seems to be to fill in the frame rate we want before this point. If you have other ways to bypass it, that's fine too (if you need to drive the decision lower).

> Though AV foundation APIs are available from 10.7. AFAIK firefox for OS X is supported on 10.6 and later.

Thanks for this info! It sounds like supporting AVFoundation should be a win for our constraints algorithm on (newer) OSX. Should we open a bug on it?

[1] http://mxr.mozilla.org/mozilla-central/source/dom/media/webrtc/MediaEngineCameraVideoSource.cpp?rev=53063473eb54&mark=165-166#160
[2] https://github.com/w3c/mediacapture-main/issues/118

Comment 9

[Haakon Sporsheim [:haaspors] (Telenor)]

I don't feel good about the patch I attached in comment 6. It feels dirty. So what do you propose?
 - Updating mHardcodedCapabilities from the constraints provided?
 - Factor mHardcodedCapabilities with maxFPS set [1-30]?
 - Something else?

I think we should definitely open a bug for supporting AVFoundation! I'm not sure how firefox for OSX is built and how we can support it by loading the framework on demand. One of the OSX release engineers could probably answer that!

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Haakon Sporsheim [:haaspors] (Telenor) from comment #9)
> I don't feel good about the patch I attached in comment 6. It feels dirty.

But does it work? :) The patch looks good to me, except we should probably default to mPrefs.mFPS, not c.mFrameRate.mMax (min/max constraints have predefined meaning in the constraints algorithm and should not be read directly for anything). This should cover the case where no frame rate constraint is specified.

> I think we should definitely open a bug for supporting AVFoundation!

I've opened Bug 1180725.

Comment 11

[steinbrecher.johann]

This has been fixed with https://bugzilla.mozilla.org/show_bug.cgi?id=1211658 ?

Comment 12

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

No, unfortunately cameras is a different beast from screensharing.

The problem is that QTkit does not expose camera capabilities at all. For example, if I take the same Logitech camera I use on Windows, and use it on OSX, then the constraint code is flying blind and has no access to the modes available. Instead the software hits the clumsy hardcoded capabilities I added that were based on empirical observation of the built-in Facetime camera on my MBP, and I only added 30 fps in that table (comment 3).

Haakon's patch aside, I'm starting to think that hardcoding 10, 15 and 60 in addition to 30 would be the least disruptive fix at this point.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

which are the frameRates I see on my Logitech camera under Windows.

Comment 14

[Nomis101]

(In reply to Haakon Sporsheim [:haaspors] (Telenor) from comment #4)

> Now, the video capture backend for OS X is currently based on QTkit which is
> deprecated in OS 10.9.
And will be removed completely in macOS 10.12.
https://developer.apple.com/library/prerelease/content/releasenotes/Miscellaneous/APIDiffsMacOS10_12/Objective-C/QTKit.html

Comment 15

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

We now use AVFoundation (bug 1180725), so comment 14 is no longer an issue.

Frame rates should now work better on OSX, though we still seem to conflate frame rate with max frame rate (bug 1273734), as from my logitech 910 and 930e I was only able to get lower frame rates on insanely high resolutions. Therefore, closing as dup of the latter action item.

Comment 16

[Andrew Morris]

(In reply to Jan-Ivar Bruaroey [:jib] from comment #15)
> We now use AVFoundation (bug 1180725), so comment 14 is no longer an issue.
> 
> Frame rates should now work better on OSX, though we still seem to conflate
> frame rate with max frame rate (bug 1273734), as from my logitech 910 and
> 930e I was only able to get lower frame rates on insanely high resolutions.
> Therefore, closing as dup of the latter action item.

With a facetime camera on osx macbook I'm getting a 30fps video with this code:

navigator.mediaDevices.getUserMedia({ video: { frameRate: 10 } }).then(stream => {
  const vid = document.createElement('video');
  vid.srcObject = stream;
  document.body.appendChild(vid);
  vid.play();
}).catch(err => console.error(err));

In fact, I'm getting OverconstraintedError for { frameRate: { max: 29 } }. Does that seem right to you? The facetime camera refuses to do <30fps? Chrome is able to deliver a low frame rate from this camera with { frameRate: { max: 10 } }.

Comment 17

[Andreas Pehrson [:pehrsons]]

(In reply to Andrew Morris from comment #16)
> In fact, I'm getting OverconstraintedError for { frameRate: { max: 29 } }.
> Does that seem right to you? The facetime camera refuses to do <30fps?
> Chrome is able to deliver a low frame rate from this camera with {
> frameRate: { max: 10 } }.

Probably the camera doesn't have any modes below 30fps.

We apply settings directly on the camera, while Chrome puts a layer on top of the stream that can scale, crop, drop, etc., video frames.