1271918 - Firefox doesn't support multiple simultaneous GUM video stream resolutions

Status: RESOLVED WONTFIX

(Core :: WebRTC, enhancement, P3)

Description

[patrik]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2611.0 Safari/537.36

Steps to reproduce:

On the same page, call getUserMedia two times, with different video constraints (eg. VGA & 720p) and render both videos, like in inline script on example page:

https://orbitbot.github.io/ff-gum-multi-res-bug-report/


Actual results:

Firefox opens two streams with the same resolution instead of separate resolutions, or sometimes fails to open the second stream.


Expected results:

Firefox should have rendered two videos with different resolution.

Chrome & Chromium render two videos with VGA & 720p resolutions, as long as the larger one is requested first.

Comment 1

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

The spec does not dictate the behavior you seek, so this is a feature request, not a bug.

getUserMedia is a device discovery API, and Firefox returns modes native to the user's specific camera(s) only, which has value since you learn what you're working with. Instead, if it always returned what you asked, then you would've discovered nothing.

Most sinks, like HTMLVideoElement will rescale for you, and cropping can be done in CSS [1].
RTCPeerConnection (in Firefox 47) has transformational capabilities (Bug 1244913) [2].

So an argument can be made against premature downscaling.

[1] http://stackoverflow.com/questions/36961228/how-to-keep-11-aspect-ratio-video-all-the-time-in-webrtc/36962250#36962250
[2] http://stackoverflow.com/questions/35516416/firefox-frame-rate-max-constraint/35566115#35566115

On the other hand, the spec does allow (but doesn't require) browsers to mutate the camera stream by downscaling and cropping (effectively creating virtual modes) provided pixels are never invented. Chrome does this, and it's a different approach with different tradeoffs:

You can end up scaling twice.
You can't query the many native modes of the user's camera(s).

I'm reluctant to call this a win, when the sink capabilities of RTCPeerConnection are still being worked out.

That said, we do support downscaling of screensharing (bug 1211656).

We're also improving concurrent camera access a bit (bug 1213397) since it's a bit lax atm. 

I know it's unfortunate that browsers have different positions here. Please try the workarounds I link to, and we're open to feedback.

(In reply to patrik from comment #0)
> Chrome & Chromium render two videos with VGA & 720p resolutions, as long as
> the larger one is requested first.

Note that you have no control over other tabs which may be using the camera at the same time, so I wouldn't rely on this behavior. The camera is a shared global resource, hence the complexity of the applyConstraints model.

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

confirming and marking as an enhancement request.  It's unclear if we'll support this given the spec and the issues mentioned by jan-ivar

Comment 3

[patrik]

I've read/skimmed through most of the links that were provided, can you confirm the functionality: If I call applyConstraints() on the videoTrack I get from getUserMedia, does this change what would be sent over a PeerConnection or is it an alternate way of scaling video elements (or both)?

We have a somewhat complicated use case with WebRTC, where some devices that we are streaming to don't send quality scaling commands to the remote party correctly. We have discovered that we can get reasonable results if we set an upper limit to VGA, but we only know what devices we are streaming to long after the stream has been initialised in the current configuration.

The workaround for Chrome is to open two different resolutions, since we want to use better quality and automatic scaling when it works, but applyConstraints() would work just as well for Firefox if it immediately affects the PC stream.

Comment 4

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to patrik from comment #3)
> If I call applyConstraints() on the videoTrack I get from getUserMedia, does
> this change what would be sent over a PeerConnection

TL;DR: yes.

Assuming the track.applyConstraints() can be satisfied (*) it affects what's output by the hardware camera, which is what's fed into the PeerConnection, so yes, a smaller dimension source means fewer bits to transmit.

The PeerConnection may additionally scale the video down based on bandwidth estimation it does and set parameters like scaleResolutionDownBy and maxBitrate.

> or is it an alternate way of scaling video elements (or both)?

An HTMLVideoElement may or may not do its own scaling (up or down) based on whether its own dimensions are fixed.

> We have a somewhat complicated use case with WebRTC, where some devices that
> we are streaming to don't send quality scaling commands to the remote party
> correctly. We have discovered that we can get reasonable results if we set
> an upper limit to VGA, but we only know what devices we are streaming to
> long after the stream has been initialised in the current configuration.
> 
> The workaround for Chrome is to open two different resolutions, since we
> want to use better quality and automatic scaling when it works, but
> applyConstraints() would work just as well for Firefox if it immediately
> affects the PC stream.

Assuming bandwidth is not an issue, then this should work.

*) Constraints from multiple tabs and requests compete for the single shared hardware camera.

Comment 5

[patrik]

Thanks for your assistance, we solved our issue with the help of applyConstraints and it seems to be working fine.

I guess based on the above comments that implementing the functionality might not be on the on the roadmap, from our perspective this issue can be resolved/closed.

Comment 6

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Great! I think I will close it then, since there's no interested party. We can always reopen it if that changes.