1286945 - Offer downscaled resolutions and decimated framerates in getUserMedia.

Status: REOPENED

(Core :: WebRTC: Audio/Video, defect, P3)

Description

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

After making constraints respect concurrent gUM access (bug 1213397), and seeing the issues with averaging and other attempts at resolving conflicts (e.g. bug 1213517 comment 152), plus the fact that we seem to mix FPS with maxFPS in our implementations (bug 1273734), it's becoming clear that we need to consider offering up downscaled resolutions and decimated framerates.

This would be similar to what Chrome does, though we need to answer whether we wish to crop or always preserve aspect (Chrome crops).

Comment 2

[David Jameson]

My preference would be to work the same as chrome, i.e. cropping, which I think is preferable to getting black bars.

The typical use case is that the site requests a 4:3 aspect ratio resolution, and the webcam is widescreen. In this case it is preferable to get the cropped 4:3 image rather than one with black bars at the top and bottom.

Comment 3

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

E.g. Jitsi could ask for { video: { width: 1280, height: 720 }, audio: true }

and it should work well for their needs, defaulting to 1280x720 or the closest to it.

It would even prompt Firefox users once for cam+mic instead of twice for each like they do now.

[1] https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#Parameters

Comment 5

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

Attachment: Bug 1286945 - modify the calculation of fitness distance to support down scaling.

Review commit: https://reviewboard.mozilla.org/r/163960/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/163960/

Comment 6

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

Attachment: Bug 1286945 - support getusermedia continuous constraint by down scaling.

Review commit: https://reviewboard.mozilla.org/r/163962/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/163962/

Comment 7

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

These patches are only WIP.
I would need more time to verify these patches on different platform and scenarios.
I am also working on decimated framerate part.

Comment 8

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

Comment on attachment 8892951 [details]
Bug 1286945 - support getusermedia continuous constraint by down scaling.

https://reviewboard.mozilla.org/r/163962/#review169276

::: dom/media/systemservices/CamerasChild.h:187
(Diff revision 1)
>                     const int capture_id, webrtc::VideoCaptureCapability& capability,
> +                   webrtc::VideoCaptureCapability& constraint,

We want to separate the two different concepts:
1) the configuration we set to the hardware (capability)
2) the constraint user sets (constraint)

For example, user may request a resolution 1000x700, which is not supported by the camera hardware.

We then config camera with the capability 1280x720.

Then downscale the output frame to the constraint 1000x700.

Comment 9

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Current front-runner on behavior is to only downscale when we'd otherwise fail with OverconstrainedError, as described in bug 1388667 comment 10.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Closed wrong bug, sorry.

Comment 14

[fburka]

Is there any progress on this? I still have compressed 4:3 instead of 16:9 with Firefox 74. As there is much more need of WebRTC these quarantine days it would be awesome if Firefox could deliver beautiful video streams with the correct aspect ratio :-) I tried Jitsi, WebEx, talky.io, ... If I try meet.google.com I see the right aspect ratio before joining a room and the wrong 4:3 after joining - so there might be a workaround existing?
Thanks!

Comment 15

[Sam Switzer]

I would also like an update of some sort. Either this or add support for the aspectRatio constraint. I'm currently developing a webapp for taking profile photos That I want to work with as many cameras as possible in their best provided resolution, and it's sad how firefox forces every camera to 640x480 no matter what getusermedia says it's capable of. I've got it working pulling native resolution in all other browsers but Firefox. I also think this is the reason Slack refuses to support firefox for video calls.

Comment 16

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Sam Switzer from comment #15)

I want to work with as many cameras as possible in their best provided resolution, and it's sad how firefox forces every camera to 640x480 no matter what getusermedia says it's capable of. I've got it working pulling native resolution in all other browsers but Firefox.

Hi Sam. Firefox should enumerate all native resolutions of your camera just fine, as you can see here: https://jsfiddle.net/jib1/4p5gqm0o/show

If you are having problems pulling a specific native resolution with that link, please provide info about you camera. Thanks!

Comment 17

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to fburka from comment #14)

I still have compressed 4:3 instead of 16:9 with Firefox 74. As there is much more need of WebRTC these quarantine days it would be awesome if Firefox could deliver beautiful video streams with the correct aspect ratio :-) I tried Jitsi, WebEx, talky.io, ... If I try meet.google.com I see the right aspect ratio before joining a room and the wrong 4:3 after joining - so there might be a workaround existing?

Hi fburka, as you can see in comment 16, Firefox supports all available native modes, and you can't get more "beautiful" than that, as far as quality.

Instead, this bug is about "compressing" higher resolutions down to emulate missing lower resolution modes, like Chrome does, even chopping off the bottom of e.g. 640x480 to produce 640x360 to emulate 16:9.

Unfortunately, the choice of camera resolution rests with the site, so it's hard to know why 4:3 was chosen for you. Many cameras have low-resolution 4:3 modes and high-resolution 16:9 modes. Likely, the site is choosing a lower resolution on purpose, either for perceived network bandwidth reasons or in the case of hangouts/meet, site bugs like bug 1630089 comment 4.

When a low resolution is chosen by the site, you'll likely see it as 4:3 in Firefox (the whole camera frame) vs a 16:9 (cropped native 4:3 camera frame) in Chrome (you're actually seeing more bits in Firefox!)

As to workarounds (for sites, not users):

1. Sites could easily use CSS at playback time on receivers if parity and the 16:9 aesthetic is important (at minor bandwidth cost)
2. If bandwidth is a concern, there are ways to downscale a high-res 16:9 mode in peer connection, using scaleResolutionDownBy but most sites don't seem to bother.
3. File a bug on hangouts/meet to not look at unrelated data like navigator.hardwareConcurrency to determine HD vs VGA (& ★ this one)

We hope to get to this soon, as we realize it's become a web compat problem, as most sites don't seem interested in handling native modes directly.

Comment 20

[marcosdg3]

Excited for this feature, thank you for identifying the root cause of 1645876. Is there an ETA for this or a way I could contribute? We are looking to use this when flash retires this year.

Comment 21

[marcosdg3]

Does anyone know of workaround? Our video applications would really like to support firefox.

Comment 22

[davidswalkabout]

FWIW, I've noticed macOS-specific behavior:
On MacBook Air 13-inch 2020 build running macOS 10.15.5 and visiting https://test.webrtc.org/
FF81: 320x240 not available
Chrome86: 320x240 is available

while in comparison on Dell Inspiron 14-inch model 7490 with builtin webcam (labelled "Integrated Webcam") running Windows 10 Home visiting the same site...
FF81 and Chrome86: 320x240 is available

Comment 23

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

The spec was changed to require this 4 days ago.

Comment 24

[Boris Zbarsky [:bzbarsky]]

For what it's worth, the lack of frame rate decimation caused a web compat problem I ran into a few days ago that was threatening to derail remote school in Firefox for a bunch of students. See https://twitter.com/bytingtheapple/status/1338901952168513537 and preceding thread. Clever put in a workaround, but it would be good to not force people to do that...

Comment 25

[marcosdg3]

For those looking for a workaround using MediaRecorder, used ffmpeg wasm for desktop support. https://ffmpegwasm.github.io/

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:jib, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 31

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S4. However, the bug has 9 duplicates.
:jib, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 33

[Alfredo]

Hi,
any update on this being one of the last pieces together with this https://bugzilla.mozilla.org/show_bug.cgi?id=1531461 to get FaceTime working?
Thanks

Comment 35

[MozilaKhan]

Getting error while recording video from camera by using custom height(720) and width (420) while it's working well in other browsers like chrome, edge and opera.
MediaStreamError { name: "OverconstrainedError", message: "Constraints could be not satisfied.", constraint: "width", stack: "" }