1286945 - Offer downscaled resolutions and decimated framerates in getUserMedia

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, enhancement, P2)

Description

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

After making constraints respect concurrent gUM access (bug 1213397), and seeing the issues with averaging and other attempts at resolving conflicts (e.g. bug 1213517 comment 152), plus the fact that we seem to mix FPS with maxFPS in our implementations (bug 1273734), it's becoming clear that we need to consider offering up downscaled resolutions and decimated framerates.

This would be similar to what Chrome does, though we need to answer whether we wish to crop or always preserve aspect (Chrome crops).

Comment 2

[David Jameson]

My preference would be to work the same as chrome, i.e. cropping, which I think is preferable to getting black bars.

The typical use case is that the site requests a 4:3 aspect ratio resolution, and the webcam is widescreen. In this case it is preferable to get the cropped 4:3 image rather than one with black bars at the top and bottom.

Comment 3

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

E.g. Jitsi could ask for { video: { width: 1280, height: 720 }, audio: true }

and it should work well for their needs, defaulting to 1280x720 or the closest to it.

It would even prompt Firefox users once for cam+mic instead of twice for each like they do now.

[1] https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#Parameters

Comment 5

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

Attachment: Bug 1286945 - modify the calculation of fitness distance to support down scaling.

Review commit: https://reviewboard.mozilla.org/r/163960/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/163960/

Comment 6

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

Attachment: Bug 1286945 - support getusermedia continuous constraint by down scaling.

Review commit: https://reviewboard.mozilla.org/r/163962/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/163962/

Comment 7

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

These patches are only WIP.
I would need more time to verify these patches on different platform and scenarios.
I am also working on decimated framerate part.

Comment 8

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

Comment on attachment 8892951 [details]
Bug 1286945 - support getusermedia continuous constraint by down scaling.

https://reviewboard.mozilla.org/r/163962/#review169276

::: dom/media/systemservices/CamerasChild.h:187
(Diff revision 1)
>                     const int capture_id, webrtc::VideoCaptureCapability& capability,
> +                   webrtc::VideoCaptureCapability& constraint,

We want to separate the two different concepts:
1) the configuration we set to the hardware (capability)
2) the constraint user sets (constraint)

For example, user may request a resolution 1000x700, which is not supported by the camera hardware.

We then config camera with the capability 1280x720.

Then downscale the output frame to the constraint 1000x700.

Comment 9

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Current front-runner on behavior is to only downscale when we'd otherwise fail with OverconstrainedError, as described in bug 1388667 comment 10.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Closed wrong bug, sorry.

Comment 14

[fburka]

Is there any progress on this? I still have compressed 4:3 instead of 16:9 with Firefox 74. As there is much more need of WebRTC these quarantine days it would be awesome if Firefox could deliver beautiful video streams with the correct aspect ratio :-) I tried Jitsi, WebEx, talky.io, ... If I try meet.google.com I see the right aspect ratio before joining a room and the wrong 4:3 after joining - so there might be a workaround existing?
Thanks!

Comment 15

[Sam Switzer]

I would also like an update of some sort. Either this or add support for the aspectRatio constraint. I'm currently developing a webapp for taking profile photos That I want to work with as many cameras as possible in their best provided resolution, and it's sad how firefox forces every camera to 640x480 no matter what getusermedia says it's capable of. I've got it working pulling native resolution in all other browsers but Firefox. I also think this is the reason Slack refuses to support firefox for video calls.

Comment 16

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Sam Switzer from comment #15)

I want to work with as many cameras as possible in their best provided resolution, and it's sad how firefox forces every camera to 640x480 no matter what getusermedia says it's capable of. I've got it working pulling native resolution in all other browsers but Firefox.

Hi Sam. Firefox should enumerate all native resolutions of your camera just fine, as you can see here: https://jsfiddle.net/jib1/4p5gqm0o/show

If you are having problems pulling a specific native resolution with that link, please provide info about you camera. Thanks!

Comment 17

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to fburka from comment #14)

I still have compressed 4:3 instead of 16:9 with Firefox 74. As there is much more need of WebRTC these quarantine days it would be awesome if Firefox could deliver beautiful video streams with the correct aspect ratio :-) I tried Jitsi, WebEx, talky.io, ... If I try meet.google.com I see the right aspect ratio before joining a room and the wrong 4:3 after joining - so there might be a workaround existing?

Hi fburka, as you can see in comment 16, Firefox supports all available native modes, and you can't get more "beautiful" than that, as far as quality.

Instead, this bug is about "compressing" higher resolutions down to emulate missing lower resolution modes, like Chrome does, even chopping off the bottom of e.g. 640x480 to produce 640x360 to emulate 16:9.

Unfortunately, the choice of camera resolution rests with the site, so it's hard to know why 4:3 was chosen for you. Many cameras have low-resolution 4:3 modes and high-resolution 16:9 modes. Likely, the site is choosing a lower resolution on purpose, either for perceived network bandwidth reasons or in the case of hangouts/meet, site bugs like bug 1630089 comment 4.

When a low resolution is chosen by the site, you'll likely see it as 4:3 in Firefox (the whole camera frame) vs a 16:9 (cropped native 4:3 camera frame) in Chrome (you're actually seeing more bits in Firefox!)

As to workarounds (for sites, not users):

1. Sites could easily use CSS at playback time on receivers if parity and the 16:9 aesthetic is important (at minor bandwidth cost)
2. If bandwidth is a concern, there are ways to downscale a high-res 16:9 mode in peer connection, using scaleResolutionDownBy but most sites don't seem to bother.
3. File a bug on hangouts/meet to not look at unrelated data like navigator.hardwareConcurrency to determine HD vs VGA (& ★ this one)

We hope to get to this soon, as we realize it's become a web compat problem, as most sites don't seem interested in handling native modes directly.

Comment 20

[marcosdg3]

Excited for this feature, thank you for identifying the root cause of 1645876. Is there an ETA for this or a way I could contribute? We are looking to use this when flash retires this year.

Comment 21

[marcosdg3]

Does anyone know of workaround? Our video applications would really like to support firefox.

Comment 22

[davidswalkabout]

FWIW, I've noticed macOS-specific behavior:
On MacBook Air 13-inch 2020 build running macOS 10.15.5 and visiting https://test.webrtc.org/
FF81: 320x240 not available
Chrome86: 320x240 is available

while in comparison on Dell Inspiron 14-inch model 7490 with builtin webcam (labelled "Integrated Webcam") running Windows 10 Home visiting the same site...
FF81 and Chrome86: 320x240 is available

Comment 23

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

The spec was changed to require this 4 days ago.

Comment 24

[Boris Zbarsky [:bzbarsky]]

For what it's worth, the lack of frame rate decimation caused a web compat problem I ran into a few days ago that was threatening to derail remote school in Firefox for a bunch of students. See https://twitter.com/bytingtheapple/status/1338901952168513537 and preceding thread. Clever put in a workaround, but it would be good to not force people to do that...

Comment 25

[marcosdg3]

For those looking for a workaround using MediaRecorder, used ffmpeg wasm for desktop support. https://ffmpegwasm.github.io/

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:jib, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 31

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S4. However, the bug has 9 duplicates.
:jib, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 33

[Alfredo]

Hi,
any update on this being one of the last pieces together with this https://bugzilla.mozilla.org/show_bug.cgi?id=1531461 to get FaceTime working?
Thanks

Comment 35

[MozilaKhan]

Getting error while recording video from camera by using custom height(720) and width (420) while it's working well in other browsers like chrome, edge and opera.
MediaStreamError { name: "OverconstrainedError", message: "Constraints could be not satisfied.", constraint: "width", stack: "" }

Comment 36

[fburka]

It might look like a enhancement, but without it my most liked camera is unusable in Chrome as it offers no smaller resolutions and chrome can only handle about 1-2 fps. Therefor I consider this a bug for my usecase. @hodson.

Comment 37

[marcosdg3]

I would consider this a defect as it doesn't meet the current minimum specifications: https://w3c.github.io/mediacapture-main/getusermedia.html#constrainable-properties

Specifically:

width: ... The User Agent MUST support downsampling to any value between the min width range value and the native resolution width...
frameRate ... The User Agent MUST support frame rates obtained from integral decimation of the native resolution frame rate...

Comment 39

[Jim Mathies [:jimm]]

The main blocker here is bug 1433480.

Comment 40

[Andreas Pehrson [:pehrsons]]

A new test for this refactors MediaEngineFakeVideoSource such that it will depend on a webrtc::videocapturemodule::VideoCaptureImpl and therefore needs webrtc::SequenceChecker to function on the MediaManager thread. Bug 1968812 should enable this without too much work in MediaManager.

Comment 41

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Pass NormalizedConstraints to StartCapture through IPC. r?jib!

The parent knowing about constraints enables it to employ various optimizations,
for instance to drop frames that are not wanted by the content process due to
a maximum framerate constraint and resizeMode crop-and-scale, before said
frame gets copied into an IPC shmem buffer.

Comment 42

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Update mozilla-specific video capture scaling WPT for resizeMode. r?jib

Comment 43

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Use "real" fake camera in MediaStreamTrack-applyConstraints.https.html. r?jib

Comment 44

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Fix some grammar in MediaStreamError. r?jib

Comment 45

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Improve casting and fix some warnings in MediaEngineRemoteVideoSource. r?jib

Comment 46

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Refactor DeviceListener::ApplyConstraints away from MediaManager::Dispatch. r?jib

This makes it possible to chain MozPromises instead of having to nest them.

Comment 47

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Update crop-and-scale WPT to differentiate captures that happen to be native vs downscaling actually in effect. r?jib

Comment 48

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Fix some includes. r?jib

Comment 49

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Move GetResizeMode to MediaConstraintsHelper and use Nothing() to denote when it's disabled by pref. r?jib

So it can be used in other modules.

Comment 50

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Add support for resizeMode to MediaEngineFake for platform-independent tests. r?jib

Comment 51

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - In MediaTrackConstraints use std::clamp. r?jib

Comment 52

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Break out StartCapture() from MERVS::Start() and move MERVS::Reconfigure to use it directly. r?jib

Comment 53

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Drop frames based on constraints in CamerasParent to avoid unnecessary IPC traffic and video frame copies. r?jib

Comment 54

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Only skip updating width and height settings when DeliverFrame has already updated them. r?jib

This also changes back domSettings to settings as the former name
was formed to avoid ambiguity in the presence of a different kind of
settings object. The approach with the other kind of settings object
was not taken in the end.

Comment 55

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Simplify Construct() calls when assigning settings. r?jib

Comment 56

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Remove main thread settings update from MERVS::Start. r?jib

Settings are updated in Allocate() instead, based on the capability available
then. For gDM where there are no capabilities, MediaManager will hold the track
until the first frame has updated the settings on main thread.

Comment 57

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Update comments and naming to clarify some MERVS members. r?jib

Comment 58

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Fix settings update on Reconfigure to use correct input size. r?jib

Comment 59

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Update CalculateDesiredSize for resizeMode. r?jib

resizeMode: none is simple enough. resizeMode: crop-and-scale per the below:

Camera:

• If ideal (or exact) constraints are given for both width and height, we crop
  and scale to exactly those.
• If ideal constraints are given for only one of the dimensions, we scale down
  maintaining aspect ratio.
• If max constraints are given for any dimensions, we scale down maintaining
  aspect ratio to satisfy them

Desktop:

• Uses spec-compliant fitness distance for finding correct scale.
  See https://jsfiddle.net/pehrsons/xwbtdn5u/ for this version. Forked from
  https://jsfiddle.net/jib1/tcz74k0w/ which shows the algorithm prior to this
  patch.

Comment 60

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Floor gDM framerate with resizeMode:none at 60 fps. r?jib

Comment 61

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Enable the resizeMode constraint and make the default value "crop-and-scale". r?jib

Comment 62

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Always use 2 for minimum size capability. r?jib

Comment 63

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Only overwrite MediaEnginePrefs::mResizeMode when the pref was read. r?jib

Comment 64

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/8c5a673d5856
https://hg.mozilla.org/integration/autoland/rev/1e45aaf59d00
Pass NormalizedConstraints to StartCapture through IPC. r=jib,ipc-reviewers,mccr8
https://github.com/mozilla-firefox/firefox/commit/1883e28ae9cd
https://hg.mozilla.org/integration/autoland/rev/d9ec243f12e9
Update mozilla-specific video capture scaling WPT for resizeMode. r=jib
https://github.com/mozilla-firefox/firefox/commit/eeb1c842e38f
https://hg.mozilla.org/integration/autoland/rev/21f6b3b499d5
Use "real" fake camera in MediaStreamTrack-applyConstraints.https.html. r=jib
https://github.com/mozilla-firefox/firefox/commit/7ec0dfdd3336
https://hg.mozilla.org/integration/autoland/rev/91d29d55eb8d
Fix some grammar in MediaStreamError. r=jib
https://github.com/mozilla-firefox/firefox/commit/17eb3becca8a
https://hg.mozilla.org/integration/autoland/rev/9cdb5f352ec7
Improve casting and fix some warnings in MediaEngineRemoteVideoSource. r=jib
https://github.com/mozilla-firefox/firefox/commit/8607aa2ac9db
https://hg.mozilla.org/integration/autoland/rev/06ddbdde8450
Refactor DeviceListener::ApplyConstraints away from MediaManager::Dispatch. r=jib
https://github.com/mozilla-firefox/firefox/commit/e21ef232139e
https://hg.mozilla.org/integration/autoland/rev/0df201009ae1
Update crop-and-scale WPT to differentiate captures that happen to be native vs downscaling actually in effect. r=jib
https://github.com/mozilla-firefox/firefox/commit/e62f49b347f2
https://hg.mozilla.org/integration/autoland/rev/11565b2fb25e
Fix some includes. r=jib
https://github.com/mozilla-firefox/firefox/commit/1cdef3b50df6
https://hg.mozilla.org/integration/autoland/rev/dc1c1739bb41
Move GetResizeMode to MediaConstraintsHelper and use Nothing() to denote when it's disabled by pref. r=jib
https://github.com/mozilla-firefox/firefox/commit/edbc9294a0c8
https://hg.mozilla.org/integration/autoland/rev/4430331f065b
Add support for resizeMode to MediaEngineFake for platform-independent tests. r=jib
https://github.com/mozilla-firefox/firefox/commit/d0697da27cd3
https://hg.mozilla.org/integration/autoland/rev/b2ce3a4735cb
In MediaTrackConstraints use std::clamp. r=jib
https://github.com/mozilla-firefox/firefox/commit/2822ce1916a9
https://hg.mozilla.org/integration/autoland/rev/2d84c5e5e5d0
Break out StartCapture() from MERVS::Start() and move MERVS::Reconfigure to use it directly. r=jib
https://github.com/mozilla-firefox/firefox/commit/b0665785fb4c
https://hg.mozilla.org/integration/autoland/rev/90bbc27dc795
Drop frames based on constraints in CamerasParent to avoid unnecessary IPC traffic and video frame copies. r=jib
https://github.com/mozilla-firefox/firefox/commit/9fdfccab3389
https://hg.mozilla.org/integration/autoland/rev/6ec606f3248a
Simplify Construct() calls when assigning settings. r=jib
https://github.com/mozilla-firefox/firefox/commit/3b04337727ac
https://hg.mozilla.org/integration/autoland/rev/61b12c747b95
Remove main thread settings update from MERVS::Start. r=jib
https://github.com/mozilla-firefox/firefox/commit/04154d727d8c
https://hg.mozilla.org/integration/autoland/rev/b0caa39bde3a
Update comments and naming to clarify some MERVS members. r=jib
https://github.com/mozilla-firefox/firefox/commit/5336fd7088b5
https://hg.mozilla.org/integration/autoland/rev/619ed201e092
Fix settings update on Reconfigure to use correct input size. r=jib
https://github.com/mozilla-firefox/firefox/commit/30412797a07b
https://hg.mozilla.org/integration/autoland/rev/61c53a4e4a70
Update CalculateDesiredSize for resizeMode. r=jib
https://github.com/mozilla-firefox/firefox/commit/ff268b4b771e
https://hg.mozilla.org/integration/autoland/rev/c232335e7b93
Floor gDM framerate with resizeMode:none at 60 fps. r=jib
https://github.com/mozilla-firefox/firefox/commit/7ea94e609e98
https://hg.mozilla.org/integration/autoland/rev/76b20634da5e
Enable the resizeMode constraint and make the default value "crop-and-scale". r=jib
https://github.com/mozilla-firefox/firefox/commit/d37fdb30a829
https://hg.mozilla.org/integration/autoland/rev/203361b291c5
Always use 2 for minimum size capability. r=jib
https://github.com/mozilla-firefox/firefox/commit/dc05b5386748
https://hg.mozilla.org/integration/autoland/rev/84867e28e903
Only overwrite MediaEnginePrefs::mResizeMode when the pref was read. r=jib

Comment 65

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/54579 for changes under testing/web-platform/tests

Comment 66

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1286945 - Remove the unused DEFAULT_AUDIO_TIMER_MS

Avoids a -Wunused-const-variable warning-as-error.

Comment 67

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/144cd945b547
https://hg.mozilla.org/integration/autoland/rev/fb50a25611e3
Remove the unused DEFAULT_AUDIO_TIMER_MS

Comment 68

[Aaron Train [:aaronmt]]

Build: 2025-08-28 (Autoland)

This is crashing Fenix and Focus tabs on access to camera in all Focus/Fenix ui-test jobs.

STR:

• https://mozilla-mobile.github.io/testapp/v2.0/permissions.html
• Tap Camera and Microphone, accept permissions

Crash

Comment 69

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e779e342bbc7
https://hg.mozilla.org/integration/autoland/rev/6efa011ee7e1
Revert "Bug 1286945 - Remove the unused DEFAULT_AUDIO_TIMER_MS" for causing Fenix anc Focus failures on SitePermissionsTest#allowCameraPermissionsTest

Comment 70

[Cosmin Sabou [:CosminS]]

Please see the above regression for context.

Backed out for causing Fenix anc Focus failures on SitePermissionsTest#allowCameraPermissionsTest

Push with failures

Failure log

Backout link

Comment 71

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a59fca8555b0
https://hg.mozilla.org/integration/autoland/rev/8649de9603d2
Pass NormalizedConstraints to StartCapture through IPC. r=jib,ipc-reviewers,mccr8
https://github.com/mozilla-firefox/firefox/commit/26c2be62f669
https://hg.mozilla.org/integration/autoland/rev/05a226d5261c
Update mozilla-specific video capture scaling WPT for resizeMode. r=jib
https://github.com/mozilla-firefox/firefox/commit/3721345df3c4
https://hg.mozilla.org/integration/autoland/rev/bb08dcd4ac87
Use "real" fake camera in MediaStreamTrack-applyConstraints.https.html. r=jib
https://github.com/mozilla-firefox/firefox/commit/f90ee1e65c0c
https://hg.mozilla.org/integration/autoland/rev/6aaa9b6ff41c
Fix some grammar in MediaStreamError. r=jib
https://github.com/mozilla-firefox/firefox/commit/d40476b24272
https://hg.mozilla.org/integration/autoland/rev/f1f078b25343
Improve casting and fix some warnings in MediaEngineRemoteVideoSource. r=jib
https://github.com/mozilla-firefox/firefox/commit/5edd0a0a5700
https://hg.mozilla.org/integration/autoland/rev/e2aea0877661
Refactor DeviceListener::ApplyConstraints away from MediaManager::Dispatch. r=jib
https://github.com/mozilla-firefox/firefox/commit/2cf043e5d2d4
https://hg.mozilla.org/integration/autoland/rev/9c430aa474ad
Update crop-and-scale WPT to differentiate captures that happen to be native vs downscaling actually in effect. r=jib
https://github.com/mozilla-firefox/firefox/commit/f473b22f5527
https://hg.mozilla.org/integration/autoland/rev/71bd1c5d7772
Fix some includes. r=jib
https://github.com/mozilla-firefox/firefox/commit/7896e2db8684
https://hg.mozilla.org/integration/autoland/rev/eb2c55de9ce7
Move GetResizeMode to MediaConstraintsHelper and use Nothing() to denote when it's disabled by pref. r=jib
https://github.com/mozilla-firefox/firefox/commit/a05b1eebb503
https://hg.mozilla.org/integration/autoland/rev/285eedeaaebe
Add support for resizeMode to MediaEngineFake for platform-independent tests. r=jib
https://github.com/mozilla-firefox/firefox/commit/6817e397acf9
https://hg.mozilla.org/integration/autoland/rev/3839f25f9631
In MediaTrackConstraints use std::clamp. r=jib
https://github.com/mozilla-firefox/firefox/commit/1237d41407fa
https://hg.mozilla.org/integration/autoland/rev/b270990e890d
Break out StartCapture() from MERVS::Start() and move MERVS::Reconfigure to use it directly. r=jib
https://github.com/mozilla-firefox/firefox/commit/b865f6c73d75
https://hg.mozilla.org/integration/autoland/rev/8d48747e0414
Drop frames based on constraints in CamerasParent to avoid unnecessary IPC traffic and video frame copies. r=jib
https://github.com/mozilla-firefox/firefox/commit/b9be22d4f89c
https://hg.mozilla.org/integration/autoland/rev/a8a4e536ef6f
Simplify Construct() calls when assigning settings. r=jib
https://github.com/mozilla-firefox/firefox/commit/58ea59af3532
https://hg.mozilla.org/integration/autoland/rev/4179c7da3556
Remove main thread settings update from MERVS::Start. r=jib
https://github.com/mozilla-firefox/firefox/commit/07a32f0f0f87
https://hg.mozilla.org/integration/autoland/rev/d68ee861218e
Update comments and naming to clarify some MERVS members. r=jib
https://github.com/mozilla-firefox/firefox/commit/6cb92cc01b31
https://hg.mozilla.org/integration/autoland/rev/89f0b923ad07
Fix settings update on Reconfigure to use correct input size. r=jib
https://github.com/mozilla-firefox/firefox/commit/8f18c64cca9b
https://hg.mozilla.org/integration/autoland/rev/031e42166831
Update CalculateDesiredSize for resizeMode. r=jib
https://github.com/mozilla-firefox/firefox/commit/89d46bc103b1
https://hg.mozilla.org/integration/autoland/rev/96da62c1c969
Floor gDM framerate with resizeMode:none at 60 fps. r=jib
https://github.com/mozilla-firefox/firefox/commit/0fb81ee4c6a5
https://hg.mozilla.org/integration/autoland/rev/e9801354817f
Enable the resizeMode constraint and make the default value "crop-and-scale". r=jib
https://github.com/mozilla-firefox/firefox/commit/ffe3ca8f029c
https://hg.mozilla.org/integration/autoland/rev/15595e54cf98
Always use 2 for minimum size capability. r=jib
https://github.com/mozilla-firefox/firefox/commit/5929611bd51c
https://hg.mozilla.org/integration/autoland/rev/0d7ae4c7f831
Only overwrite MediaEnginePrefs::mResizeMode when the pref was read. r=jib

Comment 72

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/8649de9603d2
https://hg.mozilla.org/mozilla-central/rev/05a226d5261c
https://hg.mozilla.org/mozilla-central/rev/bb08dcd4ac87
https://hg.mozilla.org/mozilla-central/rev/6aaa9b6ff41c
https://hg.mozilla.org/mozilla-central/rev/f1f078b25343
https://hg.mozilla.org/mozilla-central/rev/e2aea0877661
https://hg.mozilla.org/mozilla-central/rev/9c430aa474ad
https://hg.mozilla.org/mozilla-central/rev/71bd1c5d7772
https://hg.mozilla.org/mozilla-central/rev/eb2c55de9ce7
https://hg.mozilla.org/mozilla-central/rev/285eedeaaebe
https://hg.mozilla.org/mozilla-central/rev/3839f25f9631
https://hg.mozilla.org/mozilla-central/rev/b270990e890d
https://hg.mozilla.org/mozilla-central/rev/8d48747e0414
https://hg.mozilla.org/mozilla-central/rev/a8a4e536ef6f
https://hg.mozilla.org/mozilla-central/rev/4179c7da3556
https://hg.mozilla.org/mozilla-central/rev/d68ee861218e
https://hg.mozilla.org/mozilla-central/rev/89f0b923ad07
https://hg.mozilla.org/mozilla-central/rev/031e42166831
https://hg.mozilla.org/mozilla-central/rev/96da62c1c969
https://hg.mozilla.org/mozilla-central/rev/e9801354817f
https://hg.mozilla.org/mozilla-central/rev/15595e54cf98
https://hg.mozilla.org/mozilla-central/rev/0d7ae4c7f831

Comment 73

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 74

[Donal Meehan [:dmeehan]]

:pehrsons, is there anything you want to call out in a release note? Asking since it's a long-running bug with many duplicates. (Process info)

EDIT: corrected typo

Comment 75

[Andreas Pehrson [:pehrsons]]

(In reply to Donal Meehan [:dmeehan] from comment #74)

:phesrosn

☺️

is there anything you want to call out in a release note? Asking since it's a long-running bug with many duplicates. (Process info)

Thanks for the ping. I think we can add something per below.

Release Note Request (optional, but appreciated)
[Why is this notable]: Implements the resizeMode constraint for getUserMedia, to allow applications to crop and downscale camera video to any resolution. The lack of this constraint broke several sites in the wild.
[Affects Firefox for Android]: Yes
[Suggested wording]: The resizeMode getUserMedia constraint is now available, allowing developers to crop and downscale video captured from a camera to any resolution they choose.
[Links (documentation, blog post, etc)]: nothing as of now

dev-doc-needed as some updates may be warranted to the relevant pages about getUserMedia and constraints.

Note what this does in a bit more verbose terms is:

• Adds the resizeMode constraint with two modes: "none" and "crop-and-scale". We default to "crop-and-scale".
• "crop-and-scale" allows cropping (camera-only) and downscaling to constraints. A native higher framerate than requested can also be decimated to match constraints.
• "none" prohibits cropping, downscaling and framerate decimation.
• For cameras, our previous behavior was on par with "none". "crop-and-scale" makes it so we will crop and/or downscale (upscale not permitted) to the resolution with the shortest fitness distance to the requested constraints.
• For screens and windows, our previous behavior was similar to "crop-and-scale" in that we downscaled (no cropping then, still not permitted now) to something close to the requested resolution. Spec compliance has improved here in that we find the correct resolution wrt fitness distance. resizeMode "none" is also new here and will avoid scaling entirely, sticking with the physical resolution of the display. Note no screenPixelRatio setting yet (bug 1965499), and no downscaling by that ratio by default in "crop-and-scale" yet either (bug 1703991).

Comment 76

[Donal Meehan [:dmeehan]]

Thanks, added to the Fx144 nightly release notes, please allow 30 minutes for the site to update.
Keeping the relnote-firefox flag as ? to keep it on the radar for inclusion in the final Fx144 release notes.

(In reply to Andreas Pehrson [:pehrsons] from comment #75)

(In reply to Donal Meehan [:dmeehan] from comment #74)

:phesrosn

☺️

Sorry for the multiple letter typo....

Comment 77

[Hamish Willee]

FF144 MDN docs work for this can be tracked in https://github.com/mdn/content/issues/41132

This talks only about getUserMedia(), but also refers to screens/windows. Would I be correct in thinking that for this case we're talking about using the constraint with getDisplayMedia()?

In addition, for the screen case, does the crop-and-scale also reduce the frame rate to match constraints, or only the resolution?

Last of all, I see that crop-and-scale is Firefox default. However I can't see it specified as a default in the spec. Is this specified or is it up to the implementation?

Comment 78

[Andreas Pehrson [:pehrsons]]

(In reply to Hamish Willee from comment #77)

FF144 MDN docs work for this can be tracked in https://github.com/mdn/content/issues/41132

This talks only about getUserMedia(), but also refers to screens/windows. Would I be correct in thinking that for this case we're talking about using the constraint with getDisplayMedia()?

Correct.

In addition, for the screen case, does the crop-and-scale also reduce the frame rate to match constraints, or only the resolution?

Also frame rate decimation.

Note the main difference between getUserMedia and getDisplayMedia wrt resizeMode: "crop-and-scale" is that cropping is not allowed for getDisplayMedia.

Last of all, I see that crop-and-scale is Firefox default. However I can't see it specified as a default in the spec. Is this specified or is it up to the implementation?

Calling it default is a bit misleading perhaps. The spec finds the most fitting (per fitness distance in the SelectSettings algorithm) setting given some constraints. If there's no resizeMode constraint provided, all settings dictionaries, i.e. involving both "none" and "crop-and-scale", are part of calculations. Also note that, by spec, for every settings dictionary with resizeMode: "none" there is an otherwise identical settings dictionary with resizeMode: "crop-and-scale", which we pick given all else being equal. I think the relevant bit in the spec is around here:

For any property with a system default value for the selected device, the system default value SHOULD be used if compatible with the above algorithm. This is usually the case for properties like sampleRate or sampleSize. Other properties, like echoCancellation or resizeMode do not usually have system default values. The User Agent defines its own default values for these properties. Implementors need to be cautious to select good default values since they will often have an impact on how media content is generated.

Comment 79

[Hamish Willee]

Thanks @Andreas

1. What I think you're saying is that if the resizeMode is not specified the browser is allowed to find the best fitness match for all the constraints, including "generated" constraints where none was specified and where crop-and-scale were specified, and if there are still several equally fit options based on those constraints and one of the choices is a system default then that should be preferred. Right?
2. For the "cropping is not allowed for getDisplayMedia." - is that a Firefox implementation thing or a spec requirement?

Comment 80

[Andreas Pehrson [:pehrsons]]

(In reply to Hamish Willee from comment #79)

Thanks @Andreas

1. What I think you're saying is that if the resizeMode is not specified the browser is allowed to find the best fitness match for all the constraints, including "generated" constraints where none was specified and where crop-and-scale were specified, and if there are still several equally fit options based on those constraints and one of the choices is a system default then that should be preferred. Right?

Right, but note in the case of resizeMode we're down to a user agent, rather than system, default, as per the spec cited in comment 78.

2. For the "cropping is not allowed for getDisplayMedia." - is that a Firefox implementation thing or a spec requirement?

That's a MUST NOT crop by spec: https://w3c.github.io/mediacapture-screen-share/#downscaling-and-frame-decimation

Comment 81

[Donal Meehan [:dmeehan]]

Added to the Fx144 release notes