1978789 - Adjust webrtc camera preview gUM call constraints and cropping

Status: ASSIGNED

(Firefox :: Site Permissions, task, P1)

Description

[Emma Zühlcke [:emz]]

jib:

Letterbox makes sense for screen capture, where losing parts of the picture can lead to the user not recognizing a surface, and is generally considered a bug by users.

But for camera, crop to fill is better, since the focal point, the part the user cares about (e.g. their face) is generally in the center. Here, black bars are sometimes considered a bug by users, as they lack appreciation for why they are there (they'll just sit up to include their chin instead).

I think my preference would be to remove the constraints and crop to fill the preview area to 16:9. Especially if we're concerned with performance in bug 1975984.

Of course there's also the option of using the constraints that the website passed in! That way the preview is more reflective of the result the user will get. I can't think of anything harmful from doing so off the top of my head.

https://phabricator.services.mozilla.com/D256261#8925008

Comment 1

[Emma Zühlcke [:emz]]

I've experimented with dropping the resolution constraints and cropping the video. When I remove resolution constraints my camera serves a vertical video (it supports both 16:9 and 9:16). We then crop in and even though the crop is in the center it makes the preview look really weird and zoomed in.

Ideally we would not pass a desired resolution but only a desired aspect ratio, but Firefox does not support this yet (Chrome and Safari do), see Bug 1306034.

Seems it's best to keep the constraints as-is to get the desired aspect ratio:

    let constraints = {
      video: {
        mediaSource: this.mediaSource,
        deviceId: { exact: this.deviceId },
        frameRate: 30,
        width: 1280,
        height: 720,
      },
    };

We could still apply object-fit: cover; to catch cases where the camera gives us a really weird, e.g. vertical, aspect ratio. Jib, what do you think?

Comment 2

[Emma Zühlcke [:emz]]

Attachment: WIP: Bug 1978789 - Apply object-fit: cover to webrtc camera preview. r=jib!

Comment 3

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

What camera and OS is this on? Does this mean you get a vertical video at https://jan-ivar.github.io/dummy/gum_video.html ?

I've never seen this before. I guess I'm more concerned that this seems a poor default resolution for Firefox to offer than whether the preview is alright. ;)

Bug 1286945 might help here, more than the aspectRatio constraint, which does not appear to be used that much (Firefox not implementing it might be to blame here).

Comment 4

[Emma Zühlcke [:emz]]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #3)

What camera and OS is this on? Does this mean you get a vertical video at https://jan-ivar.github.io/dummy/gum_video.html ?

I do! It's this webcam: https://www.insta360.com/product/insta360-link
It supports both vertical and horizontal video. Horizontal video needs to be enabled in the companion app explicitly at which point the camera supports both aspect ratios and Firefox seems to pick vertical for some reason, while e.g. Zoom picks the horizontal 16:9.