1573356 - Requesting max fps from a camera that also supports a higher fps can fail WebRTC constraints (and thus YouTube Live)

Status: NEW

(Core :: WebRTC: Audio/Video, enhancement, P2)

Description

[liquidblueocean]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36

Steps to reproduce:

1. Attach a webcam
2. https://webrtc.github.io/samples/src/content/peerconnection/constraints/
3. Drag all the sliders for the constraints to zero
4. Change the frameRate max to a value just below the max rate of the camera, but above its minimum
5. Press "Get media" (which calls getUserMedia), and Firefox will display the camera/privacy selector

Actual results:

• Camera won't be listed in the selector
• YouTube Live (where frameRate max is 30FPS) doesn't list cameras that can do <= 30FPS if they can also do above 30
• Google Chrome works fine for both websites

Expected results:

Camera supporting a frameRate <= max should be in the selector, regardless of whether it also supports frameRates above max

Comment 1

[liquidblueocean]

Bug submitted for Firefox on Windows

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 3

[Nico Grunbaum [:ng, @chew:mozilla.org]]

Andreas, would you mind taking a first pass at triaging this one?

Comment 4

[Andreas Pehrson [:pehrsons]]

I'd like to see the capabilities your camera reports to verify your claims as I am unable to reproduce this.

Could you launch firefox with the environment variables MOZ_LOG=timestamp,sync,MediaManager:4 and MOZ_LOG_FILE=/tmp/capabilities.log (assuming /tmp is your temp dir), then do these steps:

• Go to https://mozilla.github.io/webrtc-landing/gum_test.html
• Click on Video
• Pick the camera you're having troubles with in the permission prompt
• Approve the permission prompt

That will log its capabilities and more things to the MOZ_LOG_FILE. In fact there will be a bunch of files with various child-N suffixes in addition to the one you pointed MOZ_LOG_FILE to. Could you grab the one with actual data in it and attach it to this bug? Thanks!

Comment 5

[liquidblueocean]

I'll try what you've asked when I return to work tomorrow. It happens with all 3 cameras that I have, on Win and Mac. When using the WebRTC link I've provided, choose a frame rate max between the rates supported by your cam and it won't be listed in the selection/privacy dialog.

Comment 6

[liquidblueocean]

It doesn't happen on Linux, if that's what you tried

Comment 7

[Andreas Pehrson [:pehrsons]]

(In reply to liquidblueocean from comment #6)

It doesn't happen on Linux, if that's what you tried

That explains it, thanks!

The only thing different between platforms here is the platform-dependent capture code, and the camera drivers. It could be that the set of capabilities returned on linux are different from the ones returned on mac and windows. Confirming that would be useful.

Comment 8

[liquidblueocean]

Attachment: capabilitiesGumTest.log

I have found a cam that works and that happened to be attached to my Linux box. I tried the same cam on my Mac and the bug appears there. I tried another cam on my Linux box and it fails. So, the issue exists on all 3 platforms, and it depends on which cam is attached.

Here is the requested GumTest log for Linux using a failing cam.

Comment 9

[liquidblueocean]

Attachment: capabilitiesMax28FailMax33Pass.log

Here is another log where I'm on Linux firefox 68.0.1, using the below link.
https://webrtc.github.io/samples/src/content/peerconnection/constraints/

I've run two tests:
frameRate max = 28 (FAILS)
frameRate max = 33 (PASS)

Note that the cam can do p30, so it should have passed for both.

I can see in the log the following, if it's relevant:
"GetUserMedia: bad constraint found in post enumeration promise2 success callback! Calling error handler!"

Comment 10

[Andreas Pehrson [:pehrsons]]

So for the camera that's failing on linux it reports these capabilities:

2019-08-15 00:53:26.641653 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  640 x  480 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641666 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  800 x  600 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641670 UTC - [Child 4873: MediaManager]: D/MediaManager Capability: 1280 x  720 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641673 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  320 x  240 x 30 maxFps, VP9. Distance = 0
2019-08-15 00:53:26.641675 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  640 x  480 x 30 maxFps, VP9. Distance = 0
2019-08-15 00:53:26.641677 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  320 x  240 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641680 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  640 x  480 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641682 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  320 x  240 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641684 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  640 x  480 x 30 maxFps, Unknown codec. Distance = 0

maxFps is a bit of a misnomer here because if picked that's the value we end up requesting, and we don't have frame dropping logic. Also VP9 is plain wrong.

That aside, if you ask for an fps lower than 30 for this camera (say {max: 29}), there is no capability satisfying that constraint since they'd all give us 30 fps back, and so such a request would fail.

What makes this worrying is that a large site like Youtube is constraining the fps so strictly that it'll fail for some cameras. OTOH there's no other option if they want to be guaranteed a framerate below 30.

I'll mark this as a P2 because it affects Youtube Live, and enhancement because in our code it'll be new functionality added.

Thanks again for filing!

Comment 11

[Andreas Pehrson [:pehrsons]]

Though you said Youtube Live requests max: 30 and that ended up blank. What camera was that, and would you be able to get the logs on Mac where it fails with max: 30? I wonder what capabilities it reports. Not reporting any capability with 30fps seems odd.

Comment 12

[Andreas Pehrson [:pehrsons]]

In your second log I don't see capabilities because when it fails we don't log them, and the succeeded case seems truncated.

So to clarify I'd like to see a log of the steps in comment 4 on Mac with the camera that would fail on youtube live on the same machine.

Comment 13

[Andreas Pehrson [:pehrsons]]

I'll make the title a bit more descriptive for us, so far assuming the camera you have reports only 60fps capabilities and youtube live requesting max 30.

Comment 14

[liquidblueocean]

I've restored the title because the bug can happen if you have frameRate max set to p28 and the highest rate the cam supports is p30. However, it depends upon which cam is paired to which platform. Eg. I have a Logitech V-U0009 where its max FPS is 30 and the bug happens for it on Mac but not on Linux, both with firefox 68.0.1. It's quite possible there's more than one bug with the same symptoms.

Re YouTube Live, they're restricting cams to p30 or less. Live streaming above p30 is far harder than file uploading. Note that when using Chrome on any OS, all cams I have will be listed for YouTube Live because they all support p30 or less, disregarding whether they can do higher.

I'll get around to the info request soon.

Comment 15

[liquidblueocean]

Attachment: lsusbWebPresenter.txt

Attaching lsusb (USB Descriptor) of Blackmagic Design Web Presenter I've tested in previous logs. This is the direct way to know what the device supports. Listed dwFrameInterval vals can be translated to FPS by doing 10000000/dwFrameInterval.

More to come from me yet...

Comment 16

[liquidblueocean]

(In reply to Andreas Pehrson [:pehrsons] from comment #10)

So for the camera that's failing on linux it reports these capabilities:

2019-08-15 00:53:26.641653 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  640 x  480 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641666 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  800 x  600 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641670 UTC - [Child 4873: MediaManager]: D/MediaManager Capability: 1280 x  720 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641673 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  320 x  240 x 30 maxFps, VP9. Distance = 0
2019-08-15 00:53:26.641675 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  640 x  480 x 30 maxFps, VP9. Distance = 0
2019-08-15 00:53:26.641677 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  320 x  240 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641680 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  640 x  480 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641682 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  320 x  240 x 30 maxFps, Unknown codec. Distance = 0
2019-08-15 00:53:26.641684 UTC - [Child 4873: MediaManager]: D/MediaManager Capability:  640 x  480 x 30 maxFps, Unknown codec. Distance = 0

maxFps is a bit of a misnomer here because if picked that's the value we end up requesting, and we don't have frame dropping logic. Also VP9 is plain wrong.

That aside, if you ask for an fps lower than 30 for this camera (say {max: 29}), there is no capability satisfying that constraint since they'd all give us 30 fps back, and so such a request would fail.

That camera is the Blackmagic Web Presenter, which supports 5FPS-30FPS. It can do those rates outside of firefox fine (as also show in the USB descriptor I attached), not just 30FPS

Comment 17

[liquidblueocean]

What I mean is, there is a capability that supports a requested max of 29, there are many rates that cam supports below 29.

Comment 18

[liquidblueocean]

Attachment: capabilitiesMac25Fail30PassLogitech.log

As requested, attached is another example. This cam passes in firefox on Linux and Chrome on Mac but fails in firefox on Mac. Unfortunately I don't have the cam in the office today but I can do the test you're asking for with the Logitech V-U0009 I have today this way:

The Logitech V-U0009 supports FPS of 5,10,15,20,25,30. If I use the following website to set frameRate max to 25 then firefox on Mac won't list the cam, despite the cam supporting 25FPS! Firefox on Linux does for this cam. It won't even list the built-in FaceTime cam, which can do 15/24/25/29.97 FPS. Both are listed when doing the same test in Chrome.
https://webrtc.github.io/samples/src/content/peerconnection/constraints/

Attached is the log from doing that test.

Comment 19

[liquidblueocean]

Attachment: lsusbLogitech.txt

Here is the USB descriptor of the Logitech Logitech V-U0009, to show what it supports

Comment 20

[Andreas Pehrson [:pehrsons]]

(In reply to liquidblueocean from comment #18)

Created attachment 9085944 [details]
capabilitiesMac25Fail30PassLogitech.log

As requested, attached is another example. This cam passes in firefox on Linux and Chrome on Mac but fails in firefox on Mac. Unfortunately I don't have the cam in the office today but I can do the test you're asking for with the Logitech V-U0009 I have today this way:

The Logitech V-U0009 supports FPS of 5,10,15,20,25,30. If I use the following website to set frameRate max to 25 then firefox on Mac won't list the cam, despite the cam supporting 25FPS! Firefox on Linux does for this cam. It won't even list the built-in FaceTime cam, which can do 15/24/25/29.97 FPS. Both are listed when doing the same test in Chrome.
https://webrtc.github.io/samples/src/content/peerconnection/constraints/

Attached is the log from doing that test.

Did you follow the steps in comment 4 for this log? I don't see the camera actually starting, which is what I need to see what capabilities were reported to us.

With that I can understand if there's some additional bug with the same symptoms here, or if my assumptions stand.

Note that our camera backends are platform dependent (which drivers also are) so it's totally plausible that one camera works on linux but fails on mac because they report different capabilities to us. We'll fix it eventually with the aforementioned frame-dropping logic, but if there's an additional bug in play here we could perhaps get to a smaller fix for that sooner, so I'd like to verify this.

Comment 21

[liquidblueocean]

Attachment: capabilitiesGumTestLogitech.log

Attaching log of GumTest on Mac using aforementioned Logitech. This can be contrasted with the already supplied lsusb that has the true list of camera abilities.

My other logs that don't have GumTest in the title follow the process of Comment 4 but with the WebRTC site.

Comment 22

[Andreas Pehrson [:pehrsons]]

Thanks! That confirms that our backend reports 30 fps for all capabilities from this camera on mac.