1194640 - Unable to get Camera resolution in 16:9 aspect ratio (640x360) on PC

Status: RESOLVED INVALID

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[marekzs]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.37 Safari/537.36

Steps to reproduce:

1. Please go here: http://jsfiddle.net/9t3szw0L/ in Nightly build of Firefox (v43) on a PC with a working available webcam

2. Click the Start! button

PC specs: Sony Vaio Laptop running Windows 8 and version 43.a1 (2015-08-12) Firefox Nightly


Actual results:

Local webcam video at VGA resolution is displayed


Expected results:

Instead of showing webcam video feed at 640x360 as defined in the constraints, we see VGA (4:3) video with the message, "Success: 640x480". See here: http://snag.gy/2TLPt.jpg

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

I don't see 640x360 - but then my camera (Logitech C615) doesn't have such a mode.  We only choose from the modes provided by the camera.

Please indicate the exact brand/model of camera you have (even if it's integrated), and run mozilla with:
set NSPR_LOG_MODULES=getusermedia:4
set NSPR_LOG_FILE=some random file
firefox.exe -profilemanager -no-remote 

(-profilemanager -no-remote is only needed if there's another Firefox already running with a different profile).

Most USB 16x9 camera only provide 16x9 modes above 640x480, but it does vary

Comment 2

[marekzs]

It's listed as a USB2.0 Camera even though it's integrated. It's manufactured by Microsoft and has driver version: 6.3.9600.17217.

I know the camera is capable of 16:9 resolution as Chrome WebRTC captures it at this resolution (640x360) fine.

I haven't had a chance to run mozilla with the configs you mention, but will try to later this afternoon.

Comment 3

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

marekzs, any update on this?

Comment 4

[marekzs]

Hi Jan - sorry for delay in getting back to this - I've just got back from vacation.

I have tried running Firefox the way you suggest, but end up getting a zero byte log file. If I run firefox using the below however, I get a huge file:

cd c:\
set NSPR_LOG_MODULES=timestamp,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5
set NSPR_LOG_FILE=%TEMP%\log.txt
cd "Program Files (x86)\Mozilla Firefox"
.\firefox.exe

Do you have any suggestions? Would adding getusermedia:4 to the second line get you what you're looking for do you think?

Comment 5

[Maire Reavy [:mreavy]]

Replace the other modules in NSPR_LOG_MODULES with getusermedia:4,mediamanager:4 and retry, thanks.  Also if this is running e10s (default in Nightly & Dev Edition/Aurora), the log file might be in log.txt-1

Comment 6

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Yes or log.txt.child-1 or something. The relevant part of the log is around here:

2018186000[113f1c070]: MediaManager: default prefs: 0x0 @30fps (min 10)
2018186000[113f1c070]: New Media thread for gum
668471296[11def37b0]:   Capture Device Index 0, Name FaceTime HD Camera (Built-in)
668471296[11def37b0]: Number of Capabilities -1
668471296[11def37b0]: Init
668471296[11def37b0]:  VoEHardware:GetRecordingDeviceName: Failed 9013
668471296[11def37b0]: Allocate
668471296[11def37b0]: ChooseCapability: prefs: 640x480 @30-10fps
668471296[11def37b0]: Constraints: width: { min: -2147483647, max: 2147483647, ideal: 640 }
668471296[11def37b0]:              height: { min: -2147483647, max: 2147483647, ideal: 360 }
668471296[11def37b0]:              frameRate: { min: -inf, max: inf }
668471296[11def37b0]: chose cap 640x480 @30fps codec 7 raw 0
668471296[11def37b0]: Video device 4097 allocated

Unfortunately, it wont tell us much more than we already know here. I'll work on getting you a version that has more logging enabled.

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: MozReview Request: Bug 1194640 - add NSPR logging of camera capabilities.

Bug 1194640 - add NSPR logging of camera capabilities.

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8656748 [details]
MozReview Request: Bug 1194640 - add NSPR logging of camera capabilities.

Bug 1194640 - add NSPR logging of camera capabilities.

Comment 9

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

marekzs, if you could try this build with extra logging that would be great. Thanks!
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-765dc1d393d2/

Comment 10

[marekzs]

Hi Jan,

Please find log below:

0[2311140]: mozilla::MediaManager::MediaManager: default prefs: 0x0 @30fps (min 10)
0[2311140]: New Media thread for gum
0[9b40b40]:   Capture Device Index 0, Name USB2.0 Camera
0[9b40b40]: Number of Capabilities 14
0[9b40b40]: type=2 width=640 height=480 maxFPS=30
0[9b40b40]: type=2 width=1280 height=1024 maxFPS=8
0[9b40b40]: type=2 width=1280 height=800 maxFPS=10
0[9b40b40]: type=2 width=1280 height=720 maxFPS=10
0[9b40b40]: type=2 width=320 height=240 maxFPS=30
0[9b40b40]: type=2 width=176 height=144 maxFPS=30
0[9b40b40]: type=2 width=160 height=120 maxFPS=30
0[9b40b40]: type=10 width=640 height=480 maxFPS=30
0[9b40b40]: type=10 width=1280 height=1024 maxFPS=30
0[9b40b40]: type=10 width=1280 height=800 maxFPS=30
0[9b40b40]: type=10 width=1280 height=720 maxFPS=30
0[9b40b40]: type=10 width=320 height=240 maxFPS=30
0[9b40b40]: type=10 width=176 height=144 maxFPS=30
0[9b40b40]: type=10 width=160 height=120 maxFPS=30
0[9b40b40]: void __thiscall mozilla::MediaEngineRemoteVideoSource::Init(void)
0[9b40b40]: enum nsresult __thiscall mozilla::MediaEngineRemoteVideoSource::Allocate(const struct mozilla::dom::MediaTrackConstraints &,const class mozilla::MediaEnginePrefs &,const class nsString &)
0[9b40b40]: ChooseCapability: prefs: 640x480 @30-10fps
0[9b40b40]: Constraints: width: { min: -2147483647, max: 2147483647, ideal: 640 }
0[9b40b40]:              height: { min: -2147483647, max: 2147483647, ideal: 360 }
0[9b40b40]:              frameRate: { min: -1.#INF00, max: 1.#INF00 }
0[9b40b40]: Capability:  640 x  480 x 30 maxFps, YUY2, Unknown codec. Distance = 250
0[9b40b40]: Capability: 1280 x 1024 x  8 maxFps, YUY2, Unknown codec. Distance = 1148
0[9b40b40]: Capability: 1280 x  800 x 10 maxFps, YUY2, Unknown codec. Distance = 1050
0[9b40b40]: Capability: 1280 x  720 x 10 maxFps, YUY2, Unknown codec. Distance = 1000
0[9b40b40]: Capability:  320 x  240 x 30 maxFps, YUY2, Unknown codec. Distance = 833
0[9b40b40]: Capability:  176 x  144 x 30 maxFps, YUY2, Unknown codec. Distance = 1325
0[9b40b40]: Capability:  160 x  120 x 30 maxFps, YUY2, Unknown codec. Distance = 1416
0[9b40b40]: Capability:  640 x  480 x 30 maxFps, MJPEG, Unknown codec. Distance = 250
0[9b40b40]: Capability: 1280 x 1024 x 30 maxFps, MJPEG, Unknown codec. Distance = 1148
0[9b40b40]: Capability: 1280 x  800 x 30 maxFps, MJPEG, Unknown codec. Distance = 1050
0[9b40b40]: Capability: 1280 x  720 x 30 maxFps, MJPEG, Unknown codec. Distance = 1000
0[9b40b40]: Capability:  320 x  240 x 30 maxFps, MJPEG, Unknown codec. Distance = 833
0[9b40b40]: Capability:  176 x  144 x 30 maxFps, MJPEG, Unknown codec. Distance = 1325
0[9b40b40]: Capability:  160 x  120 x 30 maxFps, MJPEG, Unknown codec. Distance = 1416
0[9b40b40]: chose cap:  640 x  480 x 30 maxFps, YUY2, Unknown codec. Distance = 250
0[9b40b40]: Video device 4097 allocated
0[9b40b40]: enum nsresult __thiscall mozilla::MediaEngineRemoteVideoSource::Start(class mozilla::SourceMediaStream *,int)
0[9b40b40]: started all sources
0[2311140]: mozilla::MediaManager::MediaCaptureWindowState: window 2147483665 capturing video     
0[2311140]: Returning success for getUserMedia()
3508[9b40de0]: MediaEngineRemoteVideoSource Video FrameSizeChange: 640x480

Comment 11

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8656748 [details]
MozReview Request: Bug 1194640 - add NSPR logging of camera capabilities.

https://reviewboard.mozilla.org/r/18239/#review16401

::: dom/media/webrtc/MediaEngineCameraVideoSource.cpp:199
(Diff revision 2)
> +  LOG(("%s: %4d x %4d x %2d maxFps, %s, %s. Distance = %lu",

width/height/maxFPS are unsigned int, not int
%u

::: dom/media/webrtc/MediaEngineCameraVideoSource.cpp:316
(Diff revision 2)
> -  LOG(("chose cap %dx%d @%dfps codec %d raw %d",
> +  LogCapability("chose cap", mCapability, sameDistance);

Maybe "Chosen Capability:"

Comment 12

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8656748 [details]
MozReview Request: Bug 1194640 - add NSPR logging of camera capabilities.

Bug 1194640 - add NSPR logging of camera capabilities.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to marekzs from comment 10)
> Please find log below:

Thanks, I see no 640x360 mode in that list, so there is no ideal algorithm bug here. I see no 16:9 modes at or below 640x480 which matches jesup's claim in comment 1.

marekzs, you said in comment 2 this worked in Chrome. Do me a favor and try https://jsfiddle.net/9t3szw0L/ in Chrome 45 (Chrome Canary).

When I try that on my Mac, Chrome does in fact give me 640x360, but after some visual inspection, it seems clear to me it is merely cropping the 640x480 output. If you could verify the same that would be great.

I think whether Chrome's behavior here is desirable or not, is debatable. I lean towards no, since it loses pixels. If cropping is all you want, that can be achieved in the video element at playback [1].

[1] http://stackoverflow.com/a/18716148/918910

Comment 14

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Cropping example: https://jsfiddle.net/jib1/ut4x94ou/

Comment 15

[marekzs]

Hi Jan,

Chrome Canary does crop the output, but it does not appear to be doing it using HTML/CSS: http://snag.gy/s36eo.jpg

(The success message also indicates it's been successful in achieving 480x360).

As you can see, the video element is actually given a 160x120 size i.e. 4:3. But the space allocated to the webcam view within that is 16:9.

Not sure if this is helpful, but I looked at the console logging and saw this: http://snag.gy/2J5Q5.jpg

Does this suggest adapter.js is doing something that works for Chrome but not Firefox? Somehow, Chrome Canary knows it should be being displayed at 480x360 and does so, but Firefox isn't able to detect the same. It would be great if Firefox behaved in the same way rather than having to do the crop.

I will speak to the team and see if handling different resolutions via cropping in Firefox the way you've done here (https://jsfiddle.net/jib1/ut4x94ou/) is good enough for us. Thanks

Comment 16

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to marekzs from comment #15)
> Chrome Canary does crop the output, but it does not appear to be doing it
> using HTML/CSS: http://snag.gy/s36eo.jpg

Of course, but it does crop the output, right? As in: The location of objects behind you in the frame suggest this is the same hardware output minus some pixel-lines, not a separate camera mode? (In my OSX test this was obvious, since its 1280x720 mode uses a noticeably wider angle, giving a completely different picture).

> (The success message also indicates it's been successful in achieving
> 480x360).

Yes, I understand that Chrome is processing the image for you. I'm questioning the value of it, versus only getting modes that are discretely native.

> Not sure if this is helpful, but I looked at the console logging and saw
> this: http://snag.gy/2J5Q5.jpg
> 
> Does this suggest adapter.js is doing something that works for Chrome but
> not Firefox?

No, the other way around. You're seeing adapter's constraints polyfill (which I wrote incidentally) which covers for Chrome not yet supporting standard constraints. See http://stackoverflow.com/a/28911694/918910

> Somehow, Chrome Canary knows it should be being displayed at
> 480x360 and does so, but Firefox isn't able to detect the same. It would be
> great if Firefox behaved in the same way rather than having to do the crop.
>
> I will speak to the team and see if handling different resolutions via
> cropping in Firefox the way you've done here
> (https://jsfiddle.net/jib1/ut4x94ou/) is good enough for us. Thanks

I would recommend cropping in both browsers, to avoid relying on native media sizes for layout. Cameras come in many different shapes and sizes, so expecting them to be the "same" seems like a mistake.

Comment 17

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7941ce03a4bf

Comment 18

[Nigel Babu [:nigelb]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7941ce03a4bf

Comment 19

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/7941ce03a4bf

Comment 20

[Maire Reavy [:mreavy]]

It sounds like this can be resolved.  jib?

Comment 21

[marekzs]

I spoke with my developers and whilst they would have preferred if things worked like in Chrome, they can probably make things work using HTML/CSS as advised above.

I'm not sure why the subsequent links above were posted repeatedly, but I don't think they changed anything with regard to the recommendation by Jan-Ivar.

So if this is still jib's suggestion, we'll go with it.

I will file a new bug if, for whatever reason, we cannot make things work nicely in our UI.

Thank you all!

Comment 22

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

marekzs, great to hear and thanks for the feedback! The commit links are for the logging I added in response to this, which we want to keep. They'll help us respond more quickly in the future.

I'm marking this bug invalid since we didn't end up changing any behavior as a result, only logging.