Open Bug 1166664 Opened 9 years ago Updated 2 years ago

WebRTC always selects MJPEG when resolution is higher than VGA

Categories

(Core :: WebRTC: Audio/Video, defect, P4)

Product:
Core
Component:
WebRTC: Audio/Video
Version:
38 Branch
Type:
defect

Tracking

()

Blocking Flags:
backlog webrtc/webaudio+

People

(Reporter: pastas4, Unassigned)

References

Details

Attachments

(3 files)

about:support raw data
3.11 KB, text/plain
Details
v4l2-ctl output
5.30 KB, text/plain
Details
v4l2-ctl --list-formats-ext output
7.94 KB, text/plain
Details
Attached file about:support raw data — 
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:37.0) Gecko/20100101 Firefox/37.0
Build ID: 2015041600

Steps to reproduce:

1. Set the Firefox options media.navigator.video.default_width and media.navigator.video.default_height above VGA (640x480), for instance to 1280x720
2. Set the v4l2 options to the desired format, for instance, YUYV 1280x720p10 (adjust media.navigator.video.default_fps if needed)
3. Testing the webcam in http://appear.in
4. Check the v4l2 settings being set


Actual results:

The v4l2 pixel format that gets selected is MJPEG, and the desired framerate is ignored altogether: the result is MJPG 1280x720p15.


Expected results:

The pixel format should have stayed as YUYV 1280x720p10.

I'm using CentOS Linux 7.1, Firefox ESR 38.0, the webcam is a Microsoft LifeCam Studio (UVC). The about:support info is attached.
Attached file v4l2-ctl output — 
The information from v4l2-ctl about the webcam is attached. The first block is the settings I applied in step 2, the second block is the settings that Firefox applied (step 4). The last block is the output of v4l2-ctl --list-formats; as you can see, the webcam supports YUYV, MJPEG and M420. The supported frame rates depend on the resolution, but 1280x720p10 is something that is supported in all the listed formats.

The issue is not tied to that specific frame size and frame rate combination; same thing happens on 800x600 or 1920x1080. However, 640x480 or below (like 320x240) select the YUYV settings correctly.
Please retest with Beta (39) - there's a fix that may well help (and will be in 38.0.5 IIRC)
Flags: needinfo?(pastas4)
Yeah Bug 1150539 (workaround is to also set media.navigator.video.default_minfps)

Also, can you try with constraints? http://jsfiddle.net/3hk82d31
Tested with the Beta channel, and this issue still remains. (On the other hand, bug #1151628 was fixed in Beta, so at least I can actually see something with the MJPEG modes now :) )

I'll try the other two suggestions now.
And neither of the other two suggestions change anything either. Setting minfps = fps = 10, as well as using the constraints form as per above, both still result in Firefox picking MJPEG 1280x720p15 (instead of anything else with 1280x720p10).
Flags: needinfo?(pastas4)
Another thing I noticed while testing is that some of the more unusual video sizes, like 960x544, are ignored completely when requested (in that case I get MJPEG 800x600 instead), even though my webcam fully supports them.

Also, thinking about it, the needed format for WebRTC is YUV 4:2:0, right? In which case, it's that what M420 in my webcam is? Maybe that's what is supposed to be preferred even over YUYV, then. And maybe that's what causes Firefox to be confused?

I attached the extended list of formats my webcam supports, for reference.
The encoders use I420 = 8 bit Y plane followed by 8 bit 2x2 subsampled U and V planes.

M420 seems to have the U and V channels interleaved.
Also, YUYV is only subsampled in one direction, so it's a bit off of what we want too. I guess either that or M420 would be preferable for the conversion to I420 than MJPEG, though.

I'm guessing our code that figures out the optimal format just doesn't know about those being preferable?
But then why do VGA and lower frame sizes prefer YUYV instead? And why does it ignore the desired frame rate?
Well, if the bug is that it's ignoring the framerate, then MJPEG is the only option that gives >=600p at 30fps, so one kinda follows from the other here.
Yes, but the framerate requested (in the settings and constraints) was 10 FPS, not 30 FPS.
Yes, I heard you. What I'm saying is that if the bug is that the framerate settings are being ignored, the selection of the MJPEG codec *necessarily follows from that*, because at 30fps it's the only one to meet the other constraints. i.e. the wrong codec being selected might be an effect of the bug, not the bug itself.

Probably we'll want to make a patch with some logging in the camera selection code (if there isn't a bunch of it there already), have you turn on the relevant logging and see what's up.
minfps is for rejecting items that don't match that - it's not a preferred fps.

In 38 there *is* a bug where it uses minfps as the preferred fps; this will be fixed in 38.0.5 and is fixed in 39/40/41 IIRC.

As for modes: We make a call to ConvertToI420(), which handles quite a lot of input modes.  However, I don't think M420 is among them; it appears we support I420, YV12, YUY2, UYVY, ARGB, RGB24, RGB565, ARGB4444, ARGB1555, MJPEG, NV12, NV21, and BGRA.  It's possible that Windows is slightly more constricted, I haven't walked through all the paths yet.
Also: I think the code may take the first match; jib?  Was there a preference for non-MJPEG modes, and if so did it survive the constraints rewrite?
(In reply to Gian-Carlo Pascutto [:gcp] from comment #12)
> Yes, I heard you. What I'm saying is that if the bug is that the framerate
> settings are being ignored, the selection of the MJPEG codec *necessarily
> follows from that*, because at 30fps it's the only one to meet the other
> constraints. i.e. the wrong codec being selected might be an effect of the
> bug, not the bug itself.

Ah, I see; however, it sets the FPS to 15, not 30. But yes, 15 is only supported by MJPEG and M420, not YUYV, so the same idea applies.
(In reply to Randell Jesup [:jesup] from comment #13)
> minfps is for rejecting items that don't match that - it's not a preferred
> fps.

Yes, I get that, but I set both media.navigator.video.default_fps and media.navigator.video.default_minfps to 10, so 10 should have been the preferred one in any case.

> As for modes: We make a call to ConvertToI420(), which handles quite a lot
> of input modes.  However, I don't think M420 is among them; it appears we
> support I420, YV12, YUY2, UYVY, ARGB, RGB24, RGB565, ARGB4444, ARGB1555,
> MJPEG, NV12, NV21, and BGRA.  It's possible that Windows is slightly more
> constricted, I haven't walked through all the paths yet.

I take it that YUYV is linked to YUY2?
(In reply to Gian-Carlo Pascutto [:gcp] from comment #16)
> We still seem to have them after the rewrite:
> https://dxr.mozilla.org/mozilla-central/source/dom/media/webrtc/
> MediaEngineCameraVideoSource.cpp?from=ChooseCapability&case=true#301

Yes. All else being equal (width, height, framerate), the last step is to prefer kVideoI420 or kVideoYUY2 or kVideoYV12 over other modes of equal ability.

This sounds like a bug in the ideal algorithm wrt frameRate. Please try this: http://jsfiddle.net/yf93un4L
(In reply to Jan-Ivar Bruaroey [:jib] from comment #18)
> This sounds like a bug in the ideal algorithm wrt frameRate. Please try
> this: http://jsfiddle.net/yf93un4L

Looks like there's something wrong with that script, as I get the error "NotFoundError: The object can not be found here.: line 0".
I think the script is working. NotFoundError means gUM could not find that resolution, which is helpful because it means I doubt there's anything wrong with the ideal algorithm, rather the information about this specific mode seems absent in the capability list coming from the device layer this for this camera on linux. Jesup, who do we think would know more about this?
Flags: needinfo?(rjesup)
...could not find that resolution and frame-rate, I should say.
My camera has 1280x720x30 (MJPEG) and x15 (YUV - type 0 and type 2).  
{ width: { exact: 1280 }, height: { exact: 720 }, frameRate: { exact: 10 } } fails (perhaps expected) - but 15, 30 and removing framerate also fail, which seems like a serious bug in exact.
Flags: needinfo?(rjesup)
Oh, I realised that I need to press "run" before the code changes take place. Now things are clearer. Here's what I got from testing various options:

* When above VGA resolution, having "frameRate: { exact: * }" results in the aforementioned error. The selected frame rate doesn't matter.
* When above VGA resolution, and the frameRate is not exact, it works at 7.5 FPS, no matter what frameRate is set to (and width/height being exact works fine).
* When at or below VGA, both exact and inexact frameRate works, but again no matter the setting I get 7.5 FPS.
* When using an unusual resolution like 960x544, I get the error if width and height are exact, otherwise I get the closest usual resolution (800x600p7.5).

So it looks like a frame rate setting issue to me.
Jan-Ivar -- Can you confirm that this is a bug?  I've made this a P3 for now.
backlog: --- → webRTC+
Rank: 35
Flags: needinfo?(jib)
Priority: -- → P3
comment 22 sounds like a bug. Randell, can you still reproduce this? Camera model #? STR?
Flags: needinfo?(jib) → needinfo?(rjesup)
comment 23 sounds like a bug as well.
(In reply to Jan-Ivar Bruaroey [:jib] from comment #25)
> comment 22 sounds like a bug. Randell, can you still reproduce this? Camera
> model #? STR?

I also had not realized I needed to hit 'run' to make the changes take effect... it should highlight Run when there are changes waiting to be run.  So it now works at 15 & 30; my camera doesn't have a 10fps mode for that resolution.

Dainius - note that cameras normally choose their fps rate based on light.  If you're in a dark (or even just artificially-lit) room, the camera may prefer 7.5 fps to keep frame quality and brightness up.  Try pointing it at a bright light or shine a flashlight in it, or a brightly-lit piece of white paper right in front of it.
Flags: needinfo?(rjesup) → needinfo?(pastas4)
Nope, that's not it. In fact, I just tried it with another camera (an HP laptop built-in one) that at 1280x720 has only 10 FPS in YUYV and 30 FPS in MJPEG, and at "exact: 10" it gave me an error, while at non-exact 10 it gave me MJPEG 1280x720p30. Again QV4L2 has no problems recording at either of the two settings.
Flags: needinfo?(pastas4)
Status: UNCONFIRMED → NEW
Ever confirmed: true
Mass change P3->P4 to align with new Mozilla triage process.
Priority: P3 → P4
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: