1151628 - MJPEG getUserMedia sources don't work

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P1)

Description

[Randell Jesup [:jesup] (needinfo me)]

Attachment: mjpeg

Cameras which return MJPEG data for the resolution we select in getUserMedia fail, since libyuv doesn't have MJPEG support compiled in.  It was accidentally disabled when bug 880419 landed (which moved libyuv to be a standalone library in media), probably because of a temporary change to test that got rolled into some updates for android.

Patch simply reinstates the pre-move state of HAVE_JPEG and resolves an compile issue on some newer platforms.

This worked in 29 and before (modulo MJPEG cameras don't generally work with the libjpeg_turbo we had before 29), and is broken in 30 and later.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=67db53f2b3eb
All failures appear to be infra or unrelated to this change

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/aef7f6bf3470

May be worth relnoting

Request uplift as soon as this merges

Comment 3

[Simon Eisenmann]

GetUserMedia now works fine with MJPEG camera. Though i see lots of errors on the console.

Corrupt JPEG data: 2 extraneous bytes before marker 0xd4
Corrupt JPEG data: 2 extraneous bytes before marker 0xd4
Corrupt JPEG data: 9 extraneous bytes before marker 0xd2
Corrupt JPEG data: 5 extraneous bytes before marker 0xd3
Corrupt JPEG data: 1 extraneous bytes before marker 0xd2
Corrupt JPEG data: 6 extraneous bytes before marker 0xd0
Corrupt JPEG data: 3 extraneous bytes before marker 0xd3

This prints constantly while the video is running. Video looks ok to my eyes though.

Comment 4

[Simon Eisenmann]

(In reply to Simon Eisenmann from comment #3)
> Corrupt JPEG data: 2 extraneous bytes before marker 0xd4

Forgot to mention that constraints are set to capture 1920x1080 from a Logitech C910 USB web cam.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/aef7f6bf3470

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8588767 [details] [diff] [review]
mjpeg

We'll also want bug 1153849, which turns off jpeg-turbo's fprintf's to stderr

Approval Request Comment
[Feature/regressing bug #]: 880149

[User impact if declined]: Certain laptops and many USB cameras (at least on Linux, probably on Windows, maybe on Mac) when running higher resolutions (HD) will not capture video.

[Describe test coverage new/current, TreeHerder]: Requires manual testing since real cameras are needed.

[Risks and why]: Very low risk.  This was fixed before, and build-system changes when moving it to a standalone directory (media/libyuv) accidentally turned this off.

[String/UUID change made/needed]: none

Comment 7

[Liz Henry (:lizzard) (relman/hg->git project)]

OK, let's uplift this once bug 1153849 lands.

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

The actual other bug for uplift is Bug 1152016, sorry (copy-pasted wrong one)

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

Florin can your team test this fix in Firefox 40? Now that I look at it again I think we should confirm this does what we think it does, even if it's turning a function we used to have back on. 

Randell would this make sense? Do you have good steps to reproduce the bug? Is this only for specific hardware? Thanks.

Comment 10

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Randell Jesup [:jesup] from comment #6) 
> Approval Request Comment
> [Feature/regressing bug #]: 880149

Bug 880149 was resolved wfm. Do you know when this issue was introduced?

Comment 11

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Lawrence Mandel [:lmandel] (use needinfo) from comment #10)
> (In reply to Randell Jesup [:jesup] from comment #6) 
> > Approval Request Comment
> > [Feature/regressing bug #]: 880149
> 
> Bug 880149 was resolved wfm. Do you know when this issue was introduced?

Bug 880419, sorry.  Landed in Fx 30

Comment 12

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Randell, how can we know if the cameras we have can be used (or not) to verify this? And do you have any steps to verify this?

Comment 13

[Randell Jesup [:jesup] (needinfo me)]

That's a bit tricky to verify.

Turn on getusermedia:5,mediamanager:5 logging.  use https://mozilla.github.io/webrtc-landing/gum_test.html, select "Video".  Then after video starts and works, select Stop.

In the log, you'll see all the resolutions and types supported by the camera.  Type 10 is MJPEG, types 0 & 2 are forms of raw YUV.  At the default 640x480, almost all cameras will have a YUV mode selected.  You can see the mode selected is in the line that starts "chose cap " - the type is what follows "raw".

To test MJPEG, set (in about:config) media.navigator.video.default_width to 1280 and media.navigator.video.default_height to 720 (or choose other HD-ish values), the select Video again, and verify it works, then Stop, and check the log again.  No need to restart after changing the pref values.  If it says "raw 10", then it's MJPEG.

Comment 14

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Thanks Randell! With the latest Firefox 40 Nightly, for my camera on Windows 7 x64 I got: "chose cap 960x720 @15fps codec 7 raw 0" (with the 1280x720 settings), so it seems I cannot use it to verify. I'll try to see what other cameras we have available here.

Comment 15

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8588767 [details] [diff] [review]
mjpeg

[Triage Comment]
OK, I am going to take it for the next beta (7). I know it has not been verified yet but I prefer get the beta user coverage + check for potential regressions right now instead of waiting a few days and have to take in beta 8 or 9.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/e287bac06fb6

Comment 17

[Ada [:adalucinet]]

With a Logitech HD Pro Webcam C920 on latest 40.0a1 (2015-04-22), under Windows 7 x64 , I also got raw 0: "chose cap 1280x720 @30fps codec 7 raw 0" (with the 1280x720 settings).

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/f6448c4cf87f

Comment 19

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

It looks like we might not have the proper hardware to verify this. Will try some more webcams tomorrow.

Comment 20

[Randell Jesup [:jesup] (needinfo me)]

Most webcams (and W540's, etc) will trigger this.  My Logitech 615 does.  Most cheapo webcams can't do raw above 640x480-ish.  You can also try 1920x1080

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/f6448c4cf87f

Comment 22

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

As far as I can tell the Logitech HD Pro Webcam C920 should support MJPEG. I tried one myself today but always got "raw 0" even with a 1920x1080 @30fps setting. I'm not sure whether something else needs to be setup or installed in order to get MJPEG to work, but right now it doesn't seem to work for us.

Please provide more info if you have any ideas on why we cannot get this to work, or perhaps try to get confirmation of the fix from someone who knows should be getting MJPEG.

Comment 23

[Simon Eisenmann]

Maybe it too depends on the OS (which provides the camera driver). I get MJPEG for the Logitech HD Pro C910 on Linux and for Chromebook HD WebCam on Linux.

Comment 24

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to Simon Eisenmann from comment #23)
> Maybe it too depends on the OS (which provides the camera driver). I get
> MJPEG for the Logitech HD Pro C910 on Linux and for Chromebook HD WebCam on
> Linux.

That may be a possibility. Simon, would you mind confirming this fix using the latest Firefox 38 Beta build at: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/38.0b8-candidates/build1/? I see that you've already confirmed the fix when it landed on inbound, but got some errors.

Comment 25

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #22)
> I tried one myself today but always got "raw 0" even with a 1920x1080 @30fps
> setting. I'm not sure whether something else needs to be setup or installed
> in order to get MJPEG to work, but right now it doesn't seem to work for us.
> 
> Please provide more info if you have any ideas on why we cannot get this to
> work, or perhaps try to get confirmation of the fix from someone who knows
> should be getting MJPEG.

The current code [1] prefers I420, YUY2, YV12 when the same resolution and frame rate are available, hence would not pick MJPEG unless other formats were unavailable at that the same rate and resolution. There's no codec constraint unfortunately.

[1] http://mxr.mozilla.org/mozilla-central/source/dom/media/webrtc/MediaEngineCameraVideoSource.cpp?rev=ce72894838de#303

Comment 26

[Simon Eisenmann]

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #24)
> That may be a possibility. Simon, would you mind confirming this fix using
> the latest Firefox 38 Beta build at:
> ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/38.0b8-candidates/
> build1/? I see that you've already confirmed the fix when it landed on
> inbound, but got some errors.

The confirm from above should still be valid. The errors i got are handled in #1152016
I test again with the version you posted asap.

Comment 27

[Simon Eisenmann]

All right i confirm that i can capture 1080p with my Logitech HD Webcam 910 on Linux with nightly/38.0b8-candidates/ as suggested in comment #24. In Firefox 37 this gives just a black screen. 

Furthermore it seems like when setting no frame rate constraint, it seems to choose I420 which gives @15fps.
FF38 1080p no fps -> ok
-11757824[7f1fd95d2500]:  VoEHardware:GetRecordingDeviceName: Failed 9013
-11757824[7f1fd95d2500]: Audio device 0 allocated
-11757824[7f1fd95d2500]: Allocate
-11757824[7f1fd95d2500]: ChooseCapability: prefs: 0x0 @30-10fps
-11757824[7f1fd95d2500]: chose cap 1920x1080 @15fps codec 7 raw 0
-11757824[7f1fd95d2500]: Video device 4097 allocated
-11757824[7f1fd95d2500]: Audio config: aec: 1, agc: -1, noise: 1
-11757824[7f1fd95d2500]: Start audio for stream 7f1fd89ad4a0
-11757824[7f1fd95d2500]: Audio config: aec: 0, agc: -1, noise: 0
-11757824[7f1fd95d2500]: Start
-11757824[7f1fd95d2500]: started all sources
438888256[7f2018c6f380]: Returning success for getUserMedia()
-1024493824[7f1fe4f4bd20]: Video FrameSizeChange: 1920x1080
.. looks good ..

FF37 1080p no fps -> not ok video black
1577887488[7f5c2d0b6740]:  VoEHardware:GetRecordingDeviceName: Failed 9013
1577887488[7f5c2d0b6740]: Audio device 0 allocated
1577887488[7f5c2d0b6740]: Allocate
1577887488[7f5c2d0b6740]: ChooseCapability: prefs: 0x0 @30-10fps
1577887488[7f5c2d0b6740]: chose cap 1280x720 @15fps codec 6 raw 2
1577887488[7f5c2d0b6740]: Video device 4097 allocated
1577887488[7f5c2d0b6740]: Audio config: aec: 1, agc: -1, noise: 1
1577887488[7f5c2d0b6740]: Start audio for stream 7f5c29969400
1577887488[7f5c2d0b6740]: Audio config: aec: 0, agc: -1, noise: 0
1577887488[7f5c2d0b6740]: Start
1577887488[7f5c2d0b6740]: started all sources
2016212800[7f5c76c48460]: Returning success for getUserMedia()
.. nothing more..

When setting the constraint so, that the camera can only return MJPEG (for me 1080p at 30fps) then MJPEG is used and does work fine in FF38.

FF38 1080p 30fps -> ok
-11757824[7f1fd95d2500]:  VoEHardware:GetRecordingDeviceName: Failed 9013
-11757824[7f1fd95d2500]: Audio device 0 allocated
-11757824[7f1fd95d2500]: Allocate
-11757824[7f1fd95d2500]: ChooseCapability: prefs: 0x0 @30-10fps
-11757824[7f1fd95d2500]: chose cap 1920x1080 @30fps codec 7 raw 10
-11757824[7f1fd95d2500]: Video device 4097 allocated
-11757824[7f1fd95d2500]: Audio config: aec: 1, agc: -1, noise: 1
-11757824[7f1fd95d2500]: Start audio for stream 7f1ffc307b50
-11757824[7f1fd95d2500]: Audio config: aec: 0, agc: -1, noise: 0
-11757824[7f1fd95d2500]: Start
-11757824[7f1fd95d2500]: started all sources
438888256[7f2018c6f380]: Returning success for getUserMedia()
-627005696[7f1fe4f4b8a0]: Video FrameSizeChange: 1920x1080
.. looks good ..

So i guess the trick to get MJPEG with the C920 is also to set the FPS so high, that only MJPEG mode is supported.

To sum this up, i confirm that the fix works fine and MJPEG mode works again in Firefox 38.

Comment 28

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Thank you guys for the additional info and testing. I'm marking this as Verified.