1150539 - getUserMedia activating camera light, but not receiving video -- audio working OK.

Status: VERIFIED FIXED

(Core :: WebRTC: Audio/Video, defect)

Description

[Bob Owen (:bobowen)]

Attachment: Hello Problem Browser Console Dump

Using my Hello conversation, other caller connected OK and could hear me, but not see me.
The indicator light on my camera was on, I had no self view.
Incoming video and audio was fine.

Latest Windows x86 Nightly built from:
https://hg.mozilla.org/mozilla-central/rev/37ddc5e2eb72

Browser console log attached.

Comment 1

[Adam Roach [:abr]]

Two quick questions:
1) Does this happen every time, or intermittently?
2) To help isolate whether this is a Hello issue or a platform issue, can you see whether you can reproduce it on <http://mozilla.github.io/webrtc-landing/gum_test.html> ?

Comment 2

[Bob Owen (:bobowen)]

(In reply to Adam Roach [:abr] from comment #1)
> Two quick questions:
> 1) Does this happen every time, or intermittently?

Every time.

> 2) To help isolate whether this is a Hello issue or a platform issue, can
> you see whether you can reproduce it on
> <http://mozilla.github.io/webrtc-landing/gum_test.html> ?

When I click the "Video" button, I get asked to share the camera and when allowed it says "Success!", but I see no video.
The camera indicator is on.

Camera is working with Vidyo.

Comment 3

[Adam Roach [:abr]]

Bob -- can you provide as much information as possible about your camera? Manufacturer, model, driver name, USB ID (cf. https://msdn.microsoft.com/en-us/library/windows/hardware/ff560019%28v=vs.85%29.aspx and http://digital.ni.com/public.nsf/allkb/335A90747734097886257070006415B9) -- actually, including all of the information from USBView would be helpful.

Comment 4

[Bob Owen (:bobowen)]

(In reply to Adam Roach [:abr] from comment #3)
> Bob -- can you provide as much information as possible about your camera?
> Manufacturer, model, driver name, USB ID (cf.
> https://msdn.microsoft.com/en-us/library/windows/hardware/ff560019%28v=vs.
> 85%29.aspx and
> http://digital.ni.com/public.nsf/allkb/335A90747734097886257070006415B9) --
> actually, including all of the information from USBView would be helpful.

Yeah, no problem, it might have to wait until Tuesday now, but I'll see if I can get it done before then.

Comment 5

[Bob Owen (:bobowen)]

Attachment: USBViewAll.txt

Had more time than I thought.

This is the integrated camera on my Lenovo W540 laptop.

Manufacturer: SunplusIT
model: can't find this anywhere
Driver Provider: SunplusIT
Driver Date: 18/03/2013
Driver Version: 3.4.7.32
Driver Signer: Microsoft Windows Hardware Compatibility Publisher

Comment 6

[Bob Owen (:bobowen)]

Attachment: USBView-Hello not running.txt

Just realised that probably included more data than was needed.
Also thought that the output might be different when Hello was running, which it seems to be.

Comment 7

[Bob Owen (:bobowen)]

Attachment: USBView-Hello running.txt

Comment 8

[Adam Roach [:abr]]

Awesome, thanks. I'm pinging Jesup to see if he's familiar with any other problems like this one.

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

We need details about the formats supported and selected for the camera:

run with NSPR_LOG_MODULES=getusermedia:5,mediamanager:5,webrtc_trace:65535 WEBRTC_TRACE_FILE=nspr NSPR_LOG_FILE=whatever

The try gum_test.html for Video and NI me back

Comment 10

[Bob Owen (:bobowen)]

Attachment: bug1150539nsprlog.txt

*.child-1 log file was empty

Comment 11

[Bob Owen (:bobowen)]

Attachment: bug1150539nsprlog2.txt.child-1

This time using the actual test you asked for. :-)

Only stuff in the child file this time.

Comment 12

[Randell Jesup [:jesup] (needinfo me)]

I don't see any of the webrtc_trace logs there - did you set WEBRTC_TRACE_FILE=nspr ?  If not, it would have saved the log file in WebRTC.log somewhere (perhaps the current directory...)

It's selecting 848x480@15fps (sigh) with type 2, which should work, so we need the trace logs.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Part 1: log constraints used in MediaManager:5

(In reply to Randell Jesup|PTO until Apr 6 [:jesup] from comment #12)
> It's selecting 848x480@15fps (sigh) with type 2, which should work, so we
> need the trace logs.

What an odd mode to select when the default 640x480 seems available. Hoping I don't have an algorithm bug, and makes me realize we're not logging constraints used.

Comment 14

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Win-build w/constraints-logging available here soon: https://treeherder.mozilla.org/#/jobs?repo=try&revision=154962707b57

Comment 15

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Jan-Ivar Bruaroey [:jib] from comment #13)
> Created attachment 8588121 [details] [diff] [review]
> log constraints used in MediaManager:5
> 
> (In reply to Randell Jesup|PTO until Apr 6 [:jesup] from comment #12)
> > It's selecting 848x480@15fps (sigh) with type 2, which should work, so we
> > need the trace logs.
> 
> What an odd mode to select when the default 640x480 seems available. Hoping
> I don't have an algorithm bug, and makes me realize we're not logging
> constraints used.

Smells like an algorithm bug perhaps predicated on one of the values being exactly the default (prefs) size

Comment 16

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

It's not unusual to support 640x480. Does Hello perhaps constraints? Smells like 16:9 math.

Comment 17

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Bob, would you mind uploading a log again from one of these builds?
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-154962707b57/try-win32

Comment 18

[Bob Owen (:bobowen)]

Attachment: bug1150539nsprlog3.txt.child-1

Sorry, been away helping my step-daughter move house and she has no broadband yet.

(In reply to Randell Jesup|PTO until Apr 6 [:jesup] from comment #12)
> I don't see any of the webrtc_trace logs there - did you set
> WEBRTC_TRACE_FILE=nspr ?  If not, it would have saved the log file in
> WebRTC.log somewhere (perhaps the current directory...)
> 
> It's selecting 848x480@15fps (sigh) with type 2, which should work, so we
> need the trace logs.

Yes, copied the environment variables from your comment, just replace the "whatever"

(In reply to Jan-Ivar Bruaroey [:jib] from comment #17)
> Bob, would you mind uploading a log again from one of these builds?
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/jbruaroey@mozilla.
> com-154962707b57/try-win32

Here's the child log, again nothing in the other one.

Comment 19

[Bob Owen (:bobowen)]

Attachment: bug1150539nsprlog4.txt.child-1

Now I'm back home I could test with the Logitech USB camera, this worked fine and there doesn't appear to be much different in the log file other than it selecting a different resolution.

Comment 20

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Part 2: fix: default to aPrefs.mFPS (30), not aPrefs.mMinFPS (10).

Thanks for the logs! It shows that Hello is not adding any constraints.

I see least two bugs here:

  1) 848x480 @15fps hardly seems like the ideal pick, and
  2) Why isn't that mode working?

This patch might fix the former, because I was erroneously using the "min" frame rate of 10 as the normalizing default rather than 30, making the fitness algorithm [1] prefer lower frame-rates.

Please try again with one of these builds: 

https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-407df847f5c9/try-win32

Separately from this, someone might want to look at why 848x480 @15fps kVideoYUY2 isn't working (is kVideoYUY2 common? just a suspicion).

[1] http://w3c.github.io/mediacapture-main/getusermedia.html#methods-5

Comment 21

[Bob Owen (:bobowen)]

Attachment: bug1150539nsprlog5.txt.child-1

(In reply to Jan-Ivar Bruaroey [:jib] from comment #20)
> Created attachment 8588428 [details] [diff] [review]
> Part 2: fix: default to aPrefs.mFPS (30), not aPrefs.mMinFPS (10).
> 
> Thanks for the logs! It shows that Hello is not adding any constraints.

I'm not sure it matters, but I was using the video test at http://mozilla.github.io/webrtc-landing/gum_test.html here.
Although, I got the same problem with Hello.

> I see least two bugs here:
> 
>   1) 848x480 @15fps hardly seems like the ideal pick, and
>   2) Why isn't that mode working?
> 
> This patch might fix the former, because I was erroneously using the "min"
> frame rate of 10 as the normalizing default rather than 30, making the
> fitness algorithm [1] prefer lower frame-rates.
> 
> Please try again with one of these builds: 
> 
> https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/
> [e-mail]-407df847f5c9/try-win32

This build worked, thanks ... log attached.

By the way I noticed that if I closed the browser without stopping the video first it seemed to leave the broker/chrome process running.
Does the same on Nightly and with my Logitech camera as well.
Is this a known problem?

Comment 22

[Simon Eisenmann]

I can reproduce this with a simple fiddle. See http://jsfiddle.net/34qxx5w1/37/

The fiddle works fine with 640x480 resolution. But as soon as trying to capture HD it does never start the video (camera LED is on and all).

I can reproduce it both in FF37 and FF40. 

I am attaching debug log from FF40. The Camera is a 720p camera which can capture 720p fine with Chrome (uvcvideo: Found UVC 1.00 device Chromebook HD WebCam (2232:1033)). You find this camera for example inside of a Chromebook Pixel.

Comment 23

[Simon Eisenmann]

Attachment: FF40 webrtc capture log with 720p constraint.

Log file when video constraint is set to 720p but no video data is produced.

Comment 24

[Randell Jesup [:jesup] (needinfo me)]

Note that depending on your camera (and how modes are selected by the constraints code), you may end up using MJPEG modes of the camera, which I've just figured out were broken: see bug 1151628 which has a patch to fix them.  I'll push a try shortly so you can download a build to verify if that causes your problem.

Note that almost all cameras support 640x480 in raw YUV which wouldn't hit this, and is our default size.  However, if the constraints code bumps you above that it might hit a MJPEG (mode 10) mode.

Comment 25

[Simon Eisenmann]

(In reply to Randell Jesup [:jesup] from comment #24)
> Note that depending on your camera (and how modes are selected by the
> constraints code), you may end up using MJPEG modes of the camera, which
> I've just figured out were broken: see bug 1151628 which has a patch to fix
> them.  I'll push a try shortly so you can download a build to verify if that
> causes your problem.
> 

This sounds promising i will test asap.

Comment 26

[Randell Jesup [:jesup] (needinfo me)]

I just landed the patch.  Simon - you can download a build either from treeherder of inbound or the Try above, or wait for the nightly tomorrow.  This may well solve your problem

Note that Bob's problem is likely *not* MJPEG, which is mostly used at higher resolutions (mode 10, and he was selecting mode 2 which is raw YUV IIRC).  However, it doesn't hurt to try it.

Comment 27

[Simon Eisenmann]

Thanks. I can confirm that http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-67db53f2b3eb/try-linux64/ build now works with 720p and 1080p capturing. This is the first time ever i saw HD WebRTC with Firefox which is awesome!

Now the only question remains is what released Firefox version this patch might get merged to? I have a patch to Spreed WebRTC (https://github.com/strukturag/spreed-webrtc/pull/188) bringing constraint support for Firefox and right now i have the HD constraints disabled. To make them active i need to know the minimal Firefox version required.

Comment 28

[Randell Jesup [:jesup] (needinfo me)]

HD capture has been working on Mac all along, at least with internal cameras.

I hope to uplift this to Beta (38); I plan to land the disable-warnings patch, then ask for uplift at least to Aurora/39.  I really want it in 38

Comment 29

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8588121 [details] [diff] [review]
Part 1: log constraints used in MediaManager:5

Review of attachment 8588121 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/media/webrtc/MediaEngineCameraVideoSource.cpp
@@ +225,5 @@
> +       aPrefs.GetWidth(), aPrefs.GetHeight(),
> +       aPrefs.mFPS, aPrefs.mMinFPS));
> +  LogConstraints(aConstraints, false);
> +  if (aConstraints.mAdvanced.WasPassed()) {
> +    LOG(("Advanced array[%d]:", aConstraints.mAdvanced.Value().Length()));

Length is typically size_t (unsigned, and may be bigger than uint).  At least make %u.  (Check GetWidth/Height/fps as well, and the logs above)

Comment 30

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Part 1: log constraints used in MediaManager:5 (2) r=jesup

Incorporate feedback (%u, the other ones are int32_t). Carrying forward r=jesup.

Comment 31

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Part 2: fix: default to aPrefs.mFPS (30), not aPrefs.mMinFPS (10). (2) r=jesup

Update commit msg.

Comment 32

[Randell Jesup [:jesup] (needinfo me)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a90f2100a7e0
https://hg.mozilla.org/integration/mozilla-inbound/rev/ce72894838de

Comment 33

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/a90f2100a7e0
https://hg.mozilla.org/mozilla-central/rev/ce72894838de

Comment 34

[Randell Jesup [:jesup] (needinfo me)]

[Tracking Requested - why for this release]:
This will cause lower-framerate, higher-resolution modes to be selected from cameras by default (for example, 1024x576@10fps instead of 640x480@30fps).  This both cuts the framerate and may increase CPU use and bandwidth use.

Very simple, very safe fix.

Comment 35

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8589712 [details] [diff] [review]
Part 2: fix: default to aPrefs.mFPS (30), not aPrefs.mMinFPS (10). (2) r=jesup

Approval Request Comment
[Feature/regressing bug #]: bug 1119335 I believe

[User impact if declined]: This will cause lower-framerate, higher-resolution modes to be selected from cameras by default (for example, 1024x576@10fps instead of 640x480@30fps).  This both cuts the framerate and may increase CPU use and bandwidth use.

[Describe test coverage new/current, TreeHerder]: Needs real cameras for testing currently.  In use on nightly for several weeks.

[Risks and why]: Very simple, very safe fix.  mMinFPS -> mFPS 

[String/UUID change made/needed]: none

Comment 36

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Jan-Ivar Bruaroey [:jib] from comment #20)
> I see least two bugs here:
> 
>   1) 848x480 @15fps hardly seems like the ideal pick, and
>   2) Why isn't that mode working?
> 
> This patch might fix the former,

Bob, please file a new bug on 848x480 @15fps not working for you (if it's still not working for you). I'll close this now to track problem #1 and the patches requested for uplift.

Comment 37

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Jan-Ivar Bruaroey [:jib] from comment #36)
> to track problem #1 and the patches requested for uplift.

... the patch (singular) requested for uplift.

Comment 38

[Bob Owen (:bobowen)]

(In reply to Jan-Ivar Bruaroey [:jib] from comment #36)
> (In reply to Jan-Ivar Bruaroey [:jib] from comment #20)
> > I see least two bugs here:
> > 
> >   1) 848x480 @15fps hardly seems like the ideal pick, and
> >   2) Why isn't that mode working?
> > 
> > This patch might fix the former,
> 
> Bob, please file a new bug on 848x480 @15fps not working for you (if it's
> still not working for you). I'll close this now to track problem #1 and the
> patches requested for uplift.

I haven't tried it recently on my laptop, but I imagine that given that 1) is fixed I won't see 2).

Is there a way I can force it to 848x480 @15fps?
If there is, I'll try it out in the morning and file a separate bug if there's still a problem.

Comment 39

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Try with this one on Nightly http://jsfiddle.net/jib1/6v518qyw

Comment 40

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8589712 [details] [diff] [review]
Part 2: fix: default to aPrefs.mFPS (30), not aPrefs.mMinFPS (10). (2) r=jesup

Approved for uplift to aurora; this fix has been on Nightly for several weeks so has gotten some manual testing and use.

Comment 41

[Liz Henry (:lizzard) (relman/hg->git project)]

Florin can your team verify this fix on 39 once it's landed? Thanks!

Comment 42

[Bob Owen (:bobowen)]

(In reply to Liz Henry (:lizzard) from comment #41)
> Florin can your team verify this fix on 39 once it's landed? Thanks!

As this was originally discovered on my laptop, I'll also verify once landed.

Comment 43

[Randell Jesup [:jesup] (needinfo me)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/31c6d3beb9be

Comment 44

[Bob Owen (:bobowen)]

Pre-test in 39.0a2 (2015-05-08) and video failed to work as expected.

Tested in 39.0a2 (2015-05-10) and video is now working.

---------------------------------------------------

Re a possible follow-up bug ...

(In reply to Jan-Ivar Bruaroey [:jib] from comment #39)
> Try with this one on Nightly http://jsfiddle.net/jib1/6v518qyw

Tried this quickly on Friday, but couldn't get my nspr logging to work for some reason.
I've got it on my list for this week and I'll raise the follow-up bug if there is still a problem.

Comment 45

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

I'm marking this as verified based on the above comment. Thank you very much Bob for verifying this!

Comment 46

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Bob Owen (:bobowen) from comment #44)
> (In reply to Jan-Ivar Bruaroey [:jib] from comment #39)
> > Try with this one on Nightly http://jsfiddle.net/jib1/6v518qyw
> 
> Tried this quickly on Friday, but couldn't get my nspr logging to work for
> some reason.

nspr logging seems hit or miss for me with e10s. What did the fiddle itself say that it got?

Comment 47

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8589712 [details] [diff] [review]
Part 2: fix: default to aPrefs.mFPS (30), not aPrefs.mMinFPS (10). (2) r=jesup

Approved for uplift to 38.0.5 (m-r)

Comment 48

[Ryan VanderMeulen [:RyanVM]]

Is this something that needs consideration for esr38 as well?

Comment 49

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8589712 [details] [diff] [review]
Part 2: fix: default to aPrefs.mFPS (30), not aPrefs.mMinFPS (10). (2) r=jesup

Makes sense, thanks! (note: initial request grew from beta to release).

Approval Request Comment
[Feature/regressing bug #]: bug 1119335 I believe

[User impact if declined]: This will cause lower-framerate, higher-resolution modes to be selected from cameras by default (for example, 1024x576@10fps instead of 640x480@30fps).  This both cuts the framerate and may increase CPU use and bandwidth use.

[Describe test coverage new/current, TreeHerder]: Needs real cameras for testing currently.  In use on nightly for several weeks.

[Risks and why]: Very simple, very safe fix.  mMinFPS -> mFPS 

[String/UUID change made/needed]: none

Comment 50

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/cfa10b9f0f9d

Comment 51

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr38/rev/f2d06043238c

Comment 52

[Andrei Vaida [:avaida]]

Bob, could you please confirm if this bug is also fixed on ESR 38.1.0 [1]?

[1] https://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.1.0esr-candidates/build1/

Comment 53

[Bob Owen (:bobowen)]

(In reply to Andrei Vaida, QA [:avaida] from comment #52)
> Bob, could you please confirm if this bug is also fixed on ESR 38.1.0 [1]?
> 
> [1]
> https://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.1.0esr-
> candidates/build1/

Video is working on that build for me now.
Thanks.

Comment 54

[Andrei Vaida [:avaida]]

Thanks a lot for helping out here, Bob! Updating the status of this bug as well, according to Comment 53.