1106941 - Firefox Hello doesn't work properly when no video camera is installed

Status: VERIFIED FIXED

(Hello (Loop) :: Client, defect, P2)

Description

[pieceredd]

Attachment: Screenshot showing expected popup, working on webrtc-landing page.

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:34.0) Gecko/20100101 Firefox/34.0
Build ID: 20141201171754

Steps to reproduce:

I fired up Firefox with a clean profile, started up Hello, and copied the link to a friend (also tested by manually typing it into my Firefox OS device with Hello installed). Upon seeing the call request, I accepted the call, and after waiting a few seconds heard the other caller and saw their webcam. However, my own audio (from my microphone) wasn't sent, and Pulseaudio showed no applications recording audio.  

I also tested my setup using this URL: http://mozilla.github.io/webrtc-landing/gum_test.html using the 'Audio' button. The box asking to share my microphone (As seen in attached file) popped up as expected, and I could hear my own microphone through my headset. 


Actual results:

No audio was captured from my microphone, and no popup window asked for the permission to share my microphone.


Expected results:

The popup window (as shown in attached screenshot) should have appeared, and allowed my microphone to be used for Hello calling.

Comment 1

[u527282]

Hello, 

I have exactly the same issue on Linux x86_64 with Firefox 34. 

When I send a link to a contact to start a Hello session, Firefox don't ask me if i want to share my mic or webcam, but when i receive a link, it asks.

Comment 2

[u279076]

I tested (and was able to reproduce) with my computer using Mac OS 10.10 and my friend's computer using Ubuntu 14.04.1 64-bit. As such this bug is not specific to Linux 64-bit. 

1. Start Firefox 34.0.5 with a new profile
2. Enter Customize mode and add the Firefox Hello button to my toolbar
3. Exit Customize mode and click the Hello button
4. Share the URL with a friend (also using Firefox 34.0.5) and have them load it
> A Firefox Hello page will load with a green Start button
5. Have the friend click the Start button
> A conversation widget appears in my browser window asking me to accept the incoming call
6. Accept the incoming call
> The call starts with my audio/video shared (no getUserMedia notification appears)

Note that in Firefox 35 and beyond the above user flow has changed dramatically but the case remains the same: the person sharing the URL is not prompted to share they're devices. This may be working as designed (bug 1015486) and the permission assumed in this case, though I'm not 100% sure.

I'm nominating to block Firefox 35 so the privacy implications of this bug can be reviewed.

Comment 3

[u279076]

Sorry, I think I may have misunderstood the point of this bug report. Upon another read through I think what's really at issue here is that no audio is being shared. I still think the privacy question I raised is valid but I'm find taking that to another bug report if necessary.

Unfortunately, I am only able to reproduce the "no permission prompt" part of this issue. The "no audio" part of this issue is not reproducible to me at all. I'll need more information to reproduce the "no audio shared" aspect of this bug. 

As a start it would be good to know your exact steps to reproduce. Secondarily it would be good to know your Linux distribution, your devices, and any configuration changes you've made to said devices and related software.

Comment 4

[Maire Reavy [:mreavy]]

Not asking for permission on the "link generator" side is by design;  it's a product requirement.

Audio is definitely not failing due to a lack of permission prompt.  Most likely the Linux box believes that another microphone is the default, and Hello currently uses the default microphone and camera.  If I'm correct, Bug 1023934 would resolve this issue.

For anyone who can reproduce this problem: As a test, check to see what microphones are offered in the gUM test.  My bet is there are more than 2 in the permission prompt. Further, check to see what your default mic is set to;  it's probably different than the one that works in the gUM test.  Changing the default mic to the one you want to use should solve the problem.  If it doesn't, that would be the real bug here.  If it does solve the problem, I'd like to resolve this as either INVALID or  a dup of Bug 1023934.

For now, I am removing the nomination for blocking Fx35 because I believe it's all working as designed.  I am anxious to implement Bug 1023934 which is scheduled for implementation shortly after the new year (and ride the train in Fx38).

Comment 5

[u527282]

gUM test asks which mic I want to use. I have only 2 choices in the permission prompt. The two choices works on the test. You can find in attachment a screenshot of permission prompt choices. 

But when I send a link, Hello don't access to the mic. 

Procedure : 

1. Generate a link and send it 
2. Receive the call 
3. Accept the call 

Results : 

The mic still not working. 

----- 

Linux Disto : Debian Jessie x86_64
Device : Dell Latitude D830 
Firefox Version : 34.0

Comment 6

[u527282]

Attachment: Capture-gUM Test Page - Mozilla Firefox.png

Permission prompt on gUM test to choice mic on Linux x86_64.

Comment 7

[Maire Reavy [:mreavy]]

I'm pretty sure this is a getUserMedia platform issue, so I'm moving this to Core::WebRTC-Audio/Video.

To [e-mail]:  If you can turn on WebRTC logging as described here: https://wiki.mozilla.org/Media/WebRTC/Logging (in particular the "Media" logging) before your next Hello call and then attach the logs from that call to this bug, those logs should give us enough info to dive deeper on this bug.  Thanks!

Comment 8

[u527282]

Please find attached the WebRTC.log as you requested.

This is how I proceeded : 

1. Export the environment variables in a shell 
2. Start Firefox using the shell
3. Start Hello conversation 

Tell me if i'm wrong anywhere in the process this is the first time I do it. :-)

Comment 9

[u527282]

Attachment: WebRTC log

Comment 10

[Mark Banner (:standard8)]

sami.rhatay: Do you have a camera available on your machine? or are you audio-only? I'm wondering if something to do with that might be messing us up.

Comment 11

[u527282]

Hello Mark, 

I don't have camera on my computer, I'm using Hello with audio only.

Comment 12

[Mark Banner (:standard8)]

Moving back to Loop product as this seems to be an issue with no-camera.

Comment 13

[Mark Banner (:standard8)]

From bug 1124422 comment 0:

> If you start a conversation on a computer which has no video devices
> available, the gUM call still succeeds because it delivers just an audio
> stream back (note: the same happens in Firefox (Chrome handles this
> differently) if the user chooses to not grant access to his camera - but for
> the native client this does not apply because of the implicit permissions)
> and a conversation gets created. But if someone joins the call from the
> standalone page the call never gets established properly.
> 
> As Firefox has not yet implemented enumerate devices, there is currently no
> other option to catch this then checking the tracks in the stream from gUM.

Comment 15

[Mark Banner (:standard8)]

I took a bit of time looking at this on a VM I hooked up.

It appears that if an application requests gUM for audio + video, but only audio is available then it will only get audio information.

However, if we're bypassing the permissions dialog, as we do for Loop (due to the built-in design), then the gUM request fails with "NotFoundError", coming from:

http://mxr.mozilla.org/mozilla-central/source/dom/media/MediaManager.cpp#1236

My current thinking is that the gUM dialog/mechanism is automatically detecting that there's no video devices, and adjusting the constraints before we hit the GetUserMediaTask.

I've taken a look and it seems like we could potentially set a value on the task to ignore missing video in the privileged case. However, I'm not sure that's the right approach.

Randell/Florian, any suggestions here?

Comment 16

[Florian Quèze [:florian]]

(In reply to Mark Banner (:standard8) from comment #15)

> I've taken a look and it seems like we could potentially set a value on the
> task to ignore missing video in the privileged case. However, I'm not sure
> that's the right approach.
> 
> Randell/Florian, any suggestions here?

I think making the code path for the privileged Hello behavior match the behavior of the prompt would make sense, yes.

Comment 18

[Maire Reavy [:mreavy]]

For Hello, I think you could special case it until we have device enumeration support.

However, this is fundamentally a platform issue and a gUM spec issue.  I organized a quick meeting of the Firefox WebRTC platform folks on Friday to discuss what we should do on the platform side.  We will add device enumeration support (targeting Fx38) to the platform code which will require a little bit of UI work to copy the device list from the permission doorhanger to the in-call doorhanger.  (I'll file a bug for that next.)  Once we have device enumeration support and it has been in release for a little while, we'll fail a request for video+audio if video is not available.  However, with device enumeration, the app should only request audio and we'll send black (or an icon to indicate there is purposely no video) to the other side when video is not available or muted.

I think Hello should switch to use device enumeration once it's landed, but for now we can kludge it.

Comment 19

[:shell escalante]

triaging to blocking-loop:Fx38+ for tracking this release, all bugs in 38+ and 38? now, will move to "backlog+" for Fx39.  After that - we won't triage to a specific release any longer.  we will work from an ordered backlog.  Lower rank in P2 only as not sure it's actionable until after platform work.

Comment 20

[Nils Ohlmeier [:drno]]

Bug 1124422 describes how this can be fixed today, without waiting for the enumerate devices implementation from platform. That workaround would/should then get replaced once enumerate devices support is available from platform.
But I don't think we should for the platform fix through enumerate devices here, because the enumerate devices support would ride the trains and therefore end customers would be left with this issue for lots of weeks ahead.

Comment 21

[Mark Banner (:standard8)]

Attachment: Proof of concept fix

This is a hacked up proof of concept fix.

The idea is that rather than failing when GetUserMediaTask detects no video, we'll detect the "privileged" situation and assume there are no camera options available. Hence, we turn off the video feeds, and proceed with adding any audio feed.

I think we should still fail if there's no audio feed - that's kinda critical for detecting we're not going to succeed at all.

I've not done any testing on this yet other than to see if it works on my setup where I can reproduce this (which it does).

Comment 22

[Mark Banner (:standard8)]

[Tracking Requested - why for this release]: This is a pretty non-obvious bug for a user that they'll hit if they don't have a camera installed and they do have a microphone.

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

Tracking since this does appear to be a bad user experience, however we need an assignee if it's going to be resolved on a short (5 week) beta cycle of 37 - Jesup can you suggest someone?

Comment 24

[Maire Reavy [:mreavy]]

Moving the needinfo to find an owner from Randell to Gavin -- unless Mark wants to take this since he put up the proof of concept fix already.  :-)

Comment 25

[:Gavin Sharp [email: [e-mail]]]

My understanding is that this is a platform bug - if Mark has cycles to follow up on his PoC fix, that's great, but from IRC it sounded like maybe jib could help too?

Comment 26

[Maire Reavy [:mreavy]]

Just to validate some of my assumptions about what this bug is all about:
1) The best solution is for platform to support enumerate devices, which should land sometime next week (Fx39 cycle). 
2) We want a solution for the next release of Firefox (Fx37) and not to wait for device enumeration to ride the trains

Given the above, I'd prefer to solve this in JS-land.  The change in Loop would be to add code to retry a failed gUM request once (and only once) with an audio-only request to see if that succeeds.  That solution should be easy, low risk, and very upliftable with no need to back it out when device enumeration lands.   

However, I realize that things aren't always as clean as we expect.  For example, the SDK may not handle this well.  So I'd appreciate it if someone could make a patch to try this (don't spend more than 30 mins on it) and see if it's as easy as I think.  If not, we can do a hack in the platform code (which I believe is uglier but doable), but we'd need to back it out as soon as device enumeration lands.  So I'd like the platform solution to be our fallback plan, not our preferred solution.

Comment 27

[Mark Banner (:standard8)]

I've got a partial patch for improving the UX (a little) when no devices found and I think I could easily adapt it to this case. So assigning to self. I'll try and look at in within the next 24 hours.

Comment 28

[Mark Banner (:standard8)]

Attachment: WIP Handle no microphone

This is a WIP fix so that we don't attempt to get the camera if not necessary.

The basic idea here is:

- If the gUM fails for any reason
- "Turn off" video
- Re-start the publishing process

Note: Due to the way the sdk works (it always asks for video & audio, unless it knows they aren't there via device enumeration), we have to manually simulate the device enumeration. This is awkward, but possible, and I've done it in this patch. I'm going to see if I can do it only for desktop to reduce the risk points.

Comment 29

[Mark Banner (:standard8)]

Attachment: WIP Handle no microphone v2

This version has checks to see if the code is running on desktop, and if so applies the work arounds. Otherwise it doesn't.

Comment 31

[Mark Banner (:standard8)]

Sylvestre: We discussed this in-team yesterday and decided it wasn't quite enough/appropriate to ship it in an extra release (mainly due to risk profile & existing bugs already being fixed). However, please could it be added to the release notes for 36 as a known issue?

Thanks.

Comment 32

[Sylvestre Ledru [:Sylvestre]]

Sure, what kind of wording would you like ?
"Firefox Hello might have some issues if no camera has been found" ?

BTW: we are using the tracking flags for the release notes now, not the keyword.

Comment 33

[Mark Banner (:standard8)]

(In reply to Sylvestre Ledru [:sylvestre] from comment #32)
> Sure, what kind of wording would you like ?
> "Firefox Hello might have some issues if no camera has been found" ?

Something like this would be better:

"Firefox Hello does not work for link generators if there is no camera installed"

Comment 34

[Mark Banner (:standard8)]

Attachment: Part 1. Firefox Hello doesn't work properly when no video camera is installed - fix rooms and outgoing conversations.

This works around the issue just for desktop for rooms and outgoing calls. Incoming ones need a separate patch, I'm seeing what that looks like before I commit to it in this bug.

In the meantime, I think this part can start to be reviewed.

I've restricted the "retry" part of the fix just to desktop to minimise impact - the bit that is different is standalone will now pick up on complete gUM failures as well. This should improve the UI a bit ("something went wrong"), but I'm going to file a separate bug to improve the string - as we want to uplift this one.

Try push here for easier testing (I need it to confirm things on my Windows VM): https://treeherder.mozilla.org/#/jobs?repo=try&revision=93e64434fc0b

Comment 35

[Sylvestre Ledru [:Sylvestre]]

Thanks for the wording. Updated!

Comment 36

[Mike de Boer [:mikedeboer]]

Comment on attachment 8571917 [details] [diff] [review]
Part 1. Firefox Hello doesn't work properly when no video camera is installed - fix rooms and outgoing conversations.

Review of attachment 8571917 [details] [diff] [review]:
-----------------------------------------------------------------

Well, we knew up front that this would turn out to be a bit hacky.  But it works like a charm :)

Thanks!

Comment 37

[Mark Banner (:standard8)]

Attachment: Part 2. Firefox Hello doesn't work properly when no video camera is installed - fix incoming conversations.

This fixes the incoming conversations. The isDesktop passing isn't ideal, but these views will go away with the conversation view rework.

Comment 39

[Mark Banner (:standard8)]

I pushed part 1 to get earlier testing in nightlies, though part 2 will hopefully land tomorrow anyway:

https://hg.mozilla.org/integration/fx-team/rev/d94db91d6fb1

Comment 40

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/d94db91d6fb1

Comment 41

[Mike de Boer [:mikedeboer]]

Comment on attachment 8573923 [details] [diff] [review]
Part 2. Firefox Hello doesn't work properly when no video camera is installed - fix incoming conversations.

Review of attachment 8573923 [details] [diff] [review]:
-----------------------------------------------------------------

the `isDesktop` passing doesn't bother me that much, TBH.

Thanks for this! O, yeah, please remove that console.log statement before landing ;)

::: browser/components/loop/content/shared/js/views.jsx
@@ +277,5 @@
> +         * XXX This is a workaround for desktop machines that do not have a
> +         * camera installed. As we don't yet have device enumeration, when
> +         * we do, this can be removed (bug 1138851), and the sdk should handle it.
> +         */
> +console.log("loop.conversation is ", loop.conversation);

Whoops! :)

Comment 43

[Mark Banner (:standard8)]

For verification purposes (once both patches land):

- Setup a computer with a microphone but without a camera.
- Have a second computer/browser to connect to.
- On the one without a camera, create a conversation, join it with the second computer and verify the conversation works correctly (audio one way, video/audio the other way if the second computer has a camera)
- Sign into both via FxA
- Set up a direct call from of the FxA accounts to the other and verify the media works correctly.
- Set up a direct call in the other direction, and again verify media works correctly.

Comment 44

[Mark Banner (:standard8)]

https://hg.mozilla.org/integration/fx-team/rev/db75b48611b5

Comment 45

[Mark Banner (:standard8)]

Comment on attachment 8571917 [details] [diff] [review]
Part 1. Firefox Hello doesn't work properly when no video camera is installed - fix rooms and outgoing conversations.

Approval Request Comment
[Feature/regressing bug #]: Firefox Hello's conversations & direct calls
[User impact if declined]: If a user has a microphone but doesn't have a camera, then they can't connect to a conversation they generate nor make/receive direct calls.
[Describe test coverage new/current, TreeHerder]: Landed on fx-team, waiting for m-c merge. Manually tested in a variety of situations, see comment 43
[Risks and why]: Low - works around missing functionality in the platform and just adds a re-try to a media request that fails.
[String/UUID change made/needed]: None

Note: Requires bug 1140363 to land as well.

Comment 46

[Mark Banner (:standard8)]

Comment on attachment 8573923 [details] [diff] [review]
Part 2. Firefox Hello doesn't work properly when no video camera is installed - fix incoming conversations.

Approval Request Comment
[Feature/regressing bug #]: Firefox Hello's conversations & direct calls
[User impact if declined]: If a user has a microphone but doesn't have a camera, then they can't connect to a conversation they generate nor make/receive direct calls.
[Describe test coverage new/current, TreeHerder]: Landed on fx-team, waiting for m-c merge. Manually tested in a variety of situations, see comment 43
[Risks and why]: Low - works around missing functionality in the platform and just adds a re-try to a media request that fails.
[String/UUID change made/needed]: None

Note: Requires bug 1140363 to land as well.

Comment 47

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/db75b48611b5

Comment 48

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 8571917 [details] [diff] [review]
Part 1. Firefox Hello doesn't work properly when no video camera is installed - fix rooms and outgoing conversations.

Glad to see this resolved, thank you.

Comment 49

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/15dce2aeae06
https://hg.mozilla.org/releases/mozilla-aurora/rev/a37bffb6cfb0

Comment 50

[Ryan VanderMeulen [:RyanVM]]

Needs more rebasing than I'm comfortable doing for beta.

Comment 51

[Lawrence Mandel [:lmandel] (use needinfo)]

Mark - Can you please rebase for Beta, preferably before the gtb for Beta 5 on Thursday?

Comment 52

[Mark Banner (:standard8)]

I'll take a look in a few minutes.

Comment 53

[Mark Banner (:standard8)]

Ok, I've fixed the bitrot and re-tested locally. No functional changes, it was all pure bitrot.

https://hg.mozilla.org/releases/mozilla-beta/rev/245598f10dcd
https://hg.mozilla.org/releases/mozilla-beta/rev/9e52698fd237

Comment 54

[Steven McArdle]

Sorry guys but this 'privileged' functionality of Hello violates the whole point of GUM permissions.

It is NEVER alright to just access the camera and or microphone of the browser UNLESS the user has SPECIFICALLY granted permission to do so at least once. I don't care what the requirement of the app is, this should NOT be violated.

It would also be an unfair bias for Firefox to have this specific functionality and not allow other WebRTC solutions to do the same.

Comment 55

[Steven McArdle]

I think labelling my previous comment as Advocacy is a bit rich to be honest.

I am not trying to influence decisions here and I don't belong to any politically motivated group. If anything I am very much a WebRTC advocate.

I am however the designer and lead architect for a WebRTC based telehealth platform and as such I have a very vested interest in this particular topic with regards firefox.

Up until now our FF version would ask for permission to use the camera and microphone from the user every time a call is initiated even though it always happens over an SSL connection. This is in stark contrast to Chrome where permission is asked only the first time the user initiates or accepts a call. This is the functionality I want to see supported by FF.

The point I am trying to make here is one of a users sense of privacy. If FF can bypass the user giving permission to enable the camera and microphone then why can't the NSA or any other organisation for that matter ????

Users all over the World are already anti NSA and are paranoid to the extent that some even cover the camera with sticky tape when not in use.

Up until now browsers have been a relative safe haven as it has not been possible to active the users devices without their permission. This change has now negated that, and that is of interest to your users as they have lost the ability to say no... 

At the end of the day this 'privileged' functionality is at least concerning at darn right dangerous at the worst.

I see absolutely no reason whatsoever why FF Hello could not ask the user for permission, even if that permission only needs to be granted once. At least it would be a conscious decision the user would have to make.

Steven McArdle

Comment 56

[Nils Ohlmeier [:drno]]

Steven I would highly recommend you to take your concerns to a new bug, rather then commenting on a closed bug which is only vaguely related to your concern.

Comment 57

[Eric Rescorla (:ekr)]

(In reply to Nils Ohlmeier [:drno] from comment #56)
> Steven I would highly recommend you to take your concerns to a new bug,
> rather then commenting on a closed bug which is only vaguely related to your
> concern.

Indeed.

With that said, you seem to be confused about the threat model here. The consent
dialog in this case is implemented in the Firefox browser core, and exists to prevent
Web sites from accessing your camera and microphone without your consent. A malicious
attacker who controlled Firefox could just access those resources without your
consent anyway, regardless of whether Hello was designed to prompt the user for
permission under normal circumstances.

If you wish to pursue this, please file a new bug or raise it on firefox-dev.

Comment 58

[Maire Reavy [:mreavy]]

FYI: We've created a new bug (Bug 1145053) to discuss how we can improve the experience for first-time users and help them understand and be ok with their camera turning on when they click "Start a conversation".  If anyone is interested in being part of that discussion, that's the bug to follow.

Comment 59

[Bogdan Maris, Desktop Test Engineering]

I can`t seem to make an Audio call using Firefox Accounts if I don`t have a Camera on Linux or Windows (I have an iMac here and can`t disconnect that camera). 

Builds tested: Firefox 38beta6 and latest Nightly.

Logged into account [A] on Ubuntu 14.04 32-bit (no camera, only audio) and Audio called account [B] Mac OS X 10.9.5 (camera and audio)

Linux output:
> "OT.Publisher.onStreamAvailableError NotFoundError: Unknown Error while getting user media" sdk.js:1409:12
> "OT.exception :: title: Unable to Publish (1500) msg: GetUserMedia" sdk.js:1409:12
> 1500 "Unknown Error while getting user media" sdk.js:1409:12
> "OT.exception :: title: Unable to Publish (1500) msg: Unknown Error while getting user media" sdk.js:1409:12

- 'Something went wrong' message in conversation window.

Comment 60

[Mark Banner (:standard8)]

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #59)
> I can`t seem to make an Audio call using Firefox Accounts if I don`t have a
> Camera on Linux or Windows (I have an iMac here and can`t disconnect that
> camera). 
> 
> Builds tested: Firefox 38beta6 and latest Nightly.
> 
> Logged into account [A] on Ubuntu 14.04 32-bit (no camera, only audio) and
> Audio called account [B] Mac OS X 10.9.5 (camera and audio)

Ok, that's a good catch. You should find you can initiate the call as a video call (it auto-downgrades to audio), but not as an audio call(!).

So this bug improved one aspect, but not all of it.

Can you file the audio-only calls as a separate bug please? I think we could switch the backend code to a new API that's available which would fix the bug and slightly simplify the code as well.

Comment 61

[Bogdan Maris, Desktop Test Engineering]

Closing this bug as verified fixed based on testing done in comment 59, the remaining work will be tracked in dependencies bugs.