Closed Bug 1434477 Opened 2 years ago Closed 2 years ago

getUserMedia for fake streams never returns

Categories

(Core :: WebRTC, defect, P2)

58 Branch
defect

Tracking

()

RESOLVED FIXED
mozilla61
Tracking Status
firefox61 --- fixed

People

(Reporter: damencho, Assigned: bryce)

References

Details

Attachments

(4 files)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36

Steps to reproduce:

You need a machine without audio and video devices, this is a common use case running firefox for testing on selenium nodes.
Using media.navigator.permission.disabled:true and media.navigator.streams.fake:true, calling getUserMedia never ends. Attaching a sample html test case.
This is reproduced only with Firefox 58, Firefox 57 works.


Actual results:

The sample prints 'calling gum' and never prints any of the 'got media!' or 'didnt get media'


Expected results:

getUserMedia execution should finish execution. Using the sample test - it should print any of the 'got media!' or 'didnt get media'.
Component: Untriaged → WebRTC
Product: Firefox → Core
We had experienced it on Linux and Windows.
I fail to repro in a VM. But when I remove the sound card on Ubuntu Firefox still see a dummy input and output device on about:support.

Damian could share what about:support shows in the 'Media' section?
Flags: needinfo?(damencho)
Whiteboard: [needinfo 2018/01/31 to reporter]
Summary: getUserMedia never ends using streams fake → getUserMedia for fake streams never returns
Thanks for filing Damian. Can you try again with only {video:true} and after that with only {audio:true} to try to isolate it?

Also I wonder, does it happen only when you set media.navigator.streams.fake=true and not otherwise? Odd, since that's equivalent to using {video: true, audio: true, fake:true} and I wouldn't expect gUM to even really enumerate the underlying hardware stack in that case, so it's odd to learn that having no underlying devices matter. Still possible though if something isn't initialized in that case that we rely on.
We were reproducing this on a headless selenium-node, where it is hard for me to quickly gather the about:support stuff. And the windows machine where we saw that, I don't have currently have access to it.
But I will try gathering input from one of those. I also tried on ubuntu VM but I see same results as Nils shared, dummy in/out devices. Will update later with my findings.
Attached file about-support.txt
Here is the about:support output. No input/output devices in media section.
With media.navigator.streams.fake=true
- video:true, audio:false works fine.
- video:false, audio:true does not complete gum

With media.navigator.streams.fake=false
- video:true, audio:false results in NotFoundError
- video:false, audio:true results in NotFoundError

This was reproduced on a windows7 vm without sound card.
Flags: needinfo?(damencho)
Thanks Damian for the follow up work. I'm finally able to repro this with a Windows VM as well (with audio and video devices removed).
Status: UNCONFIRMED → NEW
Rank: 15
Ever confirmed: true
Priority: -- → P2
Whiteboard: [needinfo 2018/01/31 to reporter]
We are currently reproducing the same issue and with Firefox57. We are trying to identify what broke it on the selenium deployments we have, cause it was working at some point with 57 version. (Maybe there were running some kernel version that were supporting audio/video loop devices or something like that...)
(In reply to Damian Minkov from comment #8)
> We are currently reproducing the same issue and with Firefox57. We are
> trying to identify what broke it on the selenium deployments we have, cause
> it was working at some point with 57 version. (Maybe there were running some
> kernel version that were supporting audio/video loop devices or something
> like that...)

This statement was wrong. It was just the machine missing pulseaudio installed. With it, we have 57 working and 58 not working (we are testing there using tests running against a jitsi-meet deployment, havn't tested with the provided ff-test-getusermedia.html).
Hi, 

my selenium container based testing setup currently faces a similar issue with both Firefox 58.0.1 and the current beta (59.0), however a recent nightly builds (60.0a1) does not encounter this issue anymore.

The setup mostly consists of said docker container provided by the selenium project (https://hub.docker.com/r/selenium/standalone-firefox/) and a testcode controlling the browser via capybara. As docker containers by default don't have any audio or video devices in them, the setup should be pretty much similar to what Damian already described.
Depending on the version the default Firefox 58 is replaced with the beta or nightly version, but the behavior of setting the fake flag in the media constraints does not change.

So I'm just wondering if there is any chance this possible fix will at least be pushed forward to 59 if the fixing change is known, or if waiting until the release of 60 is currently the only available option.

Thanks in advance
Hi, 

we have the same problem in Firefox 58. When we used Firefox 57, all works properly. 

We try to start "pulseaudio" but sometimes works and another no. The problem we've seen is when we try to re-use the docker machine. We are using Selenium to handle the browser.

Thanks in advance

Regards
Just a note to the people experiencing this audio issue on Linux you might not face the same issue, but are affected by Firefox failing to automatically start pulseaudio. As a workaround I have been told it can help to set "--exit-idle-time=-1" to prevent the pulseaudio daemon from exiting after being idle http://manpages.ubuntu.com/manpages/trusty/man5/pulse-daemon.conf.5.html
Bug 1434392 is the pulseaudio issue on Linux I mentioned in comment #12. But that is independent of this bug, which repros on Windows.
See Also: → 1434392
Bryce do you have time to take a look at this? If not, I'll find another owner.

For me it's easy to repro with a VMware with Windows in it and all camera and audio devices removed from the VM settings.
Flags: needinfo?(bvandyk)
I'll grab this and see if I can turn anything up. Should have time for a modest look.
Assignee: nobody → bvandyk
Flags: needinfo?(bvandyk)
(In reply to Nils Ohlmeier [:drno] from comment #12)
> Just a note to the people experiencing this audio issue on Linux you might
> not face the same issue, but are affected by Firefox failing to
> automatically start pulseaudio. As a workaround I have been told it can help
> to set "--exit-idle-time=-1" to prevent the pulseaudio daemon from exiting
> after being idle
> http://manpages.ubuntu.com/manpages/trusty/man5/pulse-daemon.conf.5.html

I tried that and sometimes works but other no, so for the moment it isn't a solution. Because although the pulseaudio does not die, the getUserMedia doesn't return.
Can't repro on my physical boxes. I don't have any Windows VMs, but looking at the information here I've got some things I'd like to investigate.

- At this stage I think this looks like an audio + fakes issue based on comment #6.
- The about:support from comment #5 is interesting. There is no audio backend reported, rather than an audio backend, but with no devices.

Could those experiencing the issue please help me out via the following:
- navigate to about:support (in the URL bar)
- find the 'media' section and let me know what your backend is
- let me know what operating system you're on

If possible it would help me to know if you face the same issue when using nightly (and on what platform).

I wonder if the pulse issue discussed results in us not finding a backend, and not having a backend results in this bug.
Here are the Media section values using Firefox 58.02 where the problem is reproducible:

Media
-----

Audio Backend: unknown
Max Channels: 0
Preferred Channel Layout: unknown
Preferred Sample Rate: 44100
Output Devices
Name: Group
Input Devices
Name: Group

The OS I took those results is Windows 10 Enterprise Evaluation Insider Preview (used the free download option from Microsoft and installed it in virtualbox). We also reproduce it on a custom setup selenium-grid with ubuntu machines to run Firefox for integration testing, this is where we first identified the problem, those Linux machines are without audio support in the kernel.

I also installed Firefox Developer Edition 60.0b1, where I also reproduce the problem. The difference is that I see some other error, which I'm not sure whether it is connected: "Clamped larged numeric value.   (unknown)". The media section in about:support is the same as above.
Thank you, Damian! That's useful to know, and should help me configure an environment to try and reproduce.

If anyone else is able to provide similar information it would help me confirm and would be appreciated!
(In reply to Damian Minkov from comment #18)
> Media
> -----
> 
> Audio Backend: unknown
> Max Channels: 0
> Preferred Channel Layout: unknown
> Preferred Sample Rate: 44100
> Output Devices
> Name: Group
> Input Devices
> Name: Group

On my Windows 7 VM where I can repro this problem I have exactly the same "Media" section as Damian (with Fx 58 and 60).
I'm having the same issue with my automated tests on SauceLabs VMs. Mac OS X doesn't have this problem, but Windows 10 does. The media section for Windows 10 matches the ones listed above, and Mac has:

Media 

Audio Backend	audiounit
Max Channels	2
Preferred Channel Layout	stereo
Preferred Sample Rate	48000
Output/Input Devices are listed as Loopback Audio
Here's what I have so far:

I can repro with the VM's provided by Microsoft for dev envs: https://developer.microsoft.com/en-us/windows/downloads/virtual-machines.

We don't have an issue when using real devices because the gUM enumeration path picks up no devices and we fail out. When using a fake device we do pick up a device and then we try and get it set up, however Cubeb init fails and we don't properly handle that.

AsyncCubebTask's Run[0] tries to init Cubeb, but fails. However this error is not propagated. AudioCallbackDriver::Start[1] appears to be the what is directly triggering the init. I think the source of this in the MediaManager is InitializeAsync[2], which would need to be made aware of the failure. It looks like the GraphDriver isn't setup such that we can report these error states at the moment. :padenot, would appreciate your thoughts on if there's anything that can be done here.

The other piece, which I'm going to look more into now, is the Cubeb init failing. I'm concerned this may be a limitation imposed on us by WASAPI if we have no audio devices on Windows, but will attempt to confirm this.

[0] https://searchfox.org/mozilla-central/source/dom/media/GraphDriver.cpp#492
[1] https://searchfox.org/mozilla-central/source/dom/media/GraphDriver.cpp#742
[2] https://searchfox.org/mozilla-central/source/dom/media/MediaManager.cpp#3837
Flags: needinfo?(padenot)
As you are mentioning WASAPI and windows, I just wanted to note that this is also reproducible on linux (kernel with no audio/video support which is a common case running aws instances.). Thanks for working on that.
WASAPI issue looks to be down to needing a device to init the backend[0].

I've been unable to repro on Linux thus far. My suspicion is that if we have no audio support at all on such machines is that cubeb will find no backend, resulting in a similar issue (in which case I'm not sure there is a sane work around for cubeb init failing). I'll try to setup a VM with no sound to confirm my suspicion, though my VMs thus far have proved rather resistant to me trying to gut them ^_^

[0] https://github.com/kinetiknz/cubeb/blob/789eaaa3b0d77b55e110353972d6713889f510ea/src/cubeb_wasapi.cpp#L1176
(In reply to Bryce Van Dyk (:bryce) from comment #23)
> Here's what I have so far:
> 
> I can repro with the VM's provided by Microsoft for dev envs:
> https://developer.microsoft.com/en-us/windows/downloads/virtual-machines.
> 
> We don't have an issue when using real devices because the gUM enumeration
> path picks up no devices and we fail out. When using a fake device we do
> pick up a device and then we try and get it set up, however Cubeb init fails
> and we don't properly handle that.
> 
> AsyncCubebTask's Run[0] tries to init Cubeb, but fails. However this error
> is not propagated. AudioCallbackDriver::Start[1] appears to be the what is
> directly triggering the init. I think the source of this in the MediaManager
> is InitializeAsync[2], which would need to be made aware of the failure. It
> looks like the GraphDriver isn't setup such that we can report these error
> states at the moment. :padenot, would appreciate your thoughts on if there's
> anything that can be done here.

That's correct. However, we should reach this point [3], and we should fallback on a normal thread. The source of all this is having at least one track that is of kind "audio" in the MediaStreamGraph. As soon as this is the case, the MSG opens an audio output or input or both.

Here, this is the case: we're adding track that clearly is audio, but we don't have audio devices, so it's erroring out. Why we're not falling back on a SystemClockDriver is the real question.

I'll note in passing that "fake" is proprietary and will be removed, and that people should provision test VM with the proper modules and software to have audio devices, or skip this altogether and use HTMLMediaElement and the Web Audio API to simulate voice activity.

[3]: https://searchfox.org/mozilla-central/source/dom/media/GraphDriver.cpp#700
Flags: needinfo?(padenot)
Comment on attachment 8960330 [details]
Bug 1434477 - Add extra logging to MediaManager's device enumeration path.

https://reviewboard.mozilla.org/r/229074/#review235064
Attachment #8960330 - Flags: review?(padenot) → review+
Comment on attachment 8960331 [details]
Bug 1434477 - If AudioCallbackDriver::Init() can't get a cubeb context, fall back to system clock driver.

https://reviewboard.mozilla.org/r/229076/#review235066

This is great. Can you add a test? Bug 1443525 is about adding a test about this very specific scenario, with a strategy.
Attachment #8960331 - Flags: review?(padenot) → review+
Will do. Will get test ready before landing this.
Pushed by bvandyk@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0018a815422d
Add extra logging to MediaManager's device enumeration path. r=padenot
https://hg.mozilla.org/integration/autoland/rev/249d42dff1b5
If AudioCallbackDriver::Init() can't get a cubeb context, fall back to system clock driver. r=padenot
https://hg.mozilla.org/mozilla-central/rev/0018a815422d
https://hg.mozilla.org/mozilla-central/rev/249d42dff1b5
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
You need to log in before you can comment on or make changes to this bug.