1099057 - docker-worker: Allow containers to access loopback devices configured on the host

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Greg Arndt [:garndt]]

Certain test suites, such as mochitests using the --use-test-media-device option, require the use of a video loopback device that's provided by v4l2loopback [1].  These devices are not available to docker containers that are not running with --privileged

[1] https://github.com/umlaeute/v4l2loopback


Some reference:
https://bugzilla.mozilla.org/show_bug.cgi?id=826265
https://bugzilla.mozilla.org/show_bug.cgi?id=815002

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

lightsofapollo is the docker expert, but I'm pretty sure we can expose a specific device to a container.

So we can probably just create <capacity> number of v4l2loopback devices on the host and map into different containers.
Check out the "--device" flag for docker. 

If we do something like that, we probably don't even need to require a specific scope for tasks that wants v4l2loopback device.

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

I saw the discussion on IRC, were you consider mapping a video0 device into all containers.
Even if this is lightweight I would suggest we hide it under a feature flag,
because it modifies the container.
And we may have other video devices in the future who knows.

Unrelated remark, I'm not sure what the difference between --device and --volume when mouting from /dev/...

Comment 3

[James Lal [:lightsofapollo]]

Update:

Trying this with the native driver failed horribly because of cgroup permissions ... lxc driver allows --device with makes this easier... So far I have video bits working and audio devices mounted but pulse audio is still fighting me (permission errors?) after that is sorted we can use media devices!

Comment 4

[James Lal [:lightsofapollo]]

pulse audio starts as a service (normally!) but does not start in the container for (some probably init related reasons) workaround is easy:

pulseaudio &

... 

Whats left?

Add code to map devices to containers... I don't think we want to make a direct map of device -> container but instead introduce two new "features"

 - loopbackAudio 
 - loopbackVideo

Video is a single device /dev/videoX where audio is mounted /dev/snd/... (I have less clarity at this point which device does what but there are a few per audio loopback).

Both the video (v4l2loopback) and audio (snd-loopback) are kernel modules which must be loaded during docker-worker boot and passed various flags (see docs for each) to create individual devices. It's worth noting that the video devices seem to be limited to 8 (at least on c3.xlarge).

Comment 5

[James Lal [:lightsofapollo]]

Okay- I have some WIP stuff on the gecko side (have not yet started on the code for docker worker) but hit some other issue... The webtrc tests seem to work with minimal capabilities set (they did not before which is a good sign) but those which use more complex capabilities (like specifying a screen share) seem to still have issues... Debugging that now (to see if we need to do more here...)

Comment 6

[James Lal [:lightsofapollo]]

Handing off the WIP stuff I had (and was slow to finish) to greg

Comment 7

[(not currently active) Ted Mielczarek]

v4l2loopback allows you to pass a devices=N module parameter to get more devices:
https://github.com/umlaeute/v4l2loopback#options

Comment 8

[James Lal [:lightsofapollo]]

Yeah^ fwiw my WIP stuff I handed off attempts to set video devices to max (though I have only gotten up to 8 devices regardless of what I have tried so far....)

Comment 9

[Greg Arndt [:garndt]]

8 device limit might be related to this issue I found.  Package for ubuntu might have been built not overriding this. https://github.com/umlaeute/v4l2loopback/issues/55

Comment 10

[Greg Arndt [:garndt]]

Attachment: Worker PR 85

This PR is a continuation of the work :lightsofapollo started with enabling video and audio loopback devices on the host.

Some notes:
Currently only 8 video loopback devices are created with the ubuntu package.  Attempted to create more than this by compiling the module myself, but then there were issues using it with gstreamer.  We can enter a bug to allow more than 8 when we need it.

In tree gecko changes are needed to enable pulseaudio and update the task definition to include the devices (to be done separately)

Here is a link [1] to a graph ran with the mulet mochitests.  Chunk 2 was of importance because this is the one that was failing previously.  Currently we are still disabling the option "use-test-media-devices".  This can be re-enabled once the current work is complete and mochitests are greened up as much as possible.

[1] http://docs.taskcluster.net/tools/task-graph-inspector/#ix51JWCMQLmEPy6slG38jw

Comment 11

[Greg Arndt [:garndt]]

Comment on attachment 8603316 [details] [review]
Worker PR 85

merged https://github.com/taskcluster/docker-worker/commit/1ce48a67c0b3b45a9f321ca98e59a132db9b2978