docker-worker: Allow containers to access loopback devices configured on the host

RESOLVED FIXED

Status

Taskcluster
Docker-Worker
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: garndt, Assigned: garndt)

Tracking

Details

Attachments

(1 attachment)

52 bytes, text/x-github-pull-request
lightsofapollo
: review+
Details | Review | Splinter Review
(Assignee)

Description

3 years ago
Certain test suites, such as mochitests using the --use-test-media-device option, require the use of a video loopback device that's provided by v4l2loopback [1].  These devices are not available to docker containers that are not running with --privileged

[1] https://github.com/umlaeute/v4l2loopback


Some reference:
https://bugzilla.mozilla.org/show_bug.cgi?id=826265
https://bugzilla.mozilla.org/show_bug.cgi?id=815002
lightsofapollo is the docker expert, but I'm pretty sure we can expose a specific device to a container.

So we can probably just create <capacity> number of v4l2loopback devices on the host and map into different containers.
Check out the "--device" flag for docker. 

If we do something like that, we probably don't even need to require a specific scope for tasks that wants v4l2loopback device.
Assignee: nobody → jlal
Status: NEW → ASSIGNED
I saw the discussion on IRC, were you consider mapping a video0 device into all containers.
Even if this is lightweight I would suggest we hide it under a feature flag,
because it modifies the container.
And we may have other video devices in the future who knows.

Unrelated remark, I'm not sure what the difference between --device and --volume when mouting from /dev/...
Depends on: 1141422
Update:

Trying this with the native driver failed horribly because of cgroup permissions ... lxc driver allows --device with makes this easier... So far I have video bits working and audio devices mounted but pulse audio is still fighting me (permission errors?) after that is sorted we can use media devices!
pulse audio starts as a service (normally!) but does not start in the container for (some probably init related reasons) workaround is easy:

pulseaudio &

... 

Whats left?

Add code to map devices to containers... I don't think we want to make a direct map of device -> container but instead introduce two new "features"

 - loopbackAudio 
 - loopbackVideo

Video is a single device /dev/videoX where audio is mounted /dev/snd/... (I have less clarity at this point which device does what but there are a few per audio loopback).

Both the video (v4l2loopback) and audio (snd-loopback) are kernel modules which must be loaded during docker-worker boot and passed various flags (see docs for each) to create individual devices. It's worth noting that the video devices seem to be limited to 8 (at least on c3.xlarge).
Okay- I have some WIP stuff on the gecko side (have not yet started on the code for docker worker) but hit some other issue... The webtrc tests seem to work with minimal capabilities set (they did not before which is a good sign) but those which use more complex capabilities (like specifying a screen share) seem to still have issues... Debugging that now (to see if we need to do more here...)
Handing off the WIP stuff I had (and was slow to finish) to greg
Assignee: jlal → garndt
v4l2loopback allows you to pass a devices=N module parameter to get more devices:
https://github.com/umlaeute/v4l2loopback#options
Yeah^ fwiw my WIP stuff I handed off attempts to set video devices to max (though I have only gotten up to 8 devices regardless of what I have tried so far....)
(Assignee)

Comment 9

3 years ago
8 device limit might be related to this issue I found.  Package for ubuntu might have been built not overriding this. https://github.com/umlaeute/v4l2loopback/issues/55
(Assignee)

Comment 10

3 years ago
Created attachment 8603316 [details] [review]
Worker PR 85

This PR is a continuation of the work :lightsofapollo started with enabling video and audio loopback devices on the host.

Some notes:
Currently only 8 video loopback devices are created with the ubuntu package.  Attempted to create more than this by compiling the module myself, but then there were issues using it with gstreamer.  We can enter a bug to allow more than 8 when we need it.

In tree gecko changes are needed to enable pulseaudio and update the task definition to include the devices (to be done separately)

Here is a link [1] to a graph ran with the mulet mochitests.  Chunk 2 was of importance because this is the one that was failing previously.  Currently we are still disabling the option "use-test-media-devices".  This can be re-enabled once the current work is complete and mochitests are greened up as much as possible.

[1] http://docs.taskcluster.net/tools/task-graph-inspector/#ix51JWCMQLmEPy6slG38jw
Attachment #8603316 - Flags: review?(jlal)
Attachment #8603316 - Flags: review?(jlal) → review+
(Assignee)

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Component: TaskCluster → Docker-Worker
Product: Testing → Taskcluster
You need to log in before you can comment on or make changes to this bug.