826265 - Install loopback audio/video drivers on Linux test slaves

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[(not currently active) Ted Mielczarek]

In order to get more real-world test coverage on our test slaves we need to install loopback audio and video drivers on the Linux test slaves. I've borrowed a test slave and have been able to get this working. Audio and Video are two separate things.

Video is pretty simple. We just build the v4l2loopback driver, copy it to the right place and modprobe it: https://github.com/umlaeute/v4l2loopback

Audio is a little more complicated. Fedora Core 12 didn't ship with the snd-aloop kernel module, so we have to build it from the alsa-drivers package. I downloaded the version that matched the version of ALSA in use on the test slaves and built it, but copying just the snd-aloop module in caused kernel panics when I tried to modprobe it. I was able to get it to work by building all the drivers and "make install"ing them, presumably there was a version mismatch there. I think we should be able to just copy all the driver files to the right directory and have this work, but I haven't tested in a clean environment yet.

To autoload the modules I created the file:
/etc/sysconfig/modules/audiovideoloop.modules

Containing simply:
#!/bin/sh
modprobe snd-aloop
modprobe v4l2loopback

Comment 1

[(not currently active) Ted Mielczarek]

Oh, we also need a minor change to alsa.conf, I'll attach that along with the binary versions of the drivers I've built.

Comment 2

[(not currently active) Ted Mielczarek]

Attachment: v4l2loopback.ko.gz

Compiled v4l2loopback driver. This should be installed by doing:
cp v4l2loopback.ko /lib/modules/2.6.31.5-127.fc12.x86_64/extra/
depmod -a

Comment 3

[(not currently active) Ted Mielczarek]

Compiled alsa drivers:
http://people.mozilla.com/~tmielczarek/alsa-driver-modules-fc12.tar.gz

You should be able to just untar these in /lib/modules/`uname -r`/kernel/sound; run `depmod -a`; and reboot.

Comment 4

[(not currently active) Ted Mielczarek]

$ sha1sum ~/alsa-driver-modules-fc12.tar.gz
35546470e1b54b6916fc6badd7ea8d6190aa05d6  /home/luser/alsa-driver-modules-fc12.tar.gz

Comment 5

[(not currently active) Ted Mielczarek]

Attachment: audiovideoloop.modules

This script should be installed as /etc/sysconfig/modules/audiovideoloop.modules and chmod +x.

Comment 6

[(not currently active) Ted Mielczarek]

Attachment: alsa.conf patch

Patch to /usr/share/alsa.conf.

Comment 7

[cmtalbert]

This is really necessary for webRTC testing.  Do we have an ETA on this deployment?

Comment 8

[Justin Wood (:Callek)]

coop,

so clint is eager to see this done, joel and me also brought up the concept that it might be pointless to invest time to do this for Fedora though since we're almost ready for Ubuntu.

We have been living with this sub-par testing since November when we started testing webRTC, but this is needed to actually test the audio stuff properly.

Not sure if/when anyone in our group can tackle though, given the other priorities we all have. I'll leave you to juggle.

--

Ted,

Few thoughts I have here,
*) If we are to switch off fedora in this quarter on the machines that run these tests, do you feel we should still invest in moving forward on this?
**) Do we need this at all for Ubuntu?
**) Do you have any idea if this would work in AWS with no physical hardware backing things?
*) I don't [personally] care for deploying "local made binaries" of anything, without clear and explicit steps on what you did to build it, in what environment, etc. Such that we (or community) can replicate if we need, or modify it slightly, etc.
**) What did you build for those alsa binaries, where, how?
*) Is there a way to deploy this without needing a full system restart (I think of a puppet deploy being easier that way)?
*) Does it offer any risks/issues with the concept that multiple "trains" would use the same system as trunk?

I'll leave any followup Q's to whomever ends up taking on this work.

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to Justin Wood (:Callek) from comment #8) 
> Few thoughts I have here,
> *) If we are to switch off fedora in this quarter on the machines that run
> these tests, do you feel we should still invest in moving forward on this?
> **) Do we need this at all for Ubuntu?
> **) Do you have any idea if this would work in AWS with no physical hardware
> backing things?
> *) I don't [personally] care for deploying "local made binaries" of
> anything, without clear and explicit steps on what you did to build it, in
> what environment, etc. Such that we (or community) can replicate if we need,
> or modify it slightly, etc.
> **) What did you build for those alsa binaries, where, how?
> *) Is there a way to deploy this without needing a full system restart (I
> think of a puppet deploy being easier that way)?
> *) Does it offer any risks/issues with the concept that multiple "trains"
> would use the same system as trunk?

Realistically we're not going to be off of Fedora this quarter. I've played the IT lead-times game too many times before.

Besides, Ted's already done all the work here, and even with the old puppet, we should still be able to deploy this pretty easily. Also, given that we *are* moving away from Fedora sometime soon-ish, I'm not willing to push too hard on Ted to fully document his solution here.

I'll find an owner for this at our group mtg on Monday.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

what test suite/chunk do these tests live in?  This might be a delay in moving to ubuntu, we are down to just a few test cases which need attention and we will be at 100% (for the unittests)

Comment 11

[(not currently active) Ted Mielczarek]

Most of the hard work for this was getting things to work on the ancient Fedora 12. Ubuntu 12.04 should have the snd-aloop driver already installed (it worked fine out of the box on my machine), so it'd just need the modprobe bits. The v4l2loop driver is simple, it's literally just a "configure && make && sudo make install", so we could rebuild that against a kernel that matches what we've got on the Ubuntu machines pretty trivially.

If you'd like me to sort out the specifics of making this work on the Ubuntu VMs, I'd be happy to find some time given a loaner. I expect it to be fairly simple given that I've got it working locally on Ubuntu 12.04.

(In reply to Justin Wood (:Callek) from comment #8)
> Few thoughts I have here,
> *) If we are to switch off fedora in this quarter on the machines that run
> these tests, do you feel we should still invest in moving forward on this?
> **) Do we need this at all for Ubuntu?

Yes.

> **) Do you have any idea if this would work in AWS with no physical hardware
> backing things?

Should be fine, these are loopback drivers, their whole point for existing is to be fake devices.

> *) I don't [personally] care for deploying "local made binaries" of
> anything, without clear and explicit steps on what you did to build it, in
> what environment, etc. Such that we (or community) can replicate if we need,
> or modify it slightly, etc.
> **) What did you build for those alsa binaries, where, how?

I can dig up the details here if needed, it's not terribly complicated, I just grabbed the alsa-drivers source package matching the version of alsa installed on the Fedora slaves, and built it on a FC12 that I installed in a VM.

> *) Is there a way to deploy this without needing a full system restart (I
> think of a puppet deploy being easier that way)?

Unlikely, since I couldn't figure out a way to build just this one driver in a compatible way, you wind up having to reinstall all the ALSA drivers, so you have to restart to get them loaded properly.

> *) Does it offer any risks/issues with the concept that multiple "trains"
> would use the same system as trunk?

I don't think there should be any issue here, nothing is going to use these drivers unless it's explicitly trying to.

Comment 12

[Rail Aliiev [:rail]]

Ted, I have a couple of questions:

1) Is there any minimal requirements for v4l2loopback-dkms version? Would 0.4.1 be OK? We can install 0.4.1 for free - no need to compile a custom package.

2) The same for snd-aloop. This is what we have for free:
$ modinfo snd_aloop
filename:       /lib/modules/3.2.0-23-generic/kernel/sound/drivers/snd-aloop.ko
license:        GPL
description:    A loopback soundcard
author:         Jaroslav Kysela <[e-mail]>
srcversion:     5EE1B8BDC8810BAC7FFE988
depends:        snd-pcm,snd
intree:         Y
vermagic:       3.2.0-23-generic SMP mod_unload modversions 
parm:           index:Index value for loopback soundcard. (array of int)
parm:           id:ID string for loopback soundcard. (array of charp)
parm:           enable:Enable this loopback soundcard. (array of bool)
parm:           pcm_substreams:PCM substreams # (1-8) for loopback driver. (array of int)
parm:           pcm_notify:Break capture when PCM format/rate/channels changes. (array of int)

Comment 13

[(not currently active) Ted Mielczarek]

Any version of snd-aloop should be fine. I think we can use v4l2loopback 0.4.1, we worked around the thing I patched in it in bug 809554. If we can just install those both from packages let's do that and see where it gets us.

Comment 14

[Rail Aliiev [:rail]]

Attachment: install and enable loopback modules on Ubuntu

Instead of adding needed modules into one file (/etc/modules) I added a modified version of /etc/init/module-init-tools.conf to load modules from multiple files. It simplifies their management.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 723903 [details] [diff] [review]
install and enable loopback modules on Ubuntu

I like the changed modprobe script, but it's not really a tweak.  Those should probably be added with something like

  kernelmodule {
      "snd-aloop": ;
  }

There *are* alsa packages for Ubuntu.  packages::alsa should install those, not assume they are installed.

root@talos-linux64-ix-001:~# dpkg --get-selections | grep alsa
alsa-base                                       install
alsa-utils                                      install
bluez-alsa                                      install
bluez-alsa:i386                                 install
gstreamer0.10-alsa                              install

(I assume just alsa-base and alsa-utils are required?)

As the patch stands, I don't like a package::* to be adding kernel modules, but I will feel a lot better if it installs some packages (alsa-base and alsa-utils) and then does some setup (installing the kernel module).

Finally, I know you discussed the need for a restart in previous comments.  It looks like this will install the modules but not actually activate them, implicitly relying on the slave's first job not being a WebRTC test, right?  After bug 838351 lands, we'll have support for puppet indicating the system needs a reboot.  Perhaps this patch should do the same?

Comment 16

[(not currently active) Ted Mielczarek]

You shouldn't need a restart if you're just installing the packages. You can just modprobe the modules and things will work.

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

Then we'll need puppet code to do the modprobes :)

Comment 18

[Rail Aliiev [:rail]]

Attachment: install and enable loopback modules on Ubuntu v2

I haven't tested this, but so far it looks like a more elegant and flexible way to deal with kernel modules.

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 724269 [details] [diff] [review]
install and enable loopback modules on Ubuntu v2

Review of attachment 724269 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/kernelmodule/manifests/init.pp
@@ +8,5 @@
> +        CentOS, Ubuntu: {
> +            exec {
> +                "modprobe-$module":
> +                    command => "modprobe $module $module_args",
> +                    unless  => "lsmod | grep -qw ^${module}",

This should probably have some kind of termination on the regexp - whitespace?

@@ +16,5 @@
> +                package {
> +                    $packages:
> +                        ensure => latest;
> +                }
> +                Exec["modprobe-$module"] { require +> Package[$packages] }

Any particular reason to use an Exec with unless, rather than adding to /etc/modules and using refreshonly => true on the exec?

::: modules/talos/manifests/init.pp
@@ +16,5 @@
>              include packages::nodejs
>              include packages::llvm
>              # required for the 32-bit reftests per :ahal, bug 837268
>              include packages::ia32libs
> +            kernelpackage {

itym kernelmodule

Comment 20

[Rail Aliiev [:rail]]

(In reply to Dustin J. Mitchell [:dustin] from comment #19)
> > +                    unless  => "lsmod | grep -qw ^${module}",
> This should probably have some kind of termination on the regexp -
> whitespace?

grep -w (word) does that.
 
> @@ +16,5 @@
> > +                package {
> > +                    $packages:
> > +                        ensure => latest;
> > +                }
> > +                Exec["modprobe-$module"] { require +> Package[$packages] }
> 
> Any particular reason to use an Exec with unless, rather than adding to
> /etc/modules and using refreshonly => true on the exec?

I wanted to make it work not only on Ubuntu, this is why this time I tried to be straightforward. /etc/modules is Debian/Ubuntu only as well as Upstart. I'll think about this part a little bit more.

> ::: modules/talos/manifests/init.pp
> @@ +16,5 @@
> >              include packages::nodejs
> >              include packages::llvm
> >              # required for the 32-bit reftests per :ahal, bug 837268
> >              include packages::ia32libs
> > +            kernelpackage {
> 
> itym kernelmodule

Bah. Easy to fix. :)

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

Fair enough.  I think the only issue with running the exec every time is slightly slower puppet runs, which isn't a big deal.

Comment 22

[Rail Aliiev [:rail]]

Attachment: install and enable loopback modules on Ubuntu v2.1

Tested and it worked without any problems.

I checked the alsa.conf patch, all needed variables set ex needed out of the box.

Comment 23

[Rail Aliiev [:rail]]

Attachment: install and enable loopback modules on Ubuntu v2.2

s/snd-aloop/snd_aloop/

modprobe works for both snd-aloop and snd_aloop, but lsmod lists it as snd_aloop.

Comment 24

[Rail Aliiev [:rail]]

Attachment: install and enable loopback modules on Ubuntu v2.3

before => is more readable.

Comment 25

[Chris AtLee [:catlee]]

Comment on attachment 725072 [details] [diff] [review]
install and enable loopback modules on Ubuntu v2.3

Review of attachment 725072 [details] [diff] [review]:
-----------------------------------------------------------------

thanks!

Comment 26

[Rail Aliiev [:rail]]

Comment on attachment 725072 [details] [diff] [review]
install and enable loopback modules on Ubuntu v2.3

http://hg.mozilla.org/build/puppet/rev/ffc193221048

Comment 27

[Rail Aliiev [:rail]]

ted, should we bother ourselves with installing the modules on Fedora slaves? They will be history soon! :)

Comment 28

[(not currently active) Ted Mielczarek]

I think we can leave them be. Thanks for fixing this!

Comment 29

[Rail Aliiev [:rail]]

all done here!