826549 - LLVM or VirtualBox on ci.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Description

[Erik Rose [:erik][:erikrose]]

In order to build https://ci.mozilla.org/job/dxr-github/, the Jenkins build boxes will either need LLVM installed or else the ability to boot a Vagrant VM so I can install my own dependencies. Though more network-intense, the latter is more modular and will probably cause fewer headaches down the road. (/me envisions the spectre of conflicting library versions.) It'll also have the bonus of continually validating that our vagrant setup, used for devs, is good.

I see we already have the Vagrant Jenkins plugin installed. All we're missing is the vbox kernel module (deduced from https://ci.mozilla.org/job/dxr-github/23/console).

This is pretty much blocking further work on DXR.

Thanks!

Comment 1

[Shyam Mani [:fox2mike]]

IIRC this was all done in Bug 703193. I'm wondering if it got upgraded and we just need to kick some services.

Comment 2

[Jake Maul [:jakem]]

P3 (instead of P4) because it blocks DXR dev, and I'd really like to encourage DXR dev work... as there is at least a chance that it might someday replace MXR. :)

This seems likely to be just a simple package installation either way. I'm not thrilled about kernel modules, but if it's just "yum install" then I won't block it. :)

Comment 3

[Erik Rose [:erik][:erikrose]]

Yay! (I'm sure other projects with big stacks will like it as well.)

Comment 4

[Brandon Burton [:solarce]]

The kernel module daemon was failing to start, there was also an upgrade for VirtualBox available

So I uninstalled the previous package and installed the latest available, which reinstalled the kernel modules

[[e-mail]1 log]# /etc/init.d/vboxdrv start
Starting VirtualBox kernel modules                         [FAILED]
  (modprobe vboxdrv failed. Please use 'dmesg' to find out why)
[[e-mail]1 log]# modprobe vboxdrv
FATAL: Module vboxdrv not found.
[[e-mail]1 log]# yum remove VirtualBox-4.1
Loaded plugins: downloadonly, rhnplugin, security
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package VirtualBox-4.1.x86_64 0:4.1.18_78361_rhel6-1 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================
 Package                        Arch                   Version                               Repository                   Size
===============================================================================================================================
Removing:
 VirtualBox-4.1                 x86_64                 4.1.18_78361_rhel6-1                  @virtualbox                 130 M

Transaction Summary
===============================================================================================================================
Remove        1 Package(s)

Installed size: 130 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing    : VirtualBox-4.1-4.1.18_78361_rhel6-1.x86_64                                                                  1/1 
  Verifying  : VirtualBox-4.1-4.1.18_78361_rhel6-1.x86_64                                                                  1/1 

Removed:
  VirtualBox-4.1.x86_64 0:4.1.18_78361_rhel6-1                                                                                 

Complete!
[[e-mail]1 log]# yum install VirtualBox-4.1
Loaded plugins: downloadonly, rhnplugin, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package VirtualBox-4.1.x86_64 0:4.1.24_82872_el6-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================
 Package                         Arch                    Version                             Repository                   Size
===============================================================================================================================
Installing:
 VirtualBox-4.1                  x86_64                  4.1.24_82872_el6-1                  virtualbox                   64 M

Transaction Summary
===============================================================================================================================
Install       1 Package(s)

Total download size: 64 M
Installed size: 132 M
Is this ok [y/N]: y
Downloading Packages:
VirtualBox-4.1-4.1.24_82872_el6-1.x86_64.rpm                                                            |  64 MB     00:42     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : VirtualBox-4.1-4.1.24_82872_el6-1.x86_64                                                                    1/1 

Creating group 'vboxusers'. VM users must be member of that group!

No precompiled module for this kernel found -- trying to build one. Messages
emitted during module compilation will be logged to /var/log/vbox-install.log.

Stopping VirtualBox kernel modules [  OK  ]
Uninstalling old VirtualBox DKMS kernel modules [  OK  ]
Trying to register the VirtualBox kernel modules using DKMS [FAILED]
  (Failed, trying without DKMS)
Recompiling VirtualBox kernel modules [  OK  ]
Starting VirtualBox kernel modules [  OK  ]
  Verifying  : VirtualBox-4.1-4.1.24_82872_el6-1.x86_64                                                                    1/1 

Installed:
  VirtualBox-4.1.x86_64 0:4.1.24_82872_el6-1                                                                                   

Complete!

[[e-mail]1 log]# /etc/init.d/vboxdrv status
VirtualBox kernel modules (vboxdrv, vboxnetflt, vboxnetadp, vboxpci) are loaded.

I've marked this bug dependent on a Jenkins improvements bug I'm working on, as I'd like to setup a slave which is dedicated to Vagrant plugin using builds, so I am going to leave it open for now

But go ahead and try to use the Vagrant plugin and see if it works.

Comment 5

[Erik Rose [:erik][:erikrose]]

Thanks! That makes my day. It got much further this time, exploding only when it tried to mount the shared folder (the source code) into the VM. What do you think of https://ci.mozilla.org/job/dxr/10/console? Can we make NFS happy?

Comment 6

[Erik Rose [:erik][:erikrose]]

Thinking about this a little more, I remember that the NFS mount magic through which Vagrant shares the folder requires sudo access. Not ideal. Hmm. So if that makes you too uncomfortable, maybe I can come up with some crazy way of SFTP-ing the source code from the working dir into the VM or something.

Comment 7

[Shyam Mani [:fox2mike]]

I think this might need an include nfsclient added to the jenkins puppet module...but that's a 0030 guess ;)

Comment 8

[Brandon Burton [:solarce]]

(In reply to Erik Rose [:erik][:erikrose] from comment #5)
> Thanks! That makes my day. It got much further this time, exploding only
> when it tried to mount the shared folder (the source code) into the VM. What
> do you think of https://ci.mozilla.org/job/dxr/10/console? Can we make NFS
> happy?

Is there a reason you can't git clone your code inside the Vagrant VM with the "Execute shell script in Vagrant as admin" build step?

I tried this out with https://ci.mozilla.org/job/vagrant-jenkins-test-job1 and a quick repo i threw together, https://github.com/solarce/vagrant-jenkins-test-job1

It worked well in my testing so far, https://ci.mozilla.org/job/vagrant-jenkins-test-job1/4/console

Let me know

Thanks

Comment 9

[Erik Rose [:erik][:erikrose]]

It's not horrible, but there is a race condition: if somebody commits, then Jenkins pulls, then somebody else commits before the script runs, it'll test against a mishmash of the old Vagrant config and the new commit. Frankly, that's probably fine 99% of the time, and I'll happily fall back to it if NFS is a pain to set up and SFTP trickery proves nontrivial.

Comment 10

[Erik Rose [:erik][:erikrose]]

That strategy would probably also not work with Leeroy Jenkins, which we use to vet pull requests prior to merging.

Comment 11

[Brandon Burton [:solarce]]

(In reply to Erik Rose [:erik][:erikrose] from comment #10)
> That strategy would probably also not work with Leeroy Jenkins, which we use
> to vet pull requests prior to merging.

That makes sense. I haven't poked at how much work NFS would take. 

I'll do so now

Comment 12

[Brandon Burton [:solarce]]

The ci.mozilla.org maintenance last was successful.

I did learn one thing in talking with :rtyler, one of the Vagrant plugin authors on IRC this morning.

The Vagrant plugin bundles it's own copy of Vagrant, so we're "stuck" on Vagrant 1.0.2 right now because that's what 0.1.4 of the supports. We're also on VirtualBox-4.1.latest for now because that's what the plugin supports.

The Jenkins upgrade to 1.480.2 did seem to change on behavior in the Vagrant plugin, this option in the Vagrantfile doesn't work anymore, https://github.com/solarce/vagrant-jenkins-test-job1/commit/92d8c014d4704740927130df24cb47463f4e9b95#L0L14 , so if you're using it, you'll need to remove it.

If you don't remove it, you'll see an error like https://ci.mozilla.org/job/vagrant-jenkins-test-job1/lastFailedBuild/console , but without it you should be able to do a successful build, like https://ci.mozilla.org/job/vagrant-jenkins-test-job1/17/console

I am still investigating the NFS support

Comment 13

[Erik Rose [:erik][:erikrose]]

Here's a fun little piece of news: I got Vagrant and Jenkins working last night, and I did not use the Vagrant Jenkins plugin to do it. Rather than check the "Boot Vagrant VM" box, I used an Execute Shell build step which (indirectly) just runs "vagrant up" and such. I did this partly because the checkbox's launch of vagrant doesn't leave enough breadcrumbs for a subsequent "/opt/vagrant/bin/vagrant ssh -c 'cd /home/vagrant/dxr && make test'" to work, and I have to do that (rather than a "Execute shell script in vagrant" step) because I need the Jenkins env vars to be around at that point so I can toggle NFS behavior based on them. (The checkbox's launch happens so early that the vars aren't around.) Phew!

So, I don't think I'm actually using any of the facilities of the Jenkins vagrant plugin. As far as I'm concerned, we can upgrade with impunity!

The only downside to my approach (that I can think of) is that the vagrant box might not be getting torn down. I'm not sure if vagrant is putting its VMs someplace that gets blown away automatically by Jenkins. I could put a `vagrant destroy` step in my build scripts, but I'd have to make sure it ran even if something else failed before it.