1224791 - Requesting KVM support for x86-emulator on task cluster

Status: RESOLVED WONTFIX

(Taskcluster :: Operations and Service Requests, task, P2)

Description

[Josh Cheng [:josh]]

This bug is for requesting KVM support for x86-emulator on task cluster

Comment 1

[Josh Cheng [:josh]]

Hi Cervantes,
Could you provide more technical detail requirement about KVM version etc?
Thanks!

Comment 2

[Cervantes Yu [:cyu] [:cervantes]]

According to https://fedoraproject.org/wiki/How_to_enable_nested_virtualization_in_KVM , linux kernel version >= 2.6.26 should suffice.

Comment 3

[Greg Arndt [:garndt]]

Ok, so I was under the impression that this would be to run our existing docker worker on a KVM enabled host, such as digital ocean. Looking at comment 2, this is to support nested virtualization which changes the problem a little bit.  I was aware of the two conversations about this, but didn't tie the two together thinking they were related.  That was my fault. 

We would not only need to create a provisioner for digital ocean (although testing a proof of concept doesn't require this), but also we would need to write a worker backend that supported spinning up vms rather than docker containers.  We do have some initial rough plans for a worker with a pluggable execution backend, but that is a couple of quarters from being completed. 

I think we need to have a discussion as to the priority of this kind of environment and how rolling it out fits into other goals of the teams involved.

To understand the problem we're trying to solve, do we have some information by nested virtualization is the preferred method and how it stacks up against a docker based solution?  I'm curious where the speed gains are with running in a vm rather than in a docker container and if perhaps maybe there is something we could do in our existing environment to speed that up and allow people to iterate quickly.

Comment 4

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Hi

(In reply to Greg Arndt [:garndt] from comment #3)
> To understand the problem we're trying to solve, do we have some information
> by nested virtualization is the preferred method and how it stacks up
> against a docker based solution?  I'm curious where the speed gains are with
> running in a vm rather than in a docker container and if perhaps maybe there
> is something we could do in our existing environment to speed that up and
> allow people to iterate quickly.

Docker is a based on containers. It's a complete user-space solution for virtualization where all Docker containers share the same kernel and hardware. We'd have to use whatever kernel and hardware the system provides. That means that we'd have no means of controlling the hardware during our automated tests.

Emulators emulate hardware on which a complete system, including kernel, runs. It's by default all done in software and extremely slow (by an order of magnitude at least). KVM is a Linux driver that provides access to the processor's support for virtualization; so the processor executes the instructions in the virtual machine directly in hardware.

There's probably no significant performance difference between Docker and KVM-supported emulation, but Docker doesn't support our use cases.

Is this helpful?

Comment 5

[Greg Arndt [:garndt]]

Thanks for filling in the gaps in my knowledge between the two.  That's a lot more helpful, thank you.

Comment 6

[Cervantes Yu [:cyu] [:cervantes]]

So Digital Ocean provides docker containers. That means the automation runs inside a docker container instead of a layer of emulation. Then qemu only needs a level of virtualization, instead of the unsupported nested virtualization. This sounds like emulator-x86-kk in full speed is doable.

Comment 7

[Thomas Zimmermann [:tzimmermann] [:tdz]]

(In reply to Cervantes Yu [:cyu] [:cervantes] from comment #6)
> So Digital Ocean provides docker containers. That means the automation runs
> inside a docker container instead of a layer of emulation. Then qemu only
> needs a level of virtualization, instead of the unsupported nested
> virtualization. This sounds like emulator-x86-kk in full speed is doable.

That's really hard to say, because Docker and KVM are two entirely separate things. The kernel that runs the Docker containers could itself be within a virtual machine. In this case, we'd still need nested virtualization to have KVM available in the Docker containers.