INFRA-139: Investigate feasibility of running VirtualBox in a container

Description

Solution should:

• NOT require hacks in the VirtualBox installation on the hosts
• NOT require heavyweight Docker container (avoid too much logic as much as possible)
• permit inspection of running VM but this is not a hard requirement (if it requires logging into a container)

The use case is someone submitting a Kubernetes job to run tests against a Vagrant/VirtualBox image and getting results back.

Comments

• Giovanni Tirloni commented 2018-03-27T11:23:06.025-0400

  I got a working prototype (based on VirtualBox 5.2.8 and Vagrant 2.0.3, on CentOS 7.4) but I'm currently facing some issues with SELinux and VirtualBox's GUI. Still working on that.

• Giovanni Tirloni commented 2018-03-28T11:45:12.521-0400

  Running VirtualBox and Vagrant inside a container is certainly doable, provided the host has VirtualBox drivers installed. The container needs to be privileged.

  Running GUI applications inside Docker is more complicated (exemplified by the size of the x11docker shell script). When a Vagrant box runs in non-headless mode, it's creating a GUI window and thus poses these challenges. More specifically, because it was created in Qt5, it needs a working D-Bus connection and the Xvfb server alone is not enough.

  Both the CentOS and Fedora boxes work fine in headless mode. When trying to start in GUI mode, the processes are killed though:

  type=ANOM_ABEND msg=audit(1522102229.772:2026): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:spc_t:s0 pid=1784 comm="VirtualBox" reason="memory violation" sig=6
  type=ANOM_ABEND msg=audit(1522102808.466:2058): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:spc_t:s0 pid=3739 comm="QDBusConnection" reason="memory violation" sig=6
  

  Another issue faced was the default 10GiB limit on the root filesystem mounted inside the container. This cannot be customized neither by the Dockerfile nor by a command line option to `docker run`. It's configured in the docker daemon itself with the option `--storage-opt dm.basesize=100G`, so it means relying on a bigger root disk is not portable.

  I've identified that the CI pipeline for Infusion could be optimized to use less disk space (FLUID-6266 and GPII/ci-service PR#38). Currently it needs more than 10GB but by simply removing extra commands (e.g.`npm install` outside the VM, building Infusion 2x), it's possible to use 7-8GB in total.

  Another option is to provide the container with a scratch area through a Docker volume (or a Kubernetes local mount point) for the "$HOME/VirtualBox VMs" directory. This way the root disk remains almost unused, except for checking out the code repository (or even that could stay outside the container since it's going to be thrown away at the end of the process).

  It's also possible to keep a volume with the latest version of the Vagrant boxes, so containers don't need to download them every time

  Leaving the issues of GUI applications aside (and the requirements for a development-focus environment, which translates into developers needing to see the actual browser window to troubleshoot issues), it's possible to migrate the execution of the CI pipelines into containers. There's also the question of whether VirtualBox and Vagrant are even necessary in this case but I'm also ignoring that for now.

  Dockerfile:

  FROM centos:7
  
  RUN yum -y install git openssh-clients && \
      yum -y install https://releases.hashicorp.com/vagrant/2.0.3/vagrant_2.0.3_x86_64.rpm  && \
      yum -y install https://download.virtualbox.org/virtualbox/5.2.8/VirtualBox-5.2-5.2.8_121009_el7-1.x86_64.rpm
  

  Starting the container (and using an pre-existing .vagrant.d directory from the host):

  docker run --rm -ti -v /home/jenkins-fluid/.vagrant.d/boxes:/root/.vagrant.d/boxes -v /dev/vboxdrv:/dev/vboxdrv -v /dev/vboxnetctl:/dev/vboxnetctl --privileged virtualbox-vagrant
  

  The Vagrantfile needs to be modified to run in CI (it could leave in a separate directory or the existing Vagrantfile could have logic to dynamically change this, it's Ruby after all):

  --- a/Vagrantfile
  +++ b/Vagrantfile
  @@ -52,6 +52,7 @@ Vagrant.configure(2) do |config|
     config.vm.hostname = app_name
   
     config.vm.provider :virtualbox do |vm|
  +    vm.gui = false
       vm.customize ["modifyvm", :id, "--memory", ram]
       vm.customize ["modifyvm", :id, "--cpus", cpus]
       vm.customize ["modifyvm", :id, "--vram", "256"]
  

  In summary, running VirtualBox/Vagrant inside a container is doable provided the box is configured as "headless" (or "separate"). There are some challenges with storage but nothing too serious, although the issues with the GUI are hard to fix (goes against "no hacks" item above).

  Finally, all tests in Infusion and GPII/universal work fine.

• Giovanni Tirloni commented 2018-04-03T04:36:25.826-0400

  Next steps would be to go ahead and implement this for real and/or consider if it's even necessary in all use cases. New JIRA should be opened.