Homework 1

E6698 Virtual Machines Spring 2018

DUE: Monday 2/5/2018 at 11:59pm EST

This assignment will be done using KVM, a popular type-2 hypervisor. KVM is built on the Linux kernel to reuse its existing functions to support virtualization. As a part of this assignment, you will be experimenting with KVM and gaining familiarity with the development environment and add new features to it. KVM can run on many different architectures, but the specific platform we will be targeting is the x86_64 CPU family.

We will use VMware Workstation/Fusion in this assignment so your custom hypervisor can be isolated from the rest of your system. VMware Workstation/Fusion supports nested virtualization, allowing you to install, run and develop hypervisors in a virtual machine. You will first install a Linux/KVM host on VMware, and then create virtual machines on the host. Note that this is different from deployment in production platforms which the hypervisor runs directly on bare metal.

You should submit patches for your hypercall implementation to the TA via email, in a manner similar to how patches are submitted to the Linux kernel mailing lists. You will then be asked to arrange a time to demo your homework assignment to the TA.


  1. (20 pts.) Create a Virtual Machine on KVM.
    The first thing you need to do is install Linux/KVM in a VMware VM, then install another VM running Linux on top of the VMware VM. Running a VM inside of a VM is referred to as nested virtualization. The VM running inside of a VM is sometimes referred to as a nested VM. We are asking you to configure your own VMs instead of giving you a preconfigured system so you can learn how to do this on your own. The following steps have been tested on VMware Workstation 12 PRO, though they should also work with other recent versions of VMware Workstation or Fusion.
    1. Create a New Virtual Machine.
    2. Select "Custom (advanced) and click Next".
    3. Click next until you reach the page "Guest Operating System Installation"
    4. Download the iso image for Ubuntu-14.04.5 from here.
    5. Continue to setup your VM spec.
    6. We then need to expose the hardware virtualization feature to the KVM running in the VM. On VMware Workstation, go to "Processors" of your VM configuration, then select both "Virtualize Intel VT-x/EPT and AMD-V/RVI" and "Virtualize CPU performance counters".

      NOTE: The exact location to the virtualization hardware to the VM might be different depending on the VMware version.

      NOTE: You may need to enable hardware virtualization (Intel VT) features for your computer in the BIOS. You can get more information from here.
    7. Click "Finish" and Install Ubuntu-14.04.5 on your VM.

    Once your VM is set up, you can use the terminal directly or ssh into the VM to run commands.
    sudo apt-get update
    sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils libosinfo-bin libguestfs-tools virt-top virtinst

    Now we are ready to install a VM (nested VM) on KVM. Type the following command to install a Ubuntu 14.04.5 guest. virt-install \
    --name guest0 \
    --virt-type=kvm \
    --ram 1024 \
    --disk path=guest0.img,size=25 \
    --vcpus 2 \
    --os-type linux \
    --graphics none \
    --console pty,target_type=serial \
    --location 'http://us.archive.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/' \
    --extra-args 'console=ttyS0,115200n8 serial'

    You can then log in to the VM after virt-install finishes. Next, modify /etc/default/grub in your VM as below using your favorite editor: ...
    GRUB_CMDLINE_LINUX="console=ttyS0,115200n8"
    ...

    The run the following command in your VM's shell: update-grub
    Finally, shutdown and restart the VM.

    The following libvirst commands are useful for managing your VM.
    You can check the status of your VM using: virsh list
    You can connect to the console of your VM using: virsh console guest0
    You can start your VM using: virsh start guest0
    You can shutdown your VM using: virsh shutdown guest0
    You can forced-kill your VM using: virsh destroy guest0
    HINT: You can use "ctrl + ]" to exit from the VM's console.

  2. (50 pts.) Write a hypercall in KVM
    A hypercall is a way for a guest OS to make a call to the hypervisor, in some ways similar to how a system call allows an application to make a call to the OS. We are asking you to write a hypercall to become familiar with how they work and the codebase for KVM. For this part of the assignment, you will also set up your VM so you can use it for your hypercall development.

    The hypercall you write should take one argument and output the information about virtual CPU in KVM. Modern architectures provide special support for virtualization and add a privileged instruction for hypercalls, which you will use in this part of the assignment. Rather than using the default kernel that comes with Ubuntu 14.04.5, you should download and build a more recent version of the Linux kernel, kernel version 4.10, and install that in your VM. It will be this updated version of the kernel/KVM source that you will modify to implement your hypercall.

    The prototype for your hypercall will be the following. The argument vcpu_id contains the CPU id in your VM.
    int vcpu_info(int vcpu_id);

    Your hypervisor should output the following information to ftrace in your KVM host based on the input VCPU ID provided by the VM.
    pid: the corresponding PID of the VCPU thread in KVM host
    gp_regs: the values of the general purpose registers for the virtual CPU
    num_exits: number of vm exits from the VCPU
    The followings are other features you need to implement in your hypercall.
    HINT: Intel's (most likely CPU you have in your personal computer) virtualization extension (VMX) provides a "privileged" instruction called vmcall for hypercall. When the VM executes the instruction, it traps from non-root operation to root operation so the hypervisor can then handle the hypercall. You should search the Linux source and look at how vmcall is handled by KVM.

    HINT: You can use the Linux Cross-Reference (LXR) to investigate different hypercalls already defined. You should use the same calling convention as the other hypercalls.

    HINT: You should look at how the structure kvm_vcpu is defined and used in the source code.

    Compiling / Updating Linux/KVM host:

    You can download the Linux v4.10 mainline kernel source via: git clone https://github.com/torvalds/linux.git -b v4.10
    In your Linux source directory, first copy the ubuntu config to your kernel source. Note that this only has to be done once.
    cp /boot/config-#YOUR_UBUNTU_VERSION .config
    yes "" | make oldconfig

    Install proper packages for the kernel compile.
    sudo apt-get install libssl-dev bc libncurses-dev

    Run the following command to configure your kernel.
    make menuconfig

    To enable ftrace in your KVM host, go to Kernel Hacking and select Tracers. Then enter Tracers, select both Kernel Function Tracer and Kernel Function Graph Tracer. Finally, save the config and exit.

    After you add your hypercall to your Linux/KVM source, use the following command to compile the kernel.
    make -j8

    Finally, use the following command to install the new kernel to your system. The new kernel will be loaded in the next boot. make modules_install
    make install

  3. (30 pts.) Test your new hypercall
    To test your new hypercall, you will install your modifications in the guest kernel running in your nested VM for testing. As mentioned earlier, the instruction that initiates a hypercall is a privileged instruction and cannot be executed in user mode. To test the hypercall from user space, you need to add a new system call in your guest kernel which in turn calls to the vcpu_info. The prototype of the system call can be defined as the following.
    int sys_vcpu_info(void);
    Your sys_vcpu_info system call should enumerate each of the online CPU, and pass the CPU id to KVM via the vcpu_info hypercall.

    You should write a simple C program which calls to sys_vcpu_info from user space. You can get the see the output from the hypercall in your KVM host using the following command:
    cat /sys/kernel/debug/tracing/trace

    NOTE: Although system calls are generally accessed through a library (libc), your test program should access your system call directly. This is accomplished by utilizing the general purpose syscall(2) system call (consult its man page for more details).

    Compiling / Updating Linux kernel for your guest:
    You first have to clone the same mainline Linux v4.10 kernel source to your VM. To compile and update the kernel for your VM, you can use the same commands mentioned earlier for updating the kernel in KVM/Linux host.