Qemu Networking Investigation - Details

Assuming you have a virtual machine on a disk image that you want to run in qemu such that:

• the target can access the network at large from inside the target
• you can access the target's network from the host
• you don't want to assign a static IP within the image itself but you want to be able to flexibly set up the image on any sort of network (class A, class B, class C) with any IP address without having to make changes to the image itself

here are some notes on how to go about accomplishing these goals.

• create the disk file

$ qemu-img create -f qcow2 myimage.qcow2 200G

• install your favourite distribution into the disk file

$ qemu-system-x86_64 \
        -enable-kvm \
        -smp 2 \
        -cpu host \
        -m 4096 \
        -drive file=/.../myimage.qcow2,if=virtio \
        -net nic,model=virtio \
        -net user \
        -cdrom /.../openSUSE-Tumbleweed-NET-x86_64-Snapshot20150126-Media.iso

• run the image

$ qemu-system-x86_64 \
        -enable-kvm \
        -cpu host \
        -smp 6 \
        -m 4096 \
        -net nic,model=virtio \
        -net user \
        -drive file=/.../myimage.qcow2 \
        -nographic

• tweak it to your preferences

edit /etc/default/grub, edit GRUB_CMDLINE_LINUX_DEFAULT:
          to add
                " console=ttyS0,115200"
          to remove
                "splash=silent quiet"

# grub2-mkconfig -o /boot/grub2/grub.cfg

configure /tmp for tmpfs, add the following to /etc/fstab:
          none /tmp tmpfs defaults,noatime 0 0

• take a snapshot

$ qemu-img snapshot -c afterInstallAndConfig myimage.qcow2

• create a configuration file named CONFIG

checkenv() {
        if [ -z "${!1}" ]; then
                echo "required env var '$1' not defined"
                exit 1
        fi
}

findcmd() {
        which $1 > /dev/null 2>&1
        if [ $? -ne 0 ]; then
                echo "can't find required binary: '$1'"
                exit 1
        fi
}

MACADDR=DE:AD:BE:EF:00:01
USERID=trevor
GROUPID=users
IPBASE=192.168.8.
HOSTIP=${IPBASE}1

• create a "super script" called start_vm

#!/bin/bash

if [ $# -ne 1 ]; then
        echo "usage: $(basename $0) <image>"
        exit 1
fi

source CONFIG
checkenv MACADDR
checkenv USERID
checkenv GROUPID
checkenv IPBASE

THISDIR=$(pwd)
IMAGE=$1

TAPDEV=$(sudo $THISDIR/qemu-ifup)
if [ $? -ne 0 ]; then
        echo "qemu-ifup failed"
        exit 1
fi
echo "tap device: $TAPDEV"

qemu-system-x86_64 \
        -enable-kvm \
        -cpu host \
        -smp sockets=1,cores=2,threads=2 \
        -m 4096 \
        -drive file=$IMAGE,if=virtio \
        -net nic,model=virtio,macaddr=$MACADDR \
        -net tap,ifname=$TAPDEV,script=no,downscript=no \
        -nographic

sudo $THISDIR/qemu-ifdown $TAPDEV

• create an "up" script called qemu-ifup

#!/bin/bash

source CONFIG

usage() {
        echo "sudo $(basename $0)"
}

checkenv USERID
checkenv GROUPID
checkenv IPBASE
checkenv HOSTIP

findcmd tunctl
findcmd ip
findcmd iptables
findcmd dnsmasq


if [ $EUID -ne 0 ]; then
        echo "Error: This script must be run with root privileges"
        exit 1
fi

if [ $# -ne 0 ]; then
        usage
        exit 1
fi

TAPDEV=$(tunctl -b -u $USERID -g $GROUPID 2>&1)
STATUS=$?
if [ $STATUS -ne 0 ]; then
        echo "tunctl failed:"
        exit 1
fi

ip addr add $HOSTIP/32 broadcast ${IPBASE}255 dev $TAPDEV
ip link set dev $TAPDEV up
ip route add ${IPBASE}0/24 dev $TAPDEV

# setup NAT for tap$n interface to have internet access in QEMU
iptables -t nat -A POSTROUTING -j MASQUERADE -s ${IPBASE}0/24
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/$TAPDEV/proxy_arp
iptables -P FORWARD ACCEPT

# startup dnsmasq
dnsmasq \
        --strict-order \
        --except-interface=lo \
        --interface=$TAPDEV \
        --listen-address=$HOSTIP \
        --bind-interfaces \
        -d \
        -q \
        --dhcp-range=${TAPDEV},${IPBASE}5,${IPBASE}20,255.255.255.0,${IPBASE}255 \
        --conf-file="" \
        > dnsmasq.log 2>&1 &
echo $! > dnsmasq.pid

echo $TAPDEV
exit 0

• create a "down" script called qemu-ifdown

#!/bin/bash

source CONFIG
checkenv IPBASE
findcmd tunctl
findcmd iptables

usage() {
        echo "sudo $(basename $0) <tap-dev>"
}

if [ $EUID -ne 0 ]; then
        echo "Error: This script (runqemu-ifdown) must be run with root privileges"
        exit 1
fi

if [ $# -ne 1 ]; then
        usage
        exit 1
fi

TAPDEV=$1
tunctl -d $TAPDEV

# cleanup the remaining iptables rules
iptables -t nat -D POSTROUTING -j MASQUERADE -s ${IPBASE}0/24

# kill dnsmasq
if [ -f dnsmasq.pid ]; then
        kill $(cat dnsmasq.pid)
        rm -f dnsmasq.pid
        rm -f dnsmasq.log
fi

• ??
• profit!

 This setup is driven by the CONFIG file and the start_vm script. Edit CONFIG to your liking, then run start_vm specifying your image file.

Qemu Networking Investigation - On The Shoulders Of Giants

If you create a virtual disk, install your favourite distribution on it, and want to run it with qemu without having to hard-wire a static IP inside the image itself then this post provides you with some things to consider.

In previous posts I looked at how The Yocto Project is able to accomplish these goals. It does so by being able to supply a kernel and a kernel append, separate from the filesystem image itself. An IP address can be provided in the kernel append, which allows you to specify a static IP... in a flexible way :-)

When you create a virtual machine of, say, your favourite distribution, the installation puts the VM's kernel inside the filesystem, which means it's not available outside the image for you to use the kernel append trick to specify an IP address.

A flexible way for a virtual machine to handle networking is to use dhcp. This requires a dhcp server to be available somewhere on the network which can assign an IP to whoever asks. dnsmasq provides dhcp server capabilities and provides lots of configurability. As I noted in previous blog posts, since we already have a mechanism for running up and down networking-related scripts before and after running a qemu image, there is no reason why we can't start and stop an appropriately configured dhcp server to satisfy our virtual machine. We just want to make sure the dhcp server is only listening on the virtual interface and not interfering with any other interface, or with any other dhcp server running on the network.

Building on the steps the scripts from The Yocto Project use, we can simply add a line to startup dnsmasq and configure it so that it only listens to the virtual tap device which has been setup (by the rest of the up script) for the virtual machine we are bringing up:

dnsmasq \
        --strict-order \
        --except-interface=lo \
        --interface=$TAPDEV \
        --listen-address=$HOSTIP \
        --bind-interfaces \
        -d \
        -q \
        --dhcp-range=${TAPDEV},${IPBASE}5,${IPBASE}20,255.255.255.0,${IPBASE}255 \
        --conf-file="" \
        > dnsmasq.log 2>&1 &

The really hard part was the --dhcp-range option. Originally I had only specified

--dhcp-range=${TAPDEV},${IPBASE}5,${IPBASE}20

which kept leading to the following error message:

dnsmasq-dhcp no address range available for dhcp request via tap0

Wow, how frustrating was that?! As it turns out, if you don't explicitly specify the mask, it assumes 255.255.255.255, leaving absolutely no space whatsoever from which to generate an address!

This is really the only tweak to The Yocto Project's qemu scripts you need in order to run an image in qemu without having to hard-code an IP address and allowing you to access services running on the target from the host.

Before your image is started:

• a virtual tap interface is created
• various ifconfig-fu is used to setup the host's side of this interface and to setup the correct routing
• iptables is used to enable NAT on the VM's network interface
• dnsmasq is run which can provide an IP of your choosing to the VM (provided your image is configured to use dhcp to obtain its IP address)

You can then run multiple images on various virtual tap interfaces without the worry of an IP collision (provided you change the dhcp server range for each invocation) and you never need to make any changes to the image itself in order to accomplish this goal.

Qemu Networking Investigation - The Yocto Way

qemu provides many options for setting up lots of interesting things and for setting them up in lots of interesting ways. One such set of options allows you to configure how the networking is going to work between the host and the target.

Due to the fact there are so many different ways to do networking with a virtual machine, qemu simply provides the ability to run an "up" script and a "down" script. These scripts, which you can specify, are just hooks which allow you to do some funky networking fu just before qemu starts the image (up) and right after the image terminates (down).

If you do supply these scripts, and something within those scripts requires root privileges (which is most likely the case), then the entire qemu cmdline needs to be run as root. One of the clever things The Yocto Project's qemu scripts do is to not supply these scripts as part of the qemu invocation. It runs them, instead, just before and just after running the qemu command itself.

A "super script" is provided with The Yocto Project which runs the networking "up" script, then runs qemu (with all its options), then runs the networking "down" script. By doing it this way, the "super script" can simply run the up and down scripts with sudo, which gives an opportunity for the user to provide the required password. In this way the qemu image itself isn't run with root privileges, but simply with the privileges of the invoking user (which I assume is a regular, unprivileged user). In this way only the parts which really need root privileges (i.e. the networking up and down scripts) are run with root privileges, everything else is run as a regular user.

The Yocto Project's qemu "super script" is runqemu. runqemu very quickly runs runqemu-internal. It is runqemu-internal which invokes the networking up and down scripts. Within The Yocto Project the networking up script is runqemu-ifup and the networking down script is runqemu-ifdown.

I won't go into all the gory details here (you're welcome to look at The Yocto Project's scripts yourself) but at a high level the up script:

• uses tunctl to setup a new, virtual tap interface
• uses various ifconfig-fu to setup the host's side of the virtual interface as well as manipulate the host's routing tables
• fiddles with the host's firewall to enable NATing for the target

The down script reverses the work of the up script to leave the host as it was before the virtual machine was started.

In order to assign an IP address to the VM, The Yocto Project builds the kernel as a separate entity, invokes qemu specifying the kernel, and provides a kernel cmdline append specifying the IP address to use for the target.

qemu ... --kernel <path/to/kernel> ... --append "ip=192.168.7.$n2::192.168.7.$n1:255.255.255.0"

But what if you have used qemu to install a version of your favourite distribution? In this case the kernel is in the image itself, and doesn't exist outside of it. You could, theoretically, copy the kernel from inside the image to the host, but this gets messy if/when the distribution updates the kernel.

If your kernel is not outside your image then you can't use The Yocto Project's --append trick to specify an IP address for your virtual machine.

You could, during the course of installing and setting up your virtual machine, use your virtual machine distribution's networking tools to configure a static IP, but this means the networking up and down scripts would have to be tailored to match the virtual machine's static settings. This is less flexible.

My solution is to come in my next post.

Qemu Networking Investigation - Introduction

The Yocto Project's qemu integration is quite fascinating. It allows you to build an image targetting a qemu machine, then run it with their runqemu command. Various Yocto magic comes together to run the image under qemu, including getting the networking right so you can access the target from your host.

If all you need to do with a qemu image, networking-wise, is to reach out and do something over tcp (e.g. surf the web, install updates, etc) then qemu's "-net user" mode is more than adequate. One of the great features of the "-net user" mode is that it doesn't require root permissions.

If, however, instead of initiating the connection from inside the image, you want to initiate the connection from (for example) the host to connect to something running on the qemu target (e.g. you want to ssh from your host to the target), then you'll need some more qemu networking magic. As I said, The Yocto Project's qemu-fu fully handles this scenario for you. All you need to do is supply a password for sudo!

However, there are a couple caveats. The networking that The Yocto Project sets up for you supplies a static IP to the image as it is being spun up. This is done by providing a kernel cmdline append providing the IP address to use. Providing a kernel cmdline append can only be done in qemu if the kernel lives outside the image. If you have created your own image which contains its kernel inside the root filesystem, you can't use this trick to assign an IP address.

This post and a number of follow-up posts titled "Qemu Networking Investigation" explore these issues. The first issue is figuring out how to get a qemu image to acquire an IP address without having to assign it statically inside the image itself when you can't assign it with a kernel cmdline append. The second issue is figuring out how to set up the networking so that you can reach services running inside the image.

Knowing these pieces of information are useful if you're using qemu to run, say, openSUSE or if you want to use The Yocto Project's *.vmdk option.

The Yocto Project: Introducing devtool - updates!

Now that The Yocto Project's devtool is in the main trunk, some of the instructions from a previous post need to be updated. Here is a link to the updated instructions, enjoy!