Qemu Networking Investigation - Introduction

The Yocto Project's qemu integration is quite fascinating. It allows you to build an image targetting a qemu machine, then run it with their runqemu command. Various Yocto magic comes together to run the image under qemu, including getting the networking right so you can access the target from your host.

If all you need to do with a qemu image, networking-wise, is to reach out and do something over tcp (e.g. surf the web, install updates, etc) then qemu's "-net user" mode is more than adequate. One of the great features of the "-net user" mode is that it doesn't require root permissions.

If, however, instead of initiating the connection from inside the image, you want to initiate the connection from (for example) the host to connect to something running on the qemu target (e.g. you want to ssh from your host to the target), then you'll need some more qemu networking magic. As I said, The Yocto Project's qemu-fu fully handles this scenario for you. All you need to do is supply a password for sudo!

However, there are a couple caveats. The networking that The Yocto Project sets up for you supplies a static IP to the image as it is being spun up. This is done by providing a kernel cmdline append providing the IP address to use. Providing a kernel cmdline append can only be done in qemu if the kernel lives outside the image. If you have created your own image which contains its kernel inside the root filesystem, you can't use this trick to assign an IP address.

This post and a number of follow-up posts titled "Qemu Networking Investigation" explore these issues. The first issue is figuring out how to get a qemu image to acquire an IP address without having to assign it statically inside the image itself when you can't assign it with a kernel cmdline append. The second issue is figuring out how to set up the networking so that you can reach services running inside the image.

Knowing these pieces of information are useful if you're using qemu to run, say, openSUSE or if you want to use The Yocto Project's *.vmdk option.

The Yocto Project: Introducing devtool - updates!

Now that The Yocto Project's devtool is in the main trunk, some of the instructions from a previous post need to be updated. Here is a link to the updated instructions, enjoy!

Generating Yocto Documents in openSUSE 13.2

If you want to keep up-to-date with the latest Yocto Project documentation, you should be cloning git://git.yoctoproject.org/yocto-docs.

Once cloned, you can build (for example) a PDF version of the Developer's Manual by:

$ pushd documentation
$ make pdf DOC=ref-manual
$ popd

However, for some reason when I try this on my openSUSE 13.2 system I get:

cd dev-manual; ../tools/poky-docbook-to-pdf dev-manual.xml ../template; cd ..
warning: failed to load external entity "http://docbook.sourceforge.net/release/xsl/current/template/titlepage.xsl"
cannot parse http://docbook.sourceforge.net/release/xsl/current/template/titlepage.xsl

A quick work-around for this is to (temporarily) uninstall the docbook-xsl-stylesheet package:

#  rpm -e --nodeps docbook-xsl-stylesheets

Once done building the Yocto Project documentation simply re-install the package:

# zypper install docbook-xsl-stylesheets

The Yocto Project: Introducing devtool

Paul Eggleton, and a group of his co-workers at the Intel Open Source Technology Centre, have been working on a new tool to help developers and build engineers work better and more efficiently together. Integrating The Yocto Project into a developer's workflow has traditionally been a source of pain, and has held back some projects from adopting The Yocto Project and its tools.

I have written a quick tutorial on this new tool (called devtool) which you can find here. Enjoy!

FSOSS 2014 Report - The 2020 Datacentre

Chris Tyler gave a very thought-provoking and amusing talk at this year's FSOSS entitled The 2020 Datacentre.

For this talk Chris pretended (I think) to be a time traveler who had just returned from the year 2020 and was giving this talk to enlighten us about what had happened to a typical datacentre (and computing in general) during the intervening years.

What made the talk so humourous was how both Chris and the audience (!) stayed "in character" and used the past, present, and future tenses as though Chris really were from 2020. Even when asking questions, audience members would phrase a question as "...at what point did <technology> become cheap..." instead of "...when do you think <technology> will become cheap...".

But for me all of this "staying in character" also made the talk very thought-provoking. It was very interesting to pretend as though someone did live in a time when (as one example) NVDIMMs are ubiquitous. If NVDIMMs are everywhere, do people still use (traditional) hard disks? If not, what do you see when you type "ls"? Do you see files, or memory? How does one see memory as the output of an "ls" command?

Thanks Chris, for a very interesting talk!

FSOSS 2014 Report - Project Management for Open Source Development

I had the pleasure of being able to attend a couple sessions from this year's FSOSS 2014 Symposium. One such session was a presentation by David Zinman titled: Project Management for Open Source Development.

It may seem strange, for some, to put "project management" and "open source development" together in the same sentence. But as more and more companies adopt open source, there will be more interest in adopting traditional management techniques with respect to open source work (maybe not for the open source projects themselves, but certainly for a company's involvement with such projects). In other words: where there are schedules, so shall there be management :-)

David has been working for many years as a project manager at Linaro. Linaro is a company which employes engineers (who develop software in the open) and has adopted an agile project management model with which to manage said engineers. As a consequence, David has much personal experience on which to draw for a presentation such as this.

It seemed to me as though the main point of David's talk is that the success of any project is going to hinge on communication and collaboration; regardless of whether a project is closed or worked on in the open. In addition to presenting his slides, David shared his knowledge with the audience by way of anecdotes and even an exercise to emphasize the points he was making in his talk.

Overall David's talk went very well and there ended up being more questions from the inquisitive audience than could be answered during the session.

Integrate valgrind with your Testing

When testing your development work (automatically, preferably), it would be nice if the case of "forgetting to free memory" (as detected by valgrind) could be reported as a failure. To be honest, it strikes me as odd that valgrind doesn't return some sort of error status by default when it detects memory which was not freed by the developer.

Given the following code (which has an obvious memory leak, but returns a good status):

/*
 * Copyright (C) 2014  Trevor Woerner <trevor.woerner@linaro.org>
 */ 

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h> 

int
main (void)
{
        char *ptr; 

        ptr = malloc(sizeof(*ptr) * 50);
        if (ptr == NULL) {
                perror ("malloc()");
                exit(EXIT_FAILURE);
        } 

        strcpy(ptr, "hello");
        printf("ptr: %s\n", ptr); 

        return 0;
}

Running valgrind against it:

$ valgrind ./memleak
==31562== Memcheck, a memory error detector
==31562== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==31562== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==31562== Command: ./memleak
==31562==
ptr: hello
==31562==
==31562== HEAP SUMMARY:
==31562==     in use at exit: 50 bytes in 1 blocks
==31562==   total heap usage: 1 allocs, 0 frees, 50 bytes allocated
==31562==
==31562== LEAK SUMMARY:
==31562==    definitely lost: 50 bytes in 1 blocks
==31562==    indirectly lost: 0 bytes in 0 blocks
==31562==      possibly lost: 0 bytes in 0 blocks
==31562==    still reachable: 0 bytes in 0 blocks
==31562==         suppressed: 0 bytes in 0 blocks
==31562== Rerun with --leak-check=full to see details of leaked memory
==31562==
==31562== For counts of detected and suppressed errors, rerun with: -v
==31562== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) 

$ echo $?
0

By default valgrind always returns the return value of the application under test; if the app returns zero, valgrind will return zero. The first step toward getting valgrind to return something other than what the test application returns is to define what it should return if it detects an error:

--error-exitcode=<number>

By itself this is not enough. In addition you also need to explicitly ask valgrind to perform a leak check:

$ valgrind --error-exitcode=22 --leak-check=yes ./memleak
==32424== Memcheck, a memory error detector
==32424== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==32424== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==32424== Command: ./memleak
==32424==
ptr: hello
==32424==
==32424== HEAP SUMMARY:
==32424==     in use at exit: 50 bytes in 1 blocks
==32424==   total heap usage: 1 allocs, 0 frees, 50 bytes allocated
==32424==
==32424== 50 bytes in 1 blocks are definitely lost in loss record 1 of 1
==32424==    at 0x4C280CD: malloc (vg_replace_malloc.c:292)
==32424==    by 0x400651: main (in /home/trevor/devel/code/doodles/memleak/memleak)
==32424==
==32424== LEAK SUMMARY:
==32424==    definitely lost: 50 bytes in 1 blocks
==32424==    indirectly lost: 0 bytes in 0 blocks
==32424==      possibly lost: 0 bytes in 0 blocks
==32424==    still reachable: 0 bytes in 0 blocks
==32424==         suppressed: 0 bytes in 0 blocks
==32424==
==32424== For counts of detected and suppressed errors, rerun with: -v
==32424== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) 

$ echo $?
22

With these options, valgrind can be integrated into an automated test framework to provide failures if someone forgets to explicitly free allocated memory. Being able to specify the value valgrind will return in this case makes it easy to differentiate between cases where the app fails and cases where valgrind detects an issue.