eRacks Systems Tech Blog

Open Source Experts Since 1999

Joe Wolff wearing his KVM Developer Forum 2008 shirt from Napa, California
Still in the drawer: my shirt from the 2008 KVM Developer Forum in Napa.

In our last post on virtualization I promised the story of how we came to KVM, the Kernel-based Virtual Machine. Here it is. It starts in a resort outside Tucson in August 2007, at the first KVM Forum, and it ends with the small piece of software our own servers still run on.

Tucson, August 2007

The first KVM Forum ran from August 29 to 31, 2007, at the Loews Ventana Canyon in Tucson. KVM had been in the mainline Linux kernel for about six months by then: it arrived in version 2.6.20 in February, as a driver for the hardware virtualization features in Intel and AMD processors. Avi Kivity, who started it at a small company called Qumranet, opened the forum with a talk titled “KVM, One Year On”.

I went expecting to learn more about the Linux kernel and about virtualization. What I found was a technology that was going to change where virtualization went. The default answer in 2007 was Xen, a separate hypervisor that ran underneath Linux. By my count at the time it was roughly ten times the code, and its host side lived outside the kernel. KVM took the opposite approach: Linux itself becomes the hypervisor. The scheduler, the memory manager and the device drivers are the ones the kernel already has, so every improvement to Linux is an improvement to the hypervisor for free. Sitting in that room, it was obvious to me that this would replace the virtualization engines of the day. It largely has.

Two moments that stuck

The first was meeting Avi Kivity and having drinks with the whole Qumranet crew, including Benny Schnaider, the CEO, who closed the forum with the final keynote. I was so impressed with Avi: his intuitive grasp of everything he was doing, and the mental bandwidth he had for all of it at once.

The second was in a session with Rusty Russell, the kernel developer who wrote ipchains and then iptables, the Linux packet filter behind countless firewalls. His talk in Tucson was about lguest, his deliberately minimal hypervisor, but the remark I remember was about iptables. He said, more or less, that iptables was never intended as an end-user product. It was a toolkit for building one. That changed how I looked at the whole stack. The kernel gives you good tools. The product layer on top should be small, opinionated and yours, and it should not try to re-wrap every tool underneath it.

Napa, June 2008

KVM Developer Forum 2008 shirt: Tux in a wine glass, Napa California
Tux in a wine glass: the Napa forum shirt.

The second forum moved to the Napa Valley, June 11 to 13, 2008, and the agenda showed how fast things were moving: Mac OS X running in KVM, memory sharing between virtual machines, passing physical devices straight through to a guest. Three months later, in September 2008, Red Hat bought Qumranet for $107 million. KVM was no longer an interesting experiment. It was the future of Linux virtualization, and by then we were already running it.

What libvirt got wrong

The obvious way to manage KVM then, and still the way most guides assume, was libvirt. It did not work for us. In 2008 it was full of stubs and unimplemented features, and it rewrote the surface area for getting anything done, so that using it was more work than going directly to the QEMU commands underneath. QEMU is the machine emulator that KVM accelerates; it already had a perfectly good command line and monitor.

Then there was the XML. Every machine became an XML document, and XML as a configuration format is a disaster of invalid states: documents that parse but do not mean anything, settings that conflict, and no good way to see what the machine will actually do until you run it. The industry has largely moved on to simpler configuration languages since. To be fair, libvirt has picked up a few redeeming features over the years. In my view it is still not worth it, which is Rusty’s point exactly: a tool that tries to be the product ends up harder to use than the toolkit it wraps.

The first eVirt

So we built the opposite. The first eVirt did one thing: it launched virtual machines from a simple configuration file with well chosen defaults, talking to QEMU directly. It took a tiny fraction of the code of anything built on libvirt, and we have used it in production ever since. The current version, the fourth, keeps the same idea: a short YAML file per machine (CPU, memory, disks, bridge, fixed address), an Ubuntu cloud image seeded with cloud-init, and one command to build or rebuild it. Our own web, mail and lab servers are defined that way, and I described it in more detail in the last post.

What it taught us

Nearly two decades later, the lessons from those two forums are still how we build. Put the hard parts in the kernel, where they get the most eyes and the longest support. Keep the layer on top thin enough that you can read it, replace it, or skip it. And treat the management tools as a choice, not a commitment: a virtual machine disk is a file, and a good host lets you change your mind.

That is why our default virtualization host is still Ubuntu Server with KVM, and why we install whatever management layer a customer asks for: libvirt with Cockpit or virt-manager, Proxmox, Incus, IncusOS or XCP-ng. If you would rather run what we run, we install eVirt on a consulting basis.

Planning a virtualization build, or moving off a platform that has started sending license invoices? Write to us at info@eracks.com, or start with a quote request.

October 8th, 2026

Posted In: News, servers, Technology

Tags: , , , , , ,

Leave a Comment