EFI in QEMU KVM on Fedora 20

Last sprint, I was working on EFI PXE booting support for Foreman. Although I have a EFI-compatible PC in the house, I wanted a stable environment for development and testing. Virtualized, of course.

The best option is KVM/QEMU/libvirt triple which I use for regular Foreman development. I started googling around and spent a day modifying my distribution package. It did not work well. And then I stumbled upon familiar name, Laszlo Ersek from Red Hat, who helped me getting this rolling.


To get EFI working under QEMU, one needs 2.1.1+ version with modified BIOS. Since I already almost destroyed configuration of the QEMU from Fedora 20 (had to reinstall the packages), I was guided to install from sources:

wget http://wiki.qemu-project.org/download/qemu-2.1.1.tar.bz2
tar bla bla
yum -y install spice-server-devel spice-protocol SDL-devel
./configure --target-list=x86_64-softmmu --enable-spice \
    --prefix=/opt/qemu-2.1.1 --enable-debug --disable-gtk
make install
chown root:root /opt/qemu-2.1.1/libexec/qemu-bridge-helper
chmod u=rwxs,g=rx,o=rx /opt/qemu-2.1.1/libexec/qemu-bridge-helper
echo 'allow virbr0' > /opt/qemu-2.1.1/etc/qemu/bridge.conf

Having stable version installed in a separate tree (which keeps your laptop clean along smile on your face), one needs to download and install BIOS and EFI firmware. This is important step - for PXE booting, latest nightly builds are needed. Fortunately, for Fedora/Red Hat users, there is a repo out there with latest and greatest build. Do not worry, those packages are compatible with Fedora and won’t overwrite/upgrade anything. You can install them next to the official QEMU BIOS:

sudo wget https://www.kraxel.org/repos/firmware.repo -O \
yum install edk2.git-ovmf-x64

This guy installs into /usr/share/edk2.git along with some dependencies which are also compiled from git (SeaBIOS for legacy EFI mode, NIC drivers and so on).

Now, the hardest part. Since libvirt currently lacks UEFI support, we need to work directly with QEMU. I’d like to thank Laszlo for constructing me this script which sets up things correctly:


# create fresh variable store
cp $VARSTORE_TEMPLATE /tmp/guest-vars.fd

# run qemu
$QEMU_ROOT/bin/qemu-system-x86_64 \
  -M pc-i440fx-2.1 \
  -enable-kvm \
  -m 900 \
  -drive unit=0,if=pflash,format=raw,readonly,file=$OVMF_BINARY \
  -drive unit=1,if=pflash,format=raw,file=/tmp/guest-vars.fd \
  -global isa-debugcon.iobase=0x402 \
  -debugcon file:/tmp/guest.ovmf.log \
  -monitor stdio \
  -device piix3-usb-uhci -device usb-tablet \
  -netdev bridge,id=net0,br=virbr0,helper=$QEMU_ROOT/libexec/qemu-bridge-helper \
  -device virtio-net-pci,netdev=net0,romfile=,bootindex=0 \
  -device qxl-vga

You can run this script as regular user the only part requiring root access is the networking one (bridge helper) and that has suid bit set. The VM will be connected to the “default” libvirt virtual NAT network (virbr0) where you can set up your PXE environment and start playing with EFI PXE booting.

PXELinux and Foreman

In this second part of my blog entry, I want to share some news about Foreman support. Since Foreman ships with templates set for PXELinux by default, I wanted to keep on this path.

Apparently syslinux 5.x does not have EFI support and syslinux 6.02 did not work for me. So I decided to compile 6.03 from sources which did not work either. Luckily, folks on the IRC channel recommended to use 6.03-pre20 version which worked like a charm:


The following files need to be extracted into tftpboot/efi64 directory:


Also I created relative symlinks (TFTP runs in chroot usually) for configuration and kernels:

boot -> ../boot
pxelinux.cfg -> ../pxelinux.cfg

You can do the same with 32bit arch. But that’s pretty much it!

DHCP daemon configuration is straighforward and the best approach is only to hand over EFI PXELinux to DHCP clients which reports as EFI-compatible. You need something like ( network with Foreman running on .2):

# ... snippet ...

option arch code 93 = unsigned integer 16;

subnet netmask {
    pool {

    option subnet-mask;
    option routers;

    class "pxeclients" {
        match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";

        if option arch = 00:07 {
            filename "efi/syslinux.efi";
        } else if option arch = 00:06 {
            filename "efi32/syslinux.efi";
        } else {
            filename "pxelinux/pxelinux.0";

Similarly you can use Grub, you need to copy EFI binary and create configuration file. This is covered in RHEL6 (1) and RHEL7 (2) documentation.

30 September 2014 | linux | fedora

Tunnel into your libvirt NAT network with SOCKS

My development setup is based on two Fedora boxes: laptop and worstation situated in Red Hat office. Both machines are powerful enough to run few VMs, but to leverage hardware, I wanted to use libvirt virtual networks on both devices. The Xeon server is more capable of running nested KVM virtual appliances like oVirt or RHEV while laptop is good for local testing.

I have two virtual (NAT) libvirt networks called:

  • zzz.lan (server)
  • local.lan (laptop)

My goal was to use them seamlessly during my development. Let’s focus on the local.lan first.

Some time ago I was using libvirt bridges, but since I need to tackle with DHCP/TFTP/DNS services (I work on The Foreman project), NAT was the only viable option here. The configuration is pretty standard, I created new NAT network and disabled DHCP service there because I wanted to run my own DHCP server. You can use “default” one for this purpose.

The key thing is to setup local caching DNS server dnsmasq. In my case, I use Google public DNS servers as main forwarders:

laptop# cat /etc/dnsmasq.d/caching

laptop# cat /etc/resolv.dnsmasq

laptop# cat /etc/resolv.conf
domain redhat.com

laptop# chattr +i /etc/resolv.conf

Having dnsmasq set correctly, now we can add special file that will add all hosts that has been created via libvirt. This is done automatically by libvirt.

laptop# cat /etc/dnsmasq.d/local.lan

Restart (or start and enable) the service.

laptop# systemctl enable dnsmasq
laptop# systemctl start dnsmasq

Tip: When using VPN you can add extra configuration file so all *.redhat.com queries goes to internal DNS servers (these IP addresses are not real). Also we want to use public DNS server for the adress of the VPN hub, otherwise you would need to use IP address for that service.

laptop# cat /etc/dnsmasq.d/redhat.com

Now, everytime you create a new VM locally, you only need to refresh dnsmasq configuration to re-load the default.addnhosts file.

laptop# virt-install --os-variant rhel6 --vcpus 1 --ram 900 \
    --name myhost.local.lan --boot network --nodisk --noautoconsole

This is as easy as sending HUP signal (or you can do service reload) or similar:

laptop# pkill -SIGHUP dnsmasq

That’s it. You can connect to your server easily:

laptop# ssh root@myhost.local.lan

I do more. When you shutdown or restart your VM, libvirt (which uses dnsmasq for DHCP too) can assign different IP, therefore the addnhost entry is not valid anymore. Therefore I created a simple script that preallocates static DHCP entry in libvirt, so restarts won’t hurt anymore. The snippet is something like:

echo "Removing existing DHCP/DNS configuration from libvirt"
netdump="sudo virsh net-dumpxml default"
virsh_dhcp=$($netdump | xmllint --xpath "/network/ip/dhcp/host[@mac='$MAC']" - 2>/dev/null)
virsh_dns=$($netdump | xmllint --xpath "/network/dns/host/hostname[text()='$FQDN']/parent::host" - 2>/dev/null)
sudo virsh net-update default delete ip-dhcp-host --xml "$virsh_dhcp" --live --config 2>/dev/null
sudo virsh net-update default delete dns-host --xml "$virsh_dns" --live --config 2>/dev/null

while true; do
  AIP="192.168.100.$(( ( RANDOM % 250 )  + 2 ))"
  echo "Checking if random IP $AIP is not in use"
  $netdump | xmllint --xpath "/network/ip/dhcp/host[@ip='$AIP']" - &>/dev/null || break

echo "Deploying DHCP/DNS configuration via libvirt for $AIP"
sudo virsh net-update default add-last ip-dhcp-host --xml "<host mac='$MAC' name='$FQDN' ip='$AIP'/>" --live --config
sudo virsh net-update default add-last dns-host --xml "<host ip='$AIP'><hostname>$FQDN</hostname></host>"  --live --configpice,listen= -

This will remove existing entries, generates random IP address, checks if that does not exist and create DHCP (and in addition to that DNS) entries.

Now, this was the easy part. I use the same on my server, but I want to be able to access the zzz.lan VMs from my laptop too. For web access this turns out to be quite easy task. For example in Chrome you only need to created this file:

laptop# cat ~/proxies.pac
function FindProxyForURL(url, host) {
  if (shExpMatch(host, "*.zzz.lan*")) {
    return "SOCKS5 localhost:8890";
  } else {
    return "DIRECT";

Start dynamic tunnel to the libvirt hypervisor (the server):

laptop# cat .ssh/config
Host server
    HostName server.redhat.com
    DynamicForward 8890

laptop# ssh -N server &>/dev/null &

And start Chrome with this configuration:

laptop# google-chrome --proxy-pac-url=file:///home/lzap/proxies.pac

When working on The Foreman development, I also want the application to be able to connect to various services (like oVirt or RHEV compute resources). This turns to be possible too. The key utility is tsocks wrapper which is available in Fedora. We can use the very same ssh dynamic tunnel for that.

laptop# cat /etc/tsocks.conf
local =
server =
server_port = 8890

The only difference in dnsmasq configuration on the server is that I want to open it for incoming DNS requests, because my laptop needs to resolve hosts from zzz.lan:

server# cat /etc/dnsmasq.d/caching

And of course we need to open firewall port:

server# firewall-cmd --add-service=dns --permanent

Now on the laptop, one more change is needed. I need to direct my local dnsmasq daemon to resolve from the server dnsmasq (assuming my server IP address is

laptop# cat /etc/dnsmasq.d/zzz.lan

After reloading dnsmasq, I can finally use tsocks:

laptop# tsocks wget -O - http://ovirt34.zzz.lan

So I can start my application allowing it to connect to zzz.lan NAT network:

laptop# tsocks foreman_application

That’s it. Hope you were inspired.

31 July 2014 | linux | fedora

Track short lived processes with auditd

I had a hard time to find some more information about short lived processes when I was building Foreman SELinux policy for OpenStack Installer (Staypuft). I wanted to see process name and all the parameters for all possible processes spawned on my system. My plan was to write a SystemTap script, but thank to Mirek Grepl and Steve Grubb I learned much easier trick to do that. This assumes you have auditd daemon installed (RHEL/Fedoras are fine):

auditctl -a exit,always -S execve

Then tail your /var/log/audit/audit.log to see messages. Warning: This can flood your system heavily. You will see some decent content there like:

type=CWD msg=audit(1401202071.000:14728):  cwd="/"
type=PATH msg=audit(1401202071.000:14728): item=0 name="/usr/sbin/ps" nametype=UNKNOWN
type=SYSCALL msg=audit(1401202071.000:14729): arch=c000003e syscall=59 success=yes exit=0 a0=7fe10bbe6369 a1=7fe10bbe7580 a2=7fff0b1d3208 a3=0 items=2 ppid=26400 pid=5161 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=875 comm="ps" exe="/bin/ps" subj=unconfined_u:system_r:passenger_t:s0 key=(null)
type=EXECVE msg=audit(1401202071.000:14729): argc=5 a0="ps" a1="-o" a2="pid,ppid,%cpu,rss,vsize,pgid,command" a3="-p" a4="26974,26817"

I was able to tell that mod_passenger is spawning /bin/ps every few seconds. This is very useful when tracking down some processes with long puppet runs. To turn the audit rule down, replace -a option with -d:

auditctl -d exit,always -S execve

You can filter out audit logs, see auditctl manual page. For example I was interested in processes running in passenger_t SELinux domain:

auditctl -a exit,always -S execve -F subj_type=passenger_t

To add rules permanently, edit /etc/audit/audit.rules file and restart auditd daemon.

27 May 2014 | linux | fedora

SystemTap as a system wide strace tool

I needed to find a process that was searching for several file patterns. I could use strace tool but since this one was a Apache2 module, I would need to hack startup scripts and probably create a wrapper script. There was another way of doing that - using SystemTap.

My problem was simple - I wanted to see process name, pid and full absolute path for a filename pattern. The goal was set, with little bit of googling, I was able to write this (not ideal and hacky) SystemTap script:

$ cat syscall_open.stp
#!/usr/bin/env stap
# System-wide strace-like tool for catching file open syscalls.
# Usage: stap syscall_open filename_regexp
global myfilename
probe syscall.open {
  myfilename = filename
probe syscall.open.return {
  if (myfilename =~ @1) {
    printf ("%s(%d) opened %s -> %d\n", execname(), pid(), myfilename, $return)

Before we can use SystemTap, we need to install those packages (this is for RHEL6/CentOS6). Note you do need debug info packages, otherwise this will not work (enable them via subscription-manager or in repo files):

$ yum -y install systemtap systemtap-runtime kernel-debuginfo-`uname -r` \
    kernel-debuginfo-common-`uname -i`-`uname -r` kernel-devel-`uname -r`

Usage is super simple, you can either make it executable or to see more verbose output just do:

$ stap -v syscall_open.stp native_support.so

This is effective for the whole system, after Apache2 httpd server restart and first access, I was able to find why the heck mod_passenger searches on wrong paths:

ruby(4257) opened "/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/local/share/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/local/lib64/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/share/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/lib64/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/share/rubygems/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/share/ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/lib64/ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so" -> -2
ruby(4257) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.rb" -> -2
ruby(4257) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.so" -> -2
ruby(4282) opened "/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/local/share/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/local/lib64/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/share/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/lib64/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/share/rubygems/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/share/ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/lib64/ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so" -> -2
ruby(4282) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.rb" -> -2
ruby(4282) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.so" -> -2

SystemTap is awesome tool. It helps me every week :-)

16 May 2014 | linux | fedora

Virt builder quick provision script

Until now, I was using snap-guest tool together with Foreman to spawn my development/test virtual machines. I have been carefully monitoring (virt-builder)[http://libguestfs.org/virt-builder.1.html] tool in recent Fedoras and it looks like the version from Fedora 20 is finally usable for my scenarios (thanks Rich for fixing some annoying bugs for me).

Therefore I am happy to announce new tool called fvb which stands for Foreman-virt-builder. This little script is based around two little commands virt-builder/virt-install and it has grown to something bigger over last couple of months. The usage is simple:

fvb --install-deps

The above command should install all required dependencies on Red Hat systems. Note Fedora 20+ is required to get this working since this needs some latest and greatest features from virt-builder.

fvb -n my-nightly-foreman -f
fvb -n rc-foreman -- "FOREMAN_REPO=rc" "MYVAR=value"
fvb -n stable-foreman -- "FOREMAN_REPO=releases/1.4"
fvb -n bz981123 -- "KOJI_BUILD=http://mykoji.blah/taskinfo?taskID=97281"
fvb -n my-own-script --script fb-xyz.bats -- BATS_REPOOWNER=lzap BATS_BRANCH=test

You may notice download of centos-6.xz file into ~/.cache/virt-builder. This will only happen once, or when new CentOS minor release is uploaded. The template image is handled by virt-builder project itself.

The script works the following way:

  • Generates MAC address from hypervisor hostname and guest hostname (hash) so it does not change when you provision again.
  • Removes existing guest from libvirt (if --force is given).
  • Generates random IP address.
  • Allocates DHCP and DNS entry with the MAC and IP in libvirt network (“default”) overwriting existing entries. This is compatible with virsh Foreman provider (and also works the similar way).
  • Installs your public SSH key on the image.
  • Creates a new VM image based on CentOS6 using virt-builder.
  • Configures a firstboot script:
    • install git
    • install bats framework
    • install foreman-bats scripts
    • run script provided by --script option (by default fb-install-foreman.bats)
  • Spawns the new image with virt-install (with nested kvm enabled if possible).
  • Refreshes dnsmasq configuration.
  • Waits until DHCP daemon (dnsmasq) returns the IP address.
  • Calls fortune command.

Dnsmasq configuration (optional)

By default dnsmasq is installed on Fedora and Red Hat systems, but the daemon is turned off and it is only used by libvirt. What I usually do is I have it configured as a caching DNS daemon on my laptop. It not only speeds up DNS queries (web browsing is much faster on all connections), but it also allows me to redirect DNS queries for different domains to different DNS servers. For example:

# grep -v ^# /etc/dnsmasq.conf | sort -u

# cat /etc/resolv.conf
domain redhat.com

# cat /etc/resolv.dnsmasq

# cat /etc/dnsmasq.d/local.lan

In the configuration above, you can see that dnsmasq is configured to read nameservers from a separate file (/etc/resolv.dnsmasq) where it has Google DNS public servers and to respond on localhost and to act like a simple DNS caching daemon. The system is configured to take advantage of that.

The last bit is configuration for local.lan domain (default --domain for fvb script). Libvirt keeps host names in this file and since this is in dnsmasq format, we can read it directly.

What does it mean? Well, if you configure everything according to this, once you spawn a VM called “test”, the box named “test.local.lan” will be immediately available, so you can do this:

ssh root@test.local.lan

Or even visit this:


That’s kinda cool, isn’t it?

Notes to redhatters

If you are not a redhatter, skip to the next section. If you want this setup, make sure you have something like this:

# cat /etc/dnsmasq.d/redhat.com

This configuration adds separate handling for redhat.com domain which goes to internal servers (these are dummy values). Also note that I want to use public DNS for vpn concentrators and web site (so I am able to connect initially and also the site works when not connected).

This way you can enjoy advantage of fvb script, fast public DNS lookups and fast internal lookups.

What next?

The fvb script is just a little prototype. It lives here and I might move it into a separate repository eventually.

The script can be used for anything, you can provide different distribution (use virt-builder -l to list all the base images available to you) and provide any command with --script that should be executed. For example to deploy Katello Foreman plugin (which we sometimes call “foretello”), do:

fvb -n knightly --script 'git clone https://github.com/Katello/katello-deploy.git && cd katello-deploy && ./setup.rb centos6'

The script accepts any shell variables and values after the -- option, so you are able to pass anything to your own scripts when needed.

Please send me patches and comments. If you like it, I can rename the project and generalize it a bit so it is useful for others as well.

Nested virtualization

I have one extra trick for you. If you do

echo “options kvm-intel nested=1″ | sudo tee /etc/modprobe.d/kvm-intel.conf

on the host machine and restart, you should be able to use KVM inside KVM. The fvb script spawns machines with required nesting options turned on, so you can configure Foreman/Katello for provisioning there.


usage: fvb options -- VARIABLE1=value1 ...

Script for building images with Foreman preinstalled using virt-builder and
bats suite on local libvirt host. The script also modifies dnsmasq
configuration and sends SIGHUP to refresh so the hostname is instantly
available. The root password is "redhat" but you should have your public
ssh key installed by default.

  --help | -h
        Show this message

  --name | -n
        Image name, default: nightly

  --distro | -d
        Distribution base image for virt-builder, default: centos-6

  --force | -f
        Overwrite target image and VM if they exist

        Do not use sudo for building and running VM - you will need to
        set --image-dir accordingly too when running under regular user.

  --image-dir [path]
        Target images path
        Default: /var/lib/libvirt/images

  --domain [domain]
        Domain suffix like "mycompany.com", default: local.lan

  --subnet [subnet]
        Subnet to be used for writing DHCP/DNS configuration
        Default: 192.168.122 (note there is no suffix or period)

  --pub-key [key]
        Install this public ssh key
        Default: /home/you/.ssh/id_rsa.pub

  --script [name]
        BATS script to execute during first boot (can be any shell command)
        Default: fb-install-foreman.bats

        Install required dependencies

Next time!

15 May 2014 | linux | fedora

twitter.com linkedin.com
google.com/+ facebook.com
flickr.com youtube.com