Tunnel into your libvirt NAT network with SOCKS

My development setup is based on two Fedora boxes: laptop and worstation situated in Red Hat office. Both machines are powerful enough to run few VMs, but to leverage hardware, I wanted to use libvirt virtual networks on both devices. The Xeon server is more capable of running nested KVM virtual appliances like oVirt or RHEV while laptop is good for local testing.

I have two virtual (NAT) libvirt networks called:

  • zzz.lan (server)
  • local.lan (laptop)

My goal was to use them seamlessly during my development. Let’s focus on the local.lan first.

Some time ago I was using libvirt bridges, but since I need to tackle with DHCP/TFTP/DNS services (I work on The Foreman project), NAT was the only viable option here. The configuration is pretty standard, I created new NAT network and disabled DHCP service there because I wanted to run my own DHCP server. You can use “default” one for this purpose.

The key thing is to setup local caching DNS server dnsmasq. In my case, I use Google public DNS servers as main forwarders:

laptop# cat /etc/dnsmasq.d/caching
bind-interfaces
listen-address=127.0.0.1
resolv-file=/etc/resolv.dnsmasq

laptop# cat /etc/resolv.dnsmasq
nameserver 8.8.8.8
nameserver 8.8.4.4

laptop# cat /etc/resolv.conf
nameserver 127.0.0.1
domain redhat.com

laptop# chattr +i /etc/resolv.conf

Having dnsmasq set correctly, now we can add special file that will add all hosts that has been created via libvirt. This is done automatically by libvirt.

laptop# cat /etc/dnsmasq.d/local.lan
addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts

Restart (or start and enable) the service.

laptop# systemctl enable dnsmasq
laptop# systemctl start dnsmasq

Tip: When using VPN you can add extra configuration file so all *.redhat.com queries goes to internal DNS servers (these IP addresses are not real). Also we want to use public DNS server for the adress of the VPN hub, otherwise you would need to use IP address for that service.

laptop# cat /etc/dnsmasq.d/redhat.com
server=/redhat.com/10.20.30.40
server=/redhat.com/10.20.30.50
server=/vpn-hub1.redhat.com/8.8.8.8

Now, everytime you create a new VM locally, you only need to refresh dnsmasq configuration to re-load the default.addnhosts file.

laptop# virt-install --os-variant rhel6 --vcpus 1 --ram 900 \
    --name myhost.local.lan --boot network --nodisk --noautoconsole

This is as easy as sending HUP signal (or you can do service reload) or similar:

laptop# pkill -SIGHUP dnsmasq

That’s it. You can connect to your server easily:

laptop# ssh root@myhost.local.lan

I do more. When you shutdown or restart your VM, libvirt (which uses dnsmasq for DHCP too) can assign different IP, therefore the addnhost entry is not valid anymore. Therefore I created a simple script that preallocates static DHCP entry in libvirt, so restarts won’t hurt anymore. The snippet is something like:

echo "Removing existing DHCP/DNS configuration from libvirt"
netdump="sudo virsh net-dumpxml default"
virsh_dhcp=$($netdump | xmllint --xpath "/network/ip/dhcp/host[@mac='$MAC']" - 2>/dev/null)
virsh_dns=$($netdump | xmllint --xpath "/network/dns/host/hostname[text()='$FQDN']/parent::host" - 2>/dev/null)
sudo virsh net-update default delete ip-dhcp-host --xml "$virsh_dhcp" --live --config 2>/dev/null
sudo virsh net-update default delete dns-host --xml "$virsh_dns" --live --config 2>/dev/null

while true; do
  AIP="192.168.100.$(( ( RANDOM % 250 )  + 2 ))"
  echo "Checking if random IP $AIP is not in use"
  $netdump | xmllint --xpath "/network/ip/dhcp/host[@ip='$AIP']" - &>/dev/null || break
done

echo "Deploying DHCP/DNS configuration via libvirt for $AIP"
sudo virsh net-update default add-last ip-dhcp-host --xml "<host mac='$MAC' name='$FQDN' ip='$AIP'/>" --live --config
sudo virsh net-update default add-last dns-host --xml "<host ip='$AIP'><hostname>$FQDN</hostname></host>"  --live --configpice,listen=0.0.0.0 -

This will remove existing entries, generates random IP address, checks if that does not exist and create DHCP (and in addition to that DNS) entries.

Now, this was the easy part. I use the same on my server, but I want to be able to access the zzz.lan VMs from my laptop too. For web access this turns out to be quite easy task. For example in Chrome you only need to created this file:

laptop# cat ~/proxies.pac
function FindProxyForURL(url, host) {
  if (shExpMatch(host, "*.zzz.lan*")) {
    return "SOCKS5 localhost:8890";
  } else {
    return "DIRECT";
  }
}

Start dynamic tunnel to the libvirt hypervisor (the server):

laptop# cat .ssh/config
Host server
    HostName server.redhat.com
    DynamicForward 8890

laptop# ssh -N server &>/dev/null &

And start Chrome with this configuration:

laptop# google-chrome --proxy-pac-url=file:///home/lzap/proxies.pac

When working on The Foreman development, I also want the application to be able to connect to various services (like oVirt or RHEV compute resources). This turns to be possible too. The key utility is tsocks wrapper which is available in Fedora. We can use the very same ssh dynamic tunnel for that.

laptop# cat /etc/tsocks.conf
local = 192.168.0.0/255.255.255.0
server = 127.0.0.1
server_port = 8890

The only difference in dnsmasq configuration on the server is that I want to open it for incoming DNS requests, because my laptop needs to resolve hosts from zzz.lan:

server# cat /etc/dnsmasq.d/caching
bind-interfaces
resolv-file=/etc/resolv.dnsmasq
interface=em1
interface=lo
no-dhcp-interface=em1

And of course we need to open firewall port:

server# firewall-cmd --add-service=dns --permanent

Now on the laptop, one more change is needed. I need to direct my local dnsmasq daemon to resolve from the server dnsmasq (assuming my server IP address is 10.90.90.90):

laptop# cat /etc/dnsmasq.d/zzz.lan
server=/zzz.lan/10.90.90.90

After reloading dnsmasq, I can finally use tsocks:

laptop# tsocks wget -O - http://ovirt34.zzz.lan

So I can start my application allowing it to connect to zzz.lan NAT network:

laptop# tsocks foreman_application

That’s it. Hope you were inspired.

31 July 2014 | linux | fedora

Track short lived processes with auditd

I had a hard time to find some more information about short lived processes when I was building Foreman SELinux policy for OpenStack Installer (Staypuft). I wanted to see process name and all the parameters for all possible processes spawned on my system. My plan was to write a SystemTap script, but thank to Mirek Grepl and Steve Grubb I learned much easier trick to do that. This assumes you have auditd daemon installed (RHEL/Fedoras are fine):

auditctl -a exit,always -S execve

Then tail your /var/log/audit/audit.log to see messages. Warning: This can flood your system heavily. You will see some decent content there like:

type=CWD msg=audit(1401202071.000:14728):  cwd="/"
type=PATH msg=audit(1401202071.000:14728): item=0 name="/usr/sbin/ps" nametype=UNKNOWN
type=SYSCALL msg=audit(1401202071.000:14729): arch=c000003e syscall=59 success=yes exit=0 a0=7fe10bbe6369 a1=7fe10bbe7580 a2=7fff0b1d3208 a3=0 items=2 ppid=26400 pid=5161 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=875 comm="ps" exe="/bin/ps" subj=unconfined_u:system_r:passenger_t:s0 key=(null)
type=EXECVE msg=audit(1401202071.000:14729): argc=5 a0="ps" a1="-o" a2="pid,ppid,%cpu,rss,vsize,pgid,command" a3="-p" a4="26974,26817"

I was able to tell that mod_passenger is spawning /bin/ps every few seconds. This is very useful when tracking down some processes with long puppet runs. To turn the audit rule down, replace -a option with -d:

auditctl -d exit,always -S execve

You can filter out audit logs, see auditctl manual page. For example I was interested in processes running in passenger_t SELinux domain:

auditctl -a exit,always -S execve -F subj_type=passenger_t

To add rules permanently, edit /etc/audit/audit.rules file and restart auditd daemon.

27 May 2014 | linux | fedora

SystemTap as a system wide strace tool

I needed to find a process that was searching for several file patterns. I could use strace tool but since this one was a Apache2 module, I would need to hack startup scripts and probably create a wrapper script. There was another way of doing that - using SystemTap.

My problem was simple - I wanted to see process name, pid and full absolute path for a filename pattern. The goal was set, with little bit of googling, I was able to write this (not ideal and hacky) SystemTap script:

$ cat syscall_open.stp
#!/usr/bin/env stap
#
# System-wide strace-like tool for catching file open syscalls.
#
# Usage: stap syscall_open filename_regexp
#
global myfilename
probe syscall.open {
  myfilename = filename
}
probe syscall.open.return {
  if (myfilename =~ @1) {
    printf ("%s(%d) opened %s -> %d\n", execname(), pid(), myfilename, $return)
  }
}

Before we can use SystemTap, we need to install those packages (this is for RHEL6/CentOS6). Note you do need debug info packages, otherwise this will not work (enable them via subscription-manager or in repo files):

$ yum -y install systemtap systemtap-runtime kernel-debuginfo-`uname -r` \
    kernel-debuginfo-common-`uname -i`-`uname -r` kernel-devel-`uname -r`

Usage is super simple, you can either make it executable or to see more verbose output just do:

$ stap -v syscall_open.stp native_support.so

This is effective for the whole system, after Apache2 httpd server restart and first access, I was able to find why the heck mod_passenger searches on wrong paths:

ruby(4257) opened "/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/local/share/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/local/lib64/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/share/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/lib64/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/share/rubygems/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/share/ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/lib64/ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so" -> -2
ruby(4257) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.rb" -> -2
ruby(4257) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.so" -> -2
ruby(4282) opened "/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/local/share/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/local/lib64/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/share/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/lib64/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/share/rubygems/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/share/ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/lib64/ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so" -> -2
ruby(4282) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.rb" -> -2
ruby(4282) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.so" -> -2

SystemTap is awesome tool. It helps me every week :-)

16 May 2014 | linux | fedora

Virt builder quick provision script

Until now, I was using snap-guest tool together with Foreman to spawn my development/test virtual machines. I have been carefully monitoring (virt-builder)[http://libguestfs.org/virt-builder.1.html] tool in recent Fedoras and it looks like the version from Fedora 20 is finally usable for my scenarios (thanks Rich for fixing some annoying bugs for me).

Therefore I am happy to announce new tool called fvb which stands for Foreman-virt-builder. This little script is based around two little commands virt-builder/virt-install and it has grown to something bigger over last couple of months. The usage is simple:

fvb --install-deps

The above command should install all required dependencies on Red Hat systems. Note Fedora 20+ is required to get this working since this needs some latest and greatest features from virt-builder.

fvb -n my-nightly-foreman -f
fvb -n rc-foreman -- "FOREMAN_REPO=rc" "MYVAR=value"
fvb -n stable-foreman -- "FOREMAN_REPO=releases/1.4"
fvb -n bz981123 -- "KOJI_BUILD=http://mykoji.blah/taskinfo?taskID=97281"
fvb -n my-own-script --script fb-xyz.bats -- BATS_REPOOWNER=lzap BATS_BRANCH=test

You may notice download of centos-6.xz file into ~/.cache/virt-builder. This will only happen once, or when new CentOS minor release is uploaded. The template image is handled by virt-builder project itself.

The script works the following way:

  • Generates MAC address from hypervisor hostname and guest hostname (hash) so it does not change when you provision again.
  • Removes existing guest from libvirt (if --force is given).
  • Generates random IP address.
  • Allocates DHCP and DNS entry with the MAC and IP in libvirt network (“default”) overwriting existing entries. This is compatible with virsh Foreman provider (and also works the similar way).
  • Installs your public SSH key on the image.
  • Creates a new VM image based on CentOS6 using virt-builder.
  • Configures a firstboot script:
    • install git
    • install bats framework
    • install foreman-bats scripts
    • run script provided by --script option (by default fb-install-foreman.bats)
  • Spawns the new image with virt-install (with nested kvm enabled if possible).
  • Refreshes dnsmasq configuration.
  • Waits until DHCP daemon (dnsmasq) returns the IP address.
  • Calls fortune command.

Dnsmasq configuration (optional)

By default dnsmasq is installed on Fedora and Red Hat systems, but the daemon is turned off and it is only used by libvirt. What I usually do is I have it configured as a caching DNS daemon on my laptop. It not only speeds up DNS queries (web browsing is much faster on all connections), but it also allows me to redirect DNS queries for different domains to different DNS servers. For example:

# grep -v ^# /etc/dnsmasq.conf | sort -u
bind-interfaces
cache-size=500
conf-dir=/etc/dnsmasq.d
listen-address=127.0.0.1
resolv-file=/etc/resolv.dnsmasq

# cat /etc/resolv.conf
nameserver 127.0.0.1
domain redhat.com

# cat /etc/resolv.dnsmasq
nameserver 8.8.8.8
nameserver 8.8.4.4

# cat /etc/dnsmasq.d/local.lan
addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts

In the configuration above, you can see that dnsmasq is configured to read nameservers from a separate file (/etc/resolv.dnsmasq) where it has Google DNS public servers and to respond on localhost and to act like a simple DNS caching daemon. The system is configured to take advantage of that.

The last bit is configuration for local.lan domain (default --domain for fvb script). Libvirt keeps host names in this file and since this is in dnsmasq format, we can read it directly.

What does it mean? Well, if you configure everything according to this, once you spawn a VM called “test”, the box named “test.local.lan” will be immediately available, so you can do this:

ssh root@test.local.lan

Or even visit this:

http://test.local.lan

That’s kinda cool, isn’t it?

Notes to redhatters

If you are not a redhatter, skip to the next section. If you want this setup, make sure you have something like this:

# cat /etc/dnsmasq.d/redhat.com
server=/redhat.com/9.8.7.1
server=/redhat.com/9.8.7.2
server=/www.redhat.com/8.8.8.8
server=/vpn-concentrator-1.redhat.com/8.8.8.8
server=/vpn-concentrator-2.redhat.com/8.8.8.8

This configuration adds separate handling for redhat.com domain which goes to internal servers (these are dummy values). Also note that I want to use public DNS for vpn concentrators and web site (so I am able to connect initially and also the site works when not connected).

This way you can enjoy advantage of fvb script, fast public DNS lookups and fast internal lookups.

What next?

The fvb script is just a little prototype. It lives here and I might move it into a separate repository eventually.

The script can be used for anything, you can provide different distribution (use virt-builder -l to list all the base images available to you) and provide any command with --script that should be executed. For example to deploy Katello Foreman plugin (which we sometimes call “foretello”), do:

fvb -n knightly --script 'git clone https://github.com/Katello/katello-deploy.git && cd katello-deploy && ./setup.rb centos6'

The script accepts any shell variables and values after the -- option, so you are able to pass anything to your own scripts when needed.

Please send me patches and comments. If you like it, I can rename the project and generalize it a bit so it is useful for others as well.

Nested virtualization

I have one extra trick for you. If you do

echo “options kvm-intel nested=1″ | sudo tee /etc/modprobe.d/kvm-intel.conf

on the host machine and restart, you should be able to use KVM inside KVM. The fvb script spawns machines with required nesting options turned on, so you can configure Foreman/Katello for provisioning there.

Help

usage: fvb options -- VARIABLE1=value1 ...

Script for building images with Foreman preinstalled using virt-builder and
bats suite on local libvirt host. The script also modifies dnsmasq
configuration and sends SIGHUP to refresh so the hostname is instantly
available. The root password is "redhat" but you should have your public
ssh key installed by default.

OPTIONS:
  --help | -h
        Show this message

  --name | -n
        Image name, default: nightly

  --distro | -d
        Distribution base image for virt-builder, default: centos-6

  --force | -f
        Overwrite target image and VM if they exist

  --no-sudo
        Do not use sudo for building and running VM - you will need to
        set --image-dir accordingly too when running under regular user.

  --image-dir [path]
        Target images path
        Default: /var/lib/libvirt/images

  --domain [domain]
        Domain suffix like "mycompany.com", default: local.lan

  --subnet [subnet]
        Subnet to be used for writing DHCP/DNS configuration
        Default: 192.168.122 (note there is no suffix or period)

  --pub-key [key]
        Install this public ssh key
        Default: /home/you/.ssh/id_rsa.pub

  --script [name]
        BATS script to execute during first boot (can be any shell command)
        Default: fb-install-foreman.bats

  --install-deps
        Install required dependencies

Next time!

15 May 2014 | linux | fedora

Hidden feature of Fedora 20 - pass cli manager

When it comes to password management, I’ve always been happy user of KeepassX for many years. But when I stumbled upon new and simple tool called “pass” and I realized that I use mouse (trackball actually) with only two applications: Google Chrome and, of course, KeepassX in my workflow (i3, mutt, vim, ssh).

This tool is basically a shell wrapper around gpg, git and few other tools:

# yum install pass pinentry-gtk

It’s definitely not a monster tool, which I appreciate:

# rpm -ql pass
/etc/bash_completion.d/password-store
/usr/bin/pass
/usr/share/doc/pass
/usr/share/doc/pass/COPYING
/usr/share/doc/pass/README
/usr/share/man/man1/pass.1.gz

Let’s read the completion (or re-login to your shell):

# source /etc/bash_completion.d/password-store

What you want is to create separate gpg key for your passwords:

# gpg --gen-key

Give it a name (you can skip the e-mail) and comment. In my case this was “Lukas Zapletal (my passwords)”. You will not share this one at all. And make sure to use safe master password (passphrase).

Now you want to load gpg agent. Make sure you put this in your .bashrc as well, otherwise you would need to put your master password over and over again:

# eval "$(gpg-agent --daemon 2>/dev/null)"

In Fedora, do not miss the step of starting a gpg-agent, otherwise pass will not work as it spawns gpg with --batch parameter. If you do not like gpg-agent, you need to remove this option from /usr/bin/pass or upgrade to the latest upstream version 1.5+ which does not have this.

The (pass)[http://www.zx2c4.com/projects/password-store/] tool provides many helper scripts and importers including keepassx2pass.py which works great (you need to export your database to the XML format first). Setting up my database was matter of two minutes. A bit of warning - if you have multiline comments, note that the KeepassX importer only fetches the first comment (I’ll push a fix for that most likely).

Usage is simple enough:

# pass init "Lukas Zapletal (pass)"
# pass insert Business/cheese-whiz-factory
# pass -c Email/zx2c4.com
Copied Email/jason@zx2c4.com to clipboard. Will clear in 45 seconds.

Upgrade to “pass”, your whist will appreciate that.

15 April 2014 | linux | fedora

twitter.com linkedin.com
google.com/+ facebook.com
flickr.com youtube.com