Track short lived processes with auditd

I had a hard time to find some more information about short lived processes when I was building Foreman SELinux policy for OpenStack Installer (Staypuft). I wanted to see process name and all the parameters for all possible processes spawned on my system. My plan was to write a SystemTap script, but thank to Mirek Grepl and Steve Grubb I learned much easier trick to do that. This assumes you have auditd daemon installed (RHEL/Fedoras are fine):

auditctl -a exit,always -S execve

Then tail your /var/log/audit/audit.log to see messages. Warning: This can flood your system heavily. You will see some decent content there like:

type=CWD msg=audit(1401202071.000:14728):  cwd="/"
type=PATH msg=audit(1401202071.000:14728): item=0 name="/usr/sbin/ps" nametype=UNKNOWN
type=SYSCALL msg=audit(1401202071.000:14729): arch=c000003e syscall=59 success=yes exit=0 a0=7fe10bbe6369 a1=7fe10bbe7580 a2=7fff0b1d3208 a3=0 items=2 ppid=26400 pid=5161 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=875 comm="ps" exe="/bin/ps" subj=unconfined_u:system_r:passenger_t:s0 key=(null)
type=EXECVE msg=audit(1401202071.000:14729): argc=5 a0="ps" a1="-o" a2="pid,ppid,%cpu,rss,vsize,pgid,command" a3="-p" a4="26974,26817"

I was able to tell that mod_passenger is spawning /bin/ps every few seconds. This is very useful when tracking down some processes with long puppet runs. To turn the audit rule down, replace -a option with -d:

auditctl -d exit,always -S execve

You can filter out audit logs, see auditctl manual page. For example I was interested in processes running in passenger_t SELinux domain:

auditctl -a exit,always -S execve -F subj_type=passenger_t

To add rules permanently, edit /etc/audit/audit.rules file and restart auditd daemon.

SystemTap as a system wide strace tool

I needed to find a process that was searching for several file patterns. I could use strace tool but since this one was a Apache2 module, I would need to hack startup scripts and probably create a wrapper script. There was another way of doing that - using SystemTap.

My problem was simple - I wanted to see process name, pid and full absolute path for a filename pattern. The goal was set, with little bit of googling, I was able to write this (not ideal and hacky) SystemTap script:

$ cat syscall_open.stp
#!/usr/bin/env stap
#
# System-wide strace-like tool for catching file open syscalls.
#
# Usage: stap syscall_open filename_regexp
#
global myfilename
probe syscall.open {
  myfilename = filename
}
probe syscall.open.return {
  if (myfilename =~ @1) {
    printf ("%s(%d) opened %s -> %d\n", execname(), pid(), myfilename, $return)
  }
}

Before we can use SystemTap, we need to install those packages (this is for RHEL6/CentOS6). Note you do need debug info packages, otherwise this will not work (enable them via subscription-manager or in repo files):

$ yum -y install systemtap systemtap-runtime kernel-debuginfo-`uname -r` \
    kernel-debuginfo-common-`uname -i`-`uname -r` kernel-devel-`uname -r`

Usage is super simple, you can either make it executable or to see more verbose output just do:

$ stap -v syscall_open.stp native_support.so

This is effective for the whole system, after Apache2 httpd server restart and first access, I was able to find why the heck mod_passenger searches on wrong paths:

ruby(4257) opened "/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/local/share/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/local/lib64/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/share/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/lib64/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/share/rubygems/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/share/ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/opt/rh/ruby193/root/usr/lib64/ruby/native/passenger_native_support.so" -> -2
ruby(4257) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so" -> -2
ruby(4257) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.rb" -> -2
ruby(4257) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.so" -> -2
ruby(4282) opened "/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/local/share/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/local/lib64/ruby/site_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/share/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/lib64/ruby/vendor_ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/share/rubygems/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/share/ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/opt/rh/ruby193/root/usr/lib64/ruby/native/passenger_native_support.so" -> -2
ruby(4282) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so" -> -2
ruby(4282) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.rb" -> -2
ruby(4282) opened "/usr/share/foreman/.passenger/native_support/4.0.18/ruby-1.9.3-x86_64-linux/passenger_native_support.so.so" -> -2

SystemTap is awesome tool. It helps me every week :-)

Virt builder quick provision script

Until now, I was using snap-guest tool together with Foreman to spawn my development/test virtual machines. I have been carefully monitoring (virt-builder)[http://libguestfs.org/virt-builder.1.html] tool in recent Fedoras and it looks like the version from Fedora 20 is finally usable for my scenarios (thanks Rich for fixing some annoying bugs for me).

Therefore I am happy to announce new tool called fvb which stands for Foreman-virt-builder. This little script is based around two little commands virt-builder/virt-install and it has grown to something bigger over last couple of months. The usage is simple:

fvb --install-deps

The above command should install all required dependencies on Red Hat systems. Note Fedora 20+ is required to get this working since this needs some latest and greatest features from virt-builder.

fvb -n my-nightly-foreman -f
fvb -n rc-foreman -- "FOREMAN_REPO=rc" "MYVAR=value"
fvb -n stable-foreman -- "FOREMAN_REPO=releases/1.4"
fvb -n bz981123 -- "KOJI_BUILD=http://mykoji.blah/taskinfo?taskID=97281"
fvb -n my-own-script --script fb-xyz.bats -- BATS_REPOOWNER=lzap BATS_BRANCH=test

You may notice download of centos-6.xz file into ~/.cache/virt-builder. This will only happen once, or when new CentOS minor release is uploaded. The template image is handled by virt-builder project itself.

The script works the following way:

• Generates MAC address from hypervisor hostname and guest hostname (hash) so it does not change when you provision again.
• Removes existing guest from libvirt (if --force is given).
• Generates random IP address.
• Allocates DHCP and DNS entry with the MAC and IP in libvirt network (“default”) overwriting existing entries. This is compatible with virsh Foreman provider (and also works the similar way).
• Installs your public SSH key on the image.
• Creates a new VM image based on CentOS6 using virt-builder.
• Configures a firstboot script:
  • install git
  • install bats framework
  • install foreman-bats scripts
  • run script provided by --script option (by default fb-install-foreman.bats)
• Spawns the new image with virt-install (with nested kvm enabled if possible).
• Refreshes dnsmasq configuration.
• Waits until DHCP daemon (dnsmasq) returns the IP address.
• Calls fortune command.

Dnsmasq configuration (optional)

By default dnsmasq is installed on Fedora and Red Hat systems, but the daemon is turned off and it is only used by libvirt. What I usually do is I have it configured as a caching DNS daemon on my laptop. It not only speeds up DNS queries (web browsing is much faster on all connections), but it also allows me to redirect DNS queries for different domains to different DNS servers. For example:

# grep -v ^# /etc/dnsmasq.conf | sort -u
bind-interfaces
cache-size=500
conf-dir=/etc/dnsmasq.d
listen-address=127.0.0.1
resolv-file=/etc/resolv.dnsmasq

# cat /etc/resolv.conf
nameserver 127.0.0.1
domain redhat.com

# cat /etc/resolv.dnsmasq
nameserver 8.8.8.8
nameserver 8.8.4.4

# cat /etc/dnsmasq.d/local.lan
addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts

In the configuration above, you can see that dnsmasq is configured to read nameservers from a separate file (/etc/resolv.dnsmasq) where it has Google DNS public servers and to respond on localhost and to act like a simple DNS caching daemon. The system is configured to take advantage of that.

The last bit is configuration for local.lan domain (default --domain for fvb script). Libvirt keeps host names in this file and since this is in dnsmasq format, we can read it directly.

What does it mean? Well, if you configure everything according to this, once you spawn a VM called “test”, the box named “test.local.lan” will be immediately available, so you can do this:

ssh root@test.local.lan

Or even visit this:

http://test.local.lan

That’s kinda cool, isn’t it?

Notes to redhatters

If you are not a redhatter, skip to the next section. If you want this setup, make sure you have something like this:

# cat /etc/dnsmasq.d/redhat.com
server=/redhat.com/9.8.7.1
server=/redhat.com/9.8.7.2
server=/www.redhat.com/8.8.8.8
server=/vpn-concentrator-1.redhat.com/8.8.8.8
server=/vpn-concentrator-2.redhat.com/8.8.8.8

This configuration adds separate handling for redhat.com domain which goes to internal servers (these are dummy values). Also note that I want to use public DNS for vpn concentrators and web site (so I am able to connect initially and also the site works when not connected).

This way you can enjoy advantage of fvb script, fast public DNS lookups and fast internal lookups.

What next?

The fvb script is just a little prototype. It lives here and I might move it into a separate repository eventually.

The script can be used for anything, you can provide different distribution (use virt-builder -l to list all the base images available to you) and provide any command with --script that should be executed. For example to deploy Katello Foreman plugin (which we sometimes call “foretello”), do:

fvb -n knightly --script 'git clone https://github.com/Katello/katello-deploy.git && cd katello-deploy && ./setup.rb centos6'

The script accepts any shell variables and values after the -- option, so you are able to pass anything to your own scripts when needed.

Please send me patches and comments. If you like it, I can rename the project and generalize it a bit so it is useful for others as well.

Nested virtualization

I have one extra trick for you. If you do

echo “options kvm-intel nested=1″ | sudo tee /etc/modprobe.d/kvm-intel.conf

on the host machine and restart, you should be able to use KVM inside KVM. The fvb script spawns machines with required nesting options turned on, so you can configure Foreman/Katello for provisioning there.

Help

usage: fvb options -- VARIABLE1=value1 ...

Script for building images with Foreman preinstalled using virt-builder and
bats suite on local libvirt host. The script also modifies dnsmasq
configuration and sends SIGHUP to refresh so the hostname is instantly
available. The root password is "redhat" but you should have your public
ssh key installed by default.

OPTIONS:
  --help | -h
        Show this message

  --name | -n
        Image name, default: nightly

  --distro | -d
        Distribution base image for virt-builder, default: centos-6

  --force | -f
        Overwrite target image and VM if they exist

  --no-sudo
        Do not use sudo for building and running VM - you will need to
        set --image-dir accordingly too when running under regular user.

  --image-dir [path]
        Target images path
        Default: /var/lib/libvirt/images

  --domain [domain]
        Domain suffix like "mycompany.com", default: local.lan

  --subnet [subnet]
        Subnet to be used for writing DHCP/DNS configuration
        Default: 192.168.122 (note there is no suffix or period)

  --pub-key [key]
        Install this public ssh key
        Default: /home/you/.ssh/id_rsa.pub

  --script [name]
        BATS script to execute during first boot (can be any shell command)
        Default: fb-install-foreman.bats

  --install-deps
        Install required dependencies

Next time!

Hidden feature of Fedora 20 - pass cli manager

When it comes to password management, I’ve always been happy user of KeepassX for many years. But when I stumbled upon new and simple tool called “pass” and I realized that I use mouse (trackball actually) with only two applications: Google Chrome and, of course, KeepassX in my workflow (i3, mutt, vim, ssh).

This tool is basically a shell wrapper around gpg, git and few other tools:

# yum install pass pinentry-gtk

It’s definitely not a monster tool, which I appreciate:

# rpm -ql pass
/etc/bash_completion.d/password-store
/usr/bin/pass
/usr/share/doc/pass
/usr/share/doc/pass/COPYING
/usr/share/doc/pass/README
/usr/share/man/man1/pass.1.gz

Let’s read the completion (or re-login to your shell):

# source /etc/bash_completion.d/password-store

What you want is to create separate gpg key for your passwords:

# gpg --gen-key

Give it a name (you can skip the e-mail) and comment. In my case this was “Lukas Zapletal (my passwords)”. You will not share this one at all. And make sure to use safe master password (passphrase).

Now you want to load gpg agent. Make sure you put this in your .bashrc as well, otherwise you would need to put your master password over and over again:

# eval "$(gpg-agent --daemon 2>/dev/null)"

In Fedora, do not miss the step of starting a gpg-agent, otherwise pass will not work as it spawns gpg with --batch parameter. If you do not like gpg-agent, you need to remove this option from /usr/bin/pass or upgrade to the latest upstream version 1.5+ which does not have this.

The (pass)[http://www.zx2c4.com/projects/password-store/] tool provides many helper scripts and importers including keepassx2pass.py which works great (you need to export your database to the XML format first). Setting up my database was matter of two minutes. A bit of warning - if you have multiline comments, note that the KeepassX importer only fetches the first comment (I’ll push a fix for that most likely).

Usage is simple enough:

# pass init "Lukas Zapletal (pass)"
# pass insert Business/cheese-whiz-factory
# pass -c Email/zx2c4.com
Copied Email/jason@zx2c4.com to clipboard. Will clear in 45 seconds.

Upgrade to “pass”, your whist will appreciate that.

SELinux Puppet update in Fedora 20 and Rawhide

We are rolling out update of Puppet to 3.4.3 in Fedora 20 and Rawhide that adds one important change. We have found that puppet master was running unconfined, therefore the Puppet SELinux policy was not effective in Fedoras.

The puppet package update fixes one little issue (missing runtime dependency) and corrects startup wrappers for systemd which puts Puppet Master into correct SELinux domain puppetmaster_t. Since this has low security impact, we have decided to backport this change into Fedora 20 too. Another reason is the change in selinux-policy package in Fedora 20 which allows us to backport the changes into EPEL7.

• Puppet Fedora 20 update
• SELinux policy Fedora 20 update

SELinux core puppet policy was refactored in paralel so we have now puppetmaster_t and puppetagent_t domains which reflects the state much better. Previously puppet agent was running under puppet_t confined domain, now it runs under puppetagent_t domain. Also the agent has loosed security rules which is great improvement too.

To update your host do the following:

yum --enablerepo=updates-testing makecache
yum --enablerepo=updates-testing update selinux-policy puppet puppet-server

When upgrading make sure you have the correct versions on the mirror:

• puppet 3.4.3-3.fc20 or higher
• policy 3.12.1-153.fc20 or higher

Restart puppetmaster, agent and watch for denials. Report success and failures in my comments or in the update comments.

grep AVC /var/log/audit/audit.log

Let’s make sure we have rock solid version of Puppet hardened with SELinux in the best quality possible in EPEL7. Thanks for help!