Tracing Ruby apps with PCP

PCP offers two APIs for instrumented applications. The first one to mention is MMV agent which uses memory mapped files for capturing high-resolution data with minimum performance impact. Currently available languages for MMV instrumentation include C/C++, Python and Perl plus native Java,Golang and Rust ports. A second agent and approach is called PMDA trace with its higher level API. It uses TCP sockets and a simple API for capturing time spent, counters, trace points and raw value observations.

The tracing API is not ideal for measuring time spent in processing web requests, but it can still be useful for tracing things like cron jobs. The API (C, Fortran and Java) is described in pmdatrace(3) man page and it is trivial, therefore I decided to create a simple Ruby wrapper which only took one evening. The wrapper offers one-to-one mapping of all functions in Ruby and also higher-level Ruby approach with blocks and more user-friendly method naming.

The first step is to install trace PMDA (agent) and pcptrace rubygem (RPM not yet available, include files and compiler required to build the rubygem). Make sure that firewall is not blocking TCP port 4323 on localhost which is used to send data from the application. Also when using SELinux the pmcd daemon will be blocked from binding, therefore it is necessary to enable boolean flag:

setsebool -P pcp_bind_all_unreserved_ports 1

Then install and configure necessary software:

yum -y install pcp-pmda-trace pcp-devel @development-tools
cd /var/lib/pcp/pmdas/trace
gem install pcptrace

Use pcp_pmcd_selinux(8) man page for more details about SELinux booleans for PCP. If you encounter any SELinux problem with PCP, please let the PCP maintainers know as they will fix it promptly (, or open a bugzilla).  Alternatively, the workaround is to put PCP daemons into permissive mode keeping the rest of the system confined:

semanage permissive -a pcp_pmcd_t

Using the trace API is fairly straightforward, each function returns a status code (integer), if non-zero function pmtraceerrstr can be used to find error message if needed:

cat ruby_trace_example.rb
#!/usr/bin/env ruby

require "pcptrace"

# reached a point in source code

# observation of an arbitrary value
PCPTrace::obs("an_observation", 130.513)

# a counter - increasing or decreasing
PCPTrace::counter("a_counter", 1)

# time spent in a transaction (or block)
# ...

# transactions must be aborted (e.g. exception)

# all methods return non-zero code
result = PCPTrace::counter("a_counter", -5))
puts("Error: " + PCPTrace::errstr(result)) if result != 0

There is also more Ruby-friendly API available, see README file for more info:

Tracing metrics are available in trace namespace:

pminfo trace

The trace agent uses a rolling window technique to calculate rate, total, average, min and max values for some metrics, which is very useful feature for tracing durations. By default the average is recomputed every five seconds for a period covering the prior 60 seconds. To see trace counts converted to count per second:

pmval trace.point.count

PCP also provides rate for each individual metric, let’s view that with three digits precision:

pmval -f3 trace.point.rate

Execute similar commands or use other tools like pmchart to see values for counter or observation (trace.observe.value, trace.counter.value) or to see count and rate (trace.transact.count, trace.transact.rate).

Transaction metrics (time spent) provide total_time value (trace.transact.total_time) but also ave_time, min_time and max_time aggregated values. These are quite handy for ad-hoc troubleshooting.

There is also a helper utility pmtrace for emitting tracing events from scripts like cron jobs. Although the trace API is limited and multiple syscalls are being made every measurement, it is a good starting point to go further.

21 March 2018 | linux | fedora | pcp

On generating kickstart passwords

Although Fedora or RHEL kickstart accept root password in plain text, it’s not bad idea at all to encrypt them using standard Linux PAM mechanism. There used to be a utility which is no longer available. So I wrote a small script which does that. It is in Fedora 27+ now and on its way to EPEL7. Here is how to use it:

$ man pwkickstart


pwkickstart - generate kickstart passwords


  # pwkickstart

  # md5
  rootpw --iscrypted $1$SI$D964QyK1.Iz/
  # sha256
  rootpw --iscrypted $5$Ll/Q6$SEYqHGso3maNbtBc1wjyiyr2
  # sha512
  rootpw --iscrypted $6$MN7wS$1QE3FmSBrN71tXV8y.Blif1avdhTYt/


Utility pwkickstart generates kickstart passwords from password input and
prints them to standard output in three different formats: md5, sha256 and

Previously grub-crypt provided the same, but it is not longer available in
some Linux distributions.

The output in section was shortened to fit on screen in the SYNOPOSIS example


MIT License

Or you can grab it from my github repo.

Have fun!

27 February 2018 | linux | fedora

Accessing libvirt VMs via telnet

There are many “tricks” floating around how to connect to VM when you have a networking issue and ssh is not available. The idea is to use serial console to get shell access. Here is how to do this properly with RHEL7 host and guest.

First, create a VM with serial console configured with remote TCP server. There are multiple options, I find TCP server in ‘telnet’ mode the most flexible configuration because most scripting languages has the protocol built-in. You can use virt-manager or virt-install to do that:

  <serial type="tcp">
    <source mode="connect" host="" service="4555"/>
    <protocol type="telnet"/>
    <target port="0"/>

Boot the VM and then enable getty:

$ systemctl enable serial-getty@ttyS0.service
$ systemctl start serial-getty@ttyS0.service

That’s all, access the console interactively:

$ telnet localhost 4555

To access ‘raw’ console (protocol type in the XML snippet above), use netcat or similar tool. Other options in libvirt are logfile, UDP, pseudo TTY, named pipe, unix socket or null. You get the idea.

When creating multiple serial devices, only the first one (ttyS0) is allowed for root access by default. To enable second one, do:

$ echo ttyS1 >> /etc/securetty

That’s all for today.

20 February 2018 | linux | fedora | rhel | libvirt

Hidden gem of Fedora: virt-builder

Or I should say hidden gem of Fedora, CentOS and RHEL. This fantastic tool is part of package named libguestfs-tools-c and it’s fast image builder. To create Fedora 27 10GB image with root password set to “redhat”:

virt-builder fedora-27 --size=10G \
  --root-password password:redhat

To start the image use libvirt utility:

virt-install --import --name fedora --ram 2048 \
  --disk path=fedora-27.img,format=raw --os-variant fedora27

To start Fedora 25 and immediately install Xfce desktop:

virt-builder fedora-25 --install "@Xfce Desktop"

To build Debian Wheezy image, update it and inject your SSH key to root:

virt-builder debian-7 --update --ssh-inject root:~/.ssh/

To create CentOS 7.4 image on LVM volume with a hostname set:

virt-builder centos-7.4 --output /dev/vg_virt/centos \
  --root-password password:redhat --hostname centos.lan

To ___________________ (complete the sentence):

virt-builder fedora-25 \
   --hostname \
   --update \
   --install puppet \
   --append-line '/etc/puppet/puppet.conf:[agent]' \
   --append-line '/etc/puppet/puppet.conf:server =' \
   --run-command 'systemctl enable puppet' \

To list available images do:

[lzap@box ~]$ virt-builder -l
centos-6                 x86_64     CentOS 6.6
centos-7.0               x86_64     CentOS 7.0
centos-7.1               x86_64     CentOS 7.1
centos-7.2               aarch64    CentOS 7.2 (aarch64)
centos-7.2               x86_64     CentOS 7.2
centos-7.3               x86_64     CentOS 7.3
centos-7.4               x86_64     CentOS 7.4
cirros-0.3.1             x86_64     CirrOS 0.3.1
cirros-0.3.5             x86_64     CirrOS 0.3.5
debian-6                 x86_64     Debian 6 (Squeeze)
debian-7                 sparc64    Debian 7 (Wheezy) (sparc64)
debian-7                 x86_64     Debian 7 (wheezy)
debian-8                 x86_64     Debian 8 (jessie)
debian-9                 x86_64     Debian 9 (stretch)
fedora-18                x86_64     Fedora® 18
fedora-19                x86_64     Fedora® 19
fedora-20                x86_64     Fedora® 20
fedora-21                aarch64    Fedora® 21 Server (aarch64)
fedora-21                armv7l     Fedora® 21 Server (armv7l)
fedora-21                ppc64      Fedora® 21 Server (ppc64)
fedora-21                ppc64le    Fedora® 21 Server (ppc64le)
fedora-21                x86_64     Fedora® 21 Server
fedora-22                aarch64    Fedora® 22 Server (aarch64)
fedora-22                armv7l     Fedora® 22 Server (armv7l)
fedora-22                i686       Fedora® 22 Server (i686)
fedora-22                x86_64     Fedora® 22 Server
fedora-23                aarch64    Fedora® 23 Server (aarch64)
fedora-23                armv7l     Fedora® 23 Server (armv7l)
fedora-23                i686       Fedora® 23 Server (i686)
fedora-23                ppc64      Fedora® 23 Server (ppc64)
fedora-23                ppc64le    Fedora® 23 Server (ppc64le)
fedora-23                x86_64     Fedora® 23 Server
fedora-24                aarch64    Fedora® 24 Server (aarch64)
fedora-24                armv7l     Fedora® 24 Server (armv7l)
fedora-24                i686       Fedora® 24 Server (i686)
fedora-24                x86_64     Fedora® 24 Server
fedora-25                aarch64    Fedora® 25 Server (aarch64)
fedora-25                armv7l     Fedora® 25 Server (armv7l)
fedora-25                i686       Fedora® 25 Server (i686)
fedora-25                ppc64      Fedora® 25 Server (ppc64)
fedora-25                ppc64le    Fedora® 25 Server (ppc64le)
fedora-25                x86_64     Fedora® 25 Server
fedora-26                aarch64    Fedora® 26 Server (aarch64)
fedora-26                armv7l     Fedora® 26 Server (armv7l)
fedora-26                i686       Fedora® 26 Server (i686)
fedora-26                ppc64      Fedora® 26 Server (ppc64)
fedora-26                ppc64le    Fedora® 26 Server (ppc64le)
fedora-26                x86_64     Fedora® 26 Server
fedora-27                aarch64    Fedora® 27 Server (aarch64)
fedora-27                armv7l     Fedora® 27 Server (armv7l)
fedora-27                i686       Fedora® 27 Server (i686)
fedora-27                ppc64      Fedora® 27 Server (ppc64)
fedora-27                ppc64le    Fedora® 27 Server (ppc64le)
fedora-27                x86_64     Fedora® 27 Server
freebsd-11.1             x86_64     FreeBSD 11.1
scientificlinux-6        x86_64     Scientific Linux 6.5
ubuntu-10.04             x86_64     Ubuntu 10.04 (Lucid)
ubuntu-12.04             x86_64     Ubuntu 12.04 (Precise)
ubuntu-14.04             x86_64     Ubuntu 14.04 (Trusty)
ubuntu-16.04             x86_64     Ubuntu 16.04 (Xenial)
rhel-3.9                 x86_64     Red Hat Enterprise Linux 3.9
rhel-4.9                 x86_64     Red Hat Enterprise Linux 4.9
rhel-5.10                x86_64     Red Hat Enterprise Linux 5.10
rhel-5.11                x86_64     Red Hat Enterprise Linux 5.11
rhel-5.11                i686       Red Hat Enterprise Linux 5.11 (i686)
rhel-6.1                 x86_64     Red Hat Enterprise Linux 6.1
rhel-6.2                 x86_64     Red Hat Enterprise Linux 6.2
rhel-6.3                 x86_64     Red Hat Enterprise Linux 6.3
rhel-6.4                 x86_64     Red Hat Enterprise Linux 6.4
rhel-6.5                 x86_64     Red Hat Enterprise Linux 6.5
rhel-6.6                 x86_64     Red Hat Enterprise Linux 6.6
rhel-6.7                 x86_64     Red Hat Enterprise Linux 6.7
rhel-6.8                 x86_64     Red Hat Enterprise Linux 6.8
rhel-6.8                 i686       Red Hat Enterprise Linux 6.8 (i686)
rhel-7.0                 x86_64     Red Hat Enterprise Linux 7.0
rhel-7.1                 x86_64     Red Hat Enterprise Linux 7.1
rhel-7.1                 ppc64      Red Hat Enterprise Linux 7.1 (ppc64)
rhel-7.1                 ppc64le    Red Hat Enterprise Linux 7.1 (ppc64le)
rhel-7.2                 x86_64     Red Hat Enterprise Linux 7.2
rhel-7.2                 aarch64    Red Hat Enterprise Linux 7.2 for aarch64
rhel-7.2                 ppc64le    Red Hat Enterprise Linux 7.2 (ppc64le)
rhel-7.3                 x86_64     Red Hat Enterprise Linux 7.3
rhel-7.3                 aarch64    Red Hat Enterprise Linux 7.3 for aarch64
rhel-7.3                 ppc64      Red Hat Enterprise Linux 7.3 (ppc64)
rhel-7.3                 ppc64le    Red Hat Enterprise Linux 7.3 (ppc64le)
rhel-7.4                 x86_64     Red Hat Enterprise Linux 7.4
rhel-7.4                 aarch64    Red Hat Enterprise Linux 7.4 for aarch64
rhel-7.4                 ppc64      Red Hat Enterprise Linux 7.4 (ppc64)
rhel-7.4                 ppc64le    Red Hat Enterprise Linux 7.4 (ppc64le)

Keep in mind that RHEL systems are only available to redhatters, a repository must be configured via /etc/virt-builder/repos.d/rhel.conf file to have them. Search the intranet.

This post barely scratched the surface. This utility can do more. Much more.

Send beer, t-shirts and flowers to Richard W.M. Jones and his team of contributors and maintainers. Thanks folks!

06 February 2018 | linux | fedora | rhel | centos

How to remove file from last commit

Happens all the time to me. When I mis-commit a_file.txt the easiest way to revert the change from the last commit can be tricky:

git reset --soft HEAD^
git reset HEAD a_file.txt
git commit

But I figured out better way today:

git checkout HEAD^ a_file.txt
git commit -a --amend

That’s exactly one command saved plus I don’t need to dig the commit message. I know it is possible to reference it via ORIG_HEAD but I never use that and I don’t remember.


01 February 2018 | linux | git