Tracing Ruby apps with PCP
PCP offers two APIs for instrumented applications. The first one to mention is MMV agent which uses memory mapped files for capturing high-resolution data with minimum performance impact. Currently available languages for MMV instrumentation include C/C++, Python and Perl plus native Java,Golang and Rust ports. A second agent and approach is called PMDA trace with its higher level API. It uses TCP sockets and a simple API for capturing time spent, counters, trace points and raw value observations.
The tracing API is not ideal for measuring time spent in processing web requests, but it can still be useful for tracing things like cron jobs. The API (C, Fortran and Java) is described in pmdatrace(3) man page and it is trivial, therefore I decided to create a simple Ruby wrapper which only took one evening. The wrapper offers one-to-one mapping of all functions in Ruby and also higher-level Ruby approach with blocks and more user-friendly method naming.
The first step is to install trace PMDA (agent) and pcptrace rubygem (RPM not yet available, include files and compiler required to build the rubygem). Make sure that firewall is not blocking TCP port 4323 on localhost which is used to send data from the application. Also when using SELinux the pmcd daemon will be blocked from binding, therefore it is necessary to enable boolean flag:
setsebool -P pcp_bind_all_unreserved_ports 1
Then install and configure necessary software:
yum -y install pcp-pmda-trace pcp-devel @development-tools
cd /var/lib/pcp/pmdas/trace
./Install
gem install pcptrace
Use pcp_pmcd_selinux(8) man page for more details about SELinux booleans for PCP. If you encounter any SELinux problem with PCP, please let the PCP maintainers know as they will fix it promptly (pcp-team@redhat.com, or open a bugzilla). Alternatively, the workaround is to put PCP daemons into permissive mode keeping the rest of the system confined:
semanage permissive -a pcp_pmcd_t
Using the trace API is fairly straightforward, each function returns a status code (integer), if non-zero function pmtraceerrstr can be used to find error message if needed:
cat ruby_trace_example.rb
#!/usr/bin/env ruby
require "pcptrace"
# reached a point in source code
PCPTrace::point("a_point")
# observation of an arbitrary value
PCPTrace::obs("an_observation", 130.513)
# a counter - increasing or decreasing
PCPTrace::counter("a_counter", 1)
# time spent in a transaction (or block)
PCPTrace::begin("a_transaction")
# ...
PCPTrace::end("a_transaction")
# transactions must be aborted (e.g. exception)
#PCPTrace::abort("a_transaction")
# all methods return non-zero code
result = PCPTrace::counter("a_counter", -5))
puts("Error: " + PCPTrace::errstr(result)) if result != 0
There is also more Ruby-friendly API available, see README file for more info: https://github.com/lzap/ruby-pcptrace
Tracing metrics are available in trace namespace:
pminfo trace
The trace agent uses a rolling window technique to calculate rate, total, average, min and max values for some metrics, which is very useful feature for tracing durations. By default the average is recomputed every five seconds for a period covering the prior 60 seconds. To see trace counts converted to count per second:
pmval trace.point.count
PCP also provides rate for each individual metric, let’s view that with three digits precision:
pmval -f3 trace.point.rate
Execute similar commands or use other tools like pmchart to see values for counter or observation (trace.observe.value, trace.counter.value) or to see count and rate (trace.transact.count, trace.transact.rate).
Transaction metrics (time spent) provide total_time value (trace.transact.total_time) but also ave_time, min_time and max_time aggregated values. These are quite handy for ad-hoc troubleshooting.
There is also a helper utility pmtrace for emitting tracing events from scripts like cron jobs. Although the trace API is limited and multiple syscalls are being made every measurement, it is a good starting point to go further.
On generating kickstart passwords
Although Fedora or RHEL kickstart accept root password in plain text, it’s not bad idea at all to encrypt them using standard Linux PAM mechanism. There used to be a utility which is no longer available. So I wrote a small script which does that. It is in Fedora 27+ now and on its way to EPEL7. Here is how to use it:
$ man pwkickstart
NAME
pwkickstart - generate kickstart passwords
SYNOPSIS
# pwkickstart
Password:
# md5
rootpw --iscrypted $1$SI$D964QyK1.Iz/
# sha256
rootpw --iscrypted $5$Ll/Q6$SEYqHGso3maNbtBc1wjyiyr2
# sha512
rootpw --iscrypted $6$MN7wS$1QE3FmSBrN71tXV8y.Blif1avdhTYt/
DESCRIPTION
Utility pwkickstart generates kickstart passwords from password input and
prints them to standard output in three different formats: md5, sha256 and
sha512.
Previously grub-crypt provided the same, but it is not longer available in
some Linux distributions.
The output in section was shortened to fit on screen in the SYNOPOSIS example
above.
LICENSE
MIT License
Or you can grab it from my github repo.
Have fun!
Accessing libvirt VMs via telnet
There are many “tricks” floating around how to connect to VM when you have a networking issue and ssh is not available. The idea is to use serial console to get shell access. Here is how to do this properly with RHEL7 host and guest.
First, create a VM with serial console configured with remote TCP server. There are multiple options, I find TCP server in ‘telnet’ mode the most flexible configuration because most scripting languages has the protocol built-in. You can use virt-manager or virt-install to do that:
<devices>
<serial type="tcp">
<source mode="connect" host="0.0.0.0" service="4555"/>
<protocol type="telnet"/>
<target port="0"/>
</serial>
</devices>
Boot the VM and then enable getty:
$ systemctl enable serial-getty@ttyS0.service
$ systemctl start serial-getty@ttyS0.service
That’s all, access the console interactively:
$ telnet localhost 4555
To access ‘raw’ console (protocol type in the XML snippet above), use netcat or similar tool. Other options in libvirt are logfile, UDP, pseudo TTY, named pipe, unix socket or null. You get the idea.
When creating multiple serial devices, only the first one (ttyS0) is allowed
for root access by default. To enable second one, do:
$ echo ttyS1 >> /etc/securetty
That’s all for today.
Hidden gem of Fedora: virt-builder
Or I should say hidden gem of Fedora, CentOS and RHEL. This fantastic tool is part of package named libguestfs-tools-c and it’s fast image builder. To create Fedora 27 10GB image with root password set to “redhat”:
virt-builder fedora-27 --size=10G \
--root-password password:redhat
To start the image use libvirt utility:
virt-install --import --name fedora --ram 2048 \
--disk path=fedora-27.img,format=raw --os-variant fedora27
To start Fedora 25 and immediately install Xfce desktop:
virt-builder fedora-25 --install "@Xfce Desktop"
To build Debian Wheezy image, update it and inject your SSH key to root:
virt-builder debian-7 --update --ssh-inject root:~/.ssh/id_rsa.pub
To create CentOS 7.4 image on LVM volume with a hostname set:
virt-builder centos-7.4 --output /dev/vg_virt/centos \
--root-password password:redhat --hostname centos.lan
To ___________________ (complete the sentence):
virt-builder fedora-25 \
--hostname client.example.com \
--update \
--install puppet \
--append-line '/etc/puppet/puppet.conf:[agent]' \
--append-line '/etc/puppet/puppet.conf:server = puppetmaster.example.com/' \
--run-command 'systemctl enable puppet' \
--selinux-relabel
To list available images do:
[lzap@box ~]$ virt-builder -l
centos-6 x86_64 CentOS 6.6
centos-7.0 x86_64 CentOS 7.0
centos-7.1 x86_64 CentOS 7.1
centos-7.2 aarch64 CentOS 7.2 (aarch64)
centos-7.2 x86_64 CentOS 7.2
centos-7.3 x86_64 CentOS 7.3
centos-7.4 x86_64 CentOS 7.4
cirros-0.3.1 x86_64 CirrOS 0.3.1
cirros-0.3.5 x86_64 CirrOS 0.3.5
debian-6 x86_64 Debian 6 (Squeeze)
debian-7 sparc64 Debian 7 (Wheezy) (sparc64)
debian-7 x86_64 Debian 7 (wheezy)
debian-8 x86_64 Debian 8 (jessie)
debian-9 x86_64 Debian 9 (stretch)
fedora-18 x86_64 Fedora® 18
fedora-19 x86_64 Fedora® 19
fedora-20 x86_64 Fedora® 20
fedora-21 aarch64 Fedora® 21 Server (aarch64)
fedora-21 armv7l Fedora® 21 Server (armv7l)
fedora-21 ppc64 Fedora® 21 Server (ppc64)
fedora-21 ppc64le Fedora® 21 Server (ppc64le)
fedora-21 x86_64 Fedora® 21 Server
fedora-22 aarch64 Fedora® 22 Server (aarch64)
fedora-22 armv7l Fedora® 22 Server (armv7l)
fedora-22 i686 Fedora® 22 Server (i686)
fedora-22 x86_64 Fedora® 22 Server
fedora-23 aarch64 Fedora® 23 Server (aarch64)
fedora-23 armv7l Fedora® 23 Server (armv7l)
fedora-23 i686 Fedora® 23 Server (i686)
fedora-23 ppc64 Fedora® 23 Server (ppc64)
fedora-23 ppc64le Fedora® 23 Server (ppc64le)
fedora-23 x86_64 Fedora® 23 Server
fedora-24 aarch64 Fedora® 24 Server (aarch64)
fedora-24 armv7l Fedora® 24 Server (armv7l)
fedora-24 i686 Fedora® 24 Server (i686)
fedora-24 x86_64 Fedora® 24 Server
fedora-25 aarch64 Fedora® 25 Server (aarch64)
fedora-25 armv7l Fedora® 25 Server (armv7l)
fedora-25 i686 Fedora® 25 Server (i686)
fedora-25 ppc64 Fedora® 25 Server (ppc64)
fedora-25 ppc64le Fedora® 25 Server (ppc64le)
fedora-25 x86_64 Fedora® 25 Server
fedora-26 aarch64 Fedora® 26 Server (aarch64)
fedora-26 armv7l Fedora® 26 Server (armv7l)
fedora-26 i686 Fedora® 26 Server (i686)
fedora-26 ppc64 Fedora® 26 Server (ppc64)
fedora-26 ppc64le Fedora® 26 Server (ppc64le)
fedora-26 x86_64 Fedora® 26 Server
fedora-27 aarch64 Fedora® 27 Server (aarch64)
fedora-27 armv7l Fedora® 27 Server (armv7l)
fedora-27 i686 Fedora® 27 Server (i686)
fedora-27 ppc64 Fedora® 27 Server (ppc64)
fedora-27 ppc64le Fedora® 27 Server (ppc64le)
fedora-27 x86_64 Fedora® 27 Server
freebsd-11.1 x86_64 FreeBSD 11.1
scientificlinux-6 x86_64 Scientific Linux 6.5
ubuntu-10.04 x86_64 Ubuntu 10.04 (Lucid)
ubuntu-12.04 x86_64 Ubuntu 12.04 (Precise)
ubuntu-14.04 x86_64 Ubuntu 14.04 (Trusty)
ubuntu-16.04 x86_64 Ubuntu 16.04 (Xenial)
rhel-3.9 x86_64 Red Hat Enterprise Linux 3.9
rhel-4.9 x86_64 Red Hat Enterprise Linux 4.9
rhel-5.10 x86_64 Red Hat Enterprise Linux 5.10
rhel-5.11 x86_64 Red Hat Enterprise Linux 5.11
rhel-5.11 i686 Red Hat Enterprise Linux 5.11 (i686)
rhel-6.1 x86_64 Red Hat Enterprise Linux 6.1
rhel-6.2 x86_64 Red Hat Enterprise Linux 6.2
rhel-6.3 x86_64 Red Hat Enterprise Linux 6.3
rhel-6.4 x86_64 Red Hat Enterprise Linux 6.4
rhel-6.5 x86_64 Red Hat Enterprise Linux 6.5
rhel-6.6 x86_64 Red Hat Enterprise Linux 6.6
rhel-6.7 x86_64 Red Hat Enterprise Linux 6.7
rhel-6.8 x86_64 Red Hat Enterprise Linux 6.8
rhel-6.8 i686 Red Hat Enterprise Linux 6.8 (i686)
rhel-7.0 x86_64 Red Hat Enterprise Linux 7.0
rhel-7.1 x86_64 Red Hat Enterprise Linux 7.1
rhel-7.1 ppc64 Red Hat Enterprise Linux 7.1 (ppc64)
rhel-7.1 ppc64le Red Hat Enterprise Linux 7.1 (ppc64le)
rhel-7.2 x86_64 Red Hat Enterprise Linux 7.2
rhel-7.2 aarch64 Red Hat Enterprise Linux 7.2 for aarch64
rhel-7.2 ppc64le Red Hat Enterprise Linux 7.2 (ppc64le)
rhel-7.3 x86_64 Red Hat Enterprise Linux 7.3
rhel-7.3 aarch64 Red Hat Enterprise Linux 7.3 for aarch64
rhel-7.3 ppc64 Red Hat Enterprise Linux 7.3 (ppc64)
rhel-7.3 ppc64le Red Hat Enterprise Linux 7.3 (ppc64le)
rhel-7.4 x86_64 Red Hat Enterprise Linux 7.4
rhel-7.4 aarch64 Red Hat Enterprise Linux 7.4 for aarch64
rhel-7.4 ppc64 Red Hat Enterprise Linux 7.4 (ppc64)
rhel-7.4 ppc64le Red Hat Enterprise Linux 7.4 (ppc64le)
Keep in mind that RHEL systems are only available to redhatters, a repository
must be configured via /etc/virt-builder/repos.d/rhel.conf file to have them.
Search the intranet.
This post barely scratched the surface. This utility can do more. Much more.
Send beer, t-shirts and flowers to Richard W.M. Jones and his team of contributors and maintainers. Thanks folks!
How to remove file from last commit
Happens all the time to me. When I mis-commit a_file.txt the easiest way to revert the change from the last commit can be tricky:
git reset --soft HEAD^
git reset HEAD a_file.txt
git commit
But I figured out better way today:
git checkout HEAD^ a_file.txt
git commit -a --amend
That’s exactly one command saved plus I don’t need to dig the commit message. I
know it is possible to reference it via ORIG_HEAD but I never use that and I
don’t remember.
Done!
